cybergis / cybergis-compute-core

Apache License 2.0
8 stars 6 forks source link

Updating/Cacheing Models More Efficiently #113

Open alexandermichels opened 7 months ago

alexandermichels commented 7 months ago

Currently, model repos are zipped, transferred (through scp), and then unzipped every time a job is submitted, regardless of if the repo has changed. This can cause a significant delay even if the model hasn't changed. We need a solution to do this all more efficiently. For example, keeping a zipped copy of each job on each HPC and only updating it as needed (or when specifically requested by an end-user through a route/button).