search

cybergis / cybergis-compute-core

Apache License 2.0
8 stars 6 forks source link

[Bug] Limit Git Pulls/Checkouts #79

Closed alexandermichels closed 1 year ago

alexandermichels commented 1 year ago

Currently every job submission triggers a refresh of the Git repo (see: https://github.com/cybergis/cybergis-compute-core/blob/v2/src/lib/GitUtil.ts#L23). This can cause issues if many jobs using the same Git repo are sent at the same time because multiple processes will try to update the repo in parallel.

We need a better approach to this, but I am not sure what it is. Two possibilities are:

  1. Remove this refresh from the job submission workflow and instead have a different part of the Core refresh every few minutes.
  2. Somehow make it so that the first job after X minutes will trigger a refresh, but any jobs within X minutes after do not trigger a refresh.
alexandermichels commented 1 year ago

Should be closed with #82