Need a way to persist input/uploads to the S3 workspace.

VEuPathDB / lib-compute-platform

Async compute platform core.

1 stars 0 forks source link

Need a way to persist input/uploads to the S3 workspace. #4

Closed Foxcapades closed 2 years ago

Foxcapades commented 2 years ago

Presently only the config is persisted on job creation, there should be a way to persist arbitrary files as well.

Foxcapades commented 2 years ago

This will complicate the workspace reset when clearing dead jobs. Right now it just keeps the config file. For this to work it will need to know the names of the input files that are to be kept as well...

Ideas:

Store the input file names in the DB and use them in the reset?
- This may get messy and bring the DB and S3 managers too close?
Wipe out the remote workspace entirely and re-create?
- Where will the input files come from in this case, they will only exist in S3 at this point

Foxcapades commented 2 years ago

Additionally, the input files will need to be copied down from S3 when the local scratch workspace is created. This is leaning me more towards storing the names of the files in the DB