Update batch job definitions clean up process - Githubissues

AlexsLemonade / refinebio

Refine.bio harmonizes petabytes of publicly available biological data into ready-to-use datasets for cancer researchers and AI/ML scientists.

https://www.refine.bio/

Other

126 stars 19 forks source link

Update batch job definitions clean up process #3363

Closed arkid15r closed 10 months ago

arkid15r commented 10 months ago

Issue Number

Resolves #3359

Purpose/Implementation Notes

Update batch job definitions clean up process

Narrow down expected exceptions for batch job queue deletion
Use nextToken for 100+ duplicate entries cases
Add create_qn_target.json to the list of fixed RAM jobs

Types of changes

Bugfix (non-breaking change which fixes an issue)

Checklist

[x] Lint and unit tests pass locally with my changes
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have added necessary documentation (if appropriate)
[ ] Any dependent changes have been merged and published in downstream modules

arkid15r commented 10 months ago

Could you comment on why the create_qn_target.json was added to fixed ram files?

It seems it just doesn't belong to the multi-RAM domain. The existing jobs had the same name (duplicates) and the RAM amount is static for this one https://github.com/AlexsLemonade/refinebio/blob/86f8c832f41e6868e1b1cc7827f43c6cceeb26c3/workers/batch-job-templates/create_qn_target.tpl.json#L13