Open nayib-jose-gloria opened 1 month ago
Create a "clean-up" batch job that deletes DatasetArtifacts + DatasetVersions for an input List of DatasetVersionIds.
See data model in orm.py and business logic entity objects in entities.py
Leverage existing functions for artifact/DB clean-up such as: business layer function to delete dataset version artifacts from S3
persistence layer function to delete dataset version + associated dataset artifact rows from the DB
Other batch jobs created in data-portal before: Publish revisions: code and infra Dataset metadata update: code and infra
Memory / vcpu requirements should be reasonably fine-tuned during testing in rdev environment.
Note to @lvreynoso: @Bento007 has suggested looking into using an aws lambda for this instead of a batch job; I think its worth considering
Create a "clean-up" batch job that deletes DatasetArtifacts + DatasetVersions for an input List of DatasetVersionIds.
See data model in orm.py and business logic entity objects in entities.py
Leverage existing functions for artifact/DB clean-up such as: business layer function to delete dataset version artifacts from S3
persistence layer function to delete dataset version + associated dataset artifact rows from the DB
Other batch jobs created in data-portal before: Publish revisions: code and infra Dataset metadata update: code and infra
Memory / vcpu requirements should be reasonably fine-tuned during testing in rdev environment.