Create package that will project (store) information about s3 storage

[khoroshevskyi]

Bedbase not only stores data on S3 but also facilitates data transfer using standard bash commands (e.g., aws s3 cp ...). However, during the copying process, it's challenging to determine whether the destination on S3 already exists and what other files are present.

The proposed solution is to create an object that provides a projection of the files on S3. This object would assist users in uploading, deleting, and managing objects within the database.

[nsheff]

I propose no new package/object, but do this:

use boto3 to upload from within bedboss. Then, insert metadata about this upload (when completed successfully) with report using pipestat, so the database knows what files were uploaded and where they are, or whatever (or maybe it's just True if the upload was successful, or something). I guess I could see this being a JSON blob with information about all files that were transferred to s3/b2.

Then, write a function or class that can remove an entry from the database. It would query the info from the main database, then

• if qdrant is present, remove it (and update the database), then
• if s3 files are present, remove them and update the database
• remove the entry from the pephub allbeds (bedboss output pep).
• finally, if those succeeded, remove the entry from the bedbase pipestat database.

[nsheff]

This is actually the same as databio/bbconf#37

[nsheff]

I will close this in favor of the the bbconf issue