Bedbase --> next steps in details (backend)

[khoroshevskyi]

• [x] Create input schema for bedboss and add it to schema.databio.org
• [x] Unify config files for bedboss, bedhost ... (for bbconf), because now we have multiple in tutorial repository
• [x] Add qdrant credentials to config file in bbconf (similar to pepub?)
• [x] Make sure bedboss produces all files correctly with correct paths
• [x] Make sure bbconf works with new pipestat
• [x] Fix pipestat (expanding env variables @donaldcampbelljr)
• [x] Make sure bedhost works with new bbconf (distances functions were deleted from bbconf)
• [x] Add qdrant functions to bbconf
• [x] What is bedset? How do we store bedsets? PEPhub?

• [x] https://github.com/databio/bedbase/issues/41

  Automate bedfile uploading (https://github.com/databio/bedbase_autouploader) Few scripts that should be written

• [x] 1. Determine what files were uploaded in the last e.g. 10 days using PEPhub endpoint (Because of that we have to update pephub endpoint: https://github.com/pepkit/pephub/issues/221)
  2. Upload registry paths to upload_table in the db
  3. Main script: 1) Open PEP, 2) Download actual files, 3) Made modifications to PEP (create one big pep), to have required information (genome build, narrowPeak/broadPeak, organism, ...) the most important info (e.g. description). 3) Run bedboss using creaded PEP 4) Upload files to AWS s3. 5) Bedsets???
  4. Run checker.
  5. Create Docker that will receive all credentials and uploading dates (start -end ) and will have all necessary dependencies.

[khoroshevskyi]

Auto uploading will be happening in a different way now: 1) bbuploader will use sort function in pephub to determine what files were uploaded in the last few days 2) standardize found pep --> change attribute names as specified in schema 3) use bedboss to upload files to the database

[khoroshevskyi]

A lot of this things were fixed, and if not other issues were opened