chanzuckerberg / cellxgene-census

CZ CELLxGENE Discover Census
https://chanzuckerberg.github.io/cellxgene-census/
MIT License
84 stars 20 forks source link

[builder] Save configuration file as part of the build process #790

Open ebezzi opened 1 year ago

ebezzi commented 1 year ago

We should dump the configuration file used by the builder to the census artifacts. This will make it easier to debug issues with the builder and help reproducibility.

For safety and convenience, use the log (private) bucket: s3://cellxgene-data-public-logs/builder/<TAG>/config.yaml

Should be written by both cellxgene_census_builder/build_soma/__main__.py and cellxgene_census_builder/__main__.py

bkmartinjr commented 8 months ago

should also save the datasets manifest used to seed the build process