Run mappings generation as a standalone process

monarch-initiative / monarch-gene-mapping

Code for mapping source namespaces to preffered namespacing

2 stars 0 forks source link

Run mappings generation as a standalone process #5

Closed kevinschaper closed 1 year ago

kevinschaper commented 1 year ago

[ ] Make sure the downloads.yaml has everything we need
[ ] Make sure the CLI works
[ ] Add a Jenkinsfile and new job to download, generate mappings and upload

We should maybe use 'gene-mappings' as the directory in the buckets, with a dated directory and a latest in each one.

I'm not sure about upload locations, definitely gs://data-public-monarchinitiative and gs://monarch-archive. I'm not sure if it makes sense to upload to gs://monarch-ingest.

We could also consider writing to date-named GitHub releases... (and then we can bring that over to monarch-ingest)

matentzn commented 1 year ago

Can we figure out if the mapping tables are all <50MB in size? would be nice if we could use GitHub for versioning...

glass-ships commented 1 year ago

Ahh, GitHub file size limit! I'd been working with GitLab for so long I'd forgotten!

I'm not sure about all of them, but the version I just produced in testing is about 30MB, so for the time being this should be ok. What do you think @kevinschaper ?

kevinschaper commented 1 year ago

If it gets too big we can definitely break into multiple files.

We could experiment with doing it all with github even and skipping the Jenkins completely?

matentzn commented 1 year ago

I think having as much as possible done with GitHub and GitHub actions increases transparency a lot!