Run mappings generation as a standalone process

monarch-initiative / monarch-gene-mapping

Code for mapping source namespaces to preffered namespacing

2 stars 0 forks source link

Run mappings generation as a standalone process #5

Closed kevinschaper closed 2 years ago

kevinschaper commented 2 years ago

[ ] Make sure the downloads.yaml has everything we need
[ ] Make sure the CLI works
[ ] Add a Jenkinsfile and new job to download, generate mappings and upload

We should maybe use 'gene-mappings' as the directory in the buckets, with a dated directory and a latest in each one.

I'm not sure about upload locations, definitely gs://data-public-monarchinitiative and gs://monarch-archive. I'm not sure if it makes sense to upload to gs://monarch-ingest.

We could also consider writing to date-named GitHub releases... (and then we can bring that over to monarch-ingest)

matentzn commented 2 years ago

Can we figure out if the mapping tables are all <50MB in size? would be nice if we could use GitHub for versioning...

glass-ships commented 2 years ago

Ahh, GitHub file size limit! I'd been working with GitLab for so long I'd forgotten!

I'm not sure about all of them, but the version I just produced in testing is about 30MB, so for the time being this should be ok. What do you think @kevinschaper ?

kevinschaper commented 2 years ago

If it gets too big we can definitely break into multiple files.

We could experiment with doing it all with github even and skipping the Jenkins completely?

matentzn commented 2 years ago

I think having as much as possible done with GitHub and GitHub actions increases transparency a lot!