search

monarch-initiative / monarch-ingest

Data ingest application for Monarch Initiative knowledge graph using Koza
https://monarchinitiative.org
14 stars 1 forks source link

Automate public data bucket file index html generation #312

Closed kevinschaper closed 2 years ago

kevinschaper commented 2 years ago

We need to run the directory indexer script from https://github.com/monarch-initiative/monarch-file-server/tree/main/scripts after we add files to the public data bucket.

It might be enough to update our Jenkinsfile to check out the file server repo, and run: 

mkdir /tmp/mnt/ || true
gcsfuse --implicit-dirs data-public-monarchinitiative /tmp/mnt

python3.8 -m venv venv
source venv/bin/activate
python3 ./scripts/directory_indexer.py -v --inject ./scripts/directory-index-template.html --directory /tmp/mnt --prefix https://data.monarchinitiative.org -x
glass-ships commented 2 years ago

Hmm, I'm not sure I totally follow.

Would this step need to happen at a specific stage in the Jenkins pipeline? Or could it just go at the end after the "release" stage

kevinschaper commented 2 years ago

Yeah, I think that's about all we can do. It would be nice to encapsulate this better into a python library, but for now it's probably enough to just check it out and run it after the release step to distribute the files to more places. I suppose we could put it into a shell script so that it's easier to try on its own?