monarch-initiative / monarch-ingest

Data ingest application for Monarch Initiative knowledge graph using Koza
https://monarchinitiative.org
15 stars 2 forks source link

try copying to kghub S3 as well #426

Closed glass-ships closed 1 year ago

glass-ships commented 1 year ago

I think this may be ready to pull in if indexing is manual on the kgx side?

@caufieldjh can you remind me where we landed on this?

caufieldjh commented 1 year ago

I think this may be ready to pull in if indexing is manual on the kgx side?

@caufieldjh can you remind me where we landed on this?

I can either do it manually or you can import + run the multi-indexer as part of the upload. Works well if you create local file structure like what you want to see on the bucket, run the multi-indexer, then upload everything, including the index.html's. This is what we're doing for KG-IDG, for example: https://github.com/Knowledge-Graph-Hub/kg-idg/blob/43be62ff4880f098a4b40b2411095d1c9c76b00e/Jenkinsfile#L218

caufieldjh commented 1 year ago

Maybe that should just be part of kgx anyway...but for now, it isn't

glass-ships commented 1 year ago

I think this may be ready to pull in if indexing is manual on the kgx side? @caufieldjh can you remind me where we landed on this?

I can either do it manually or you can import + run the multi-indexer as part of the upload. Works well if you create local file structure like what you want to see on the bucket, run the multi-indexer, then upload everything, including the index.html's. This is what we're doing for KG-IDG, for example: https://github.com/Knowledge-Graph-Hub/kg-idg/blob/43be62ff4880f098a4b40b2411095d1c9c76b00e/Jenkinsfile#L218

Oh nice, ok I'll add something like this in! Does the indexing step change at all if I'm updating current/ as well as adding a dated release? (or is it sufficient to index once then copy to both)

caufieldjh commented 1 year ago

It's the same - you can put identical copies of the same files, including the generated indexes, in both the dated directory and the current directory.