Start DipperCache and Go, Wormbase & Reactome

[TomConlin]

DipperCache is not officially deployed yet but we can start using the preview version sitting on archive.mi/DipperCache.

• Source.py

  • uses the ingest name and a file name to try to fetch from the cache before falling back to normal behaviour of fetching from self.files[src_key]['url']
  • Now checks http result codes for failed requests, expect pots to be stirred.
  • May be slightly less wonky checking http HEAD requests for last modified headers.
  • Removed a function previously needed only by Go & Reactome for gaf-eco mapping

• Go & Reactome got minimal refactors to pull a a preprocessed gaf-eco mapping file from Dipper cache.

• Wormbase needed a more extensive minimal refactor

• test and translation tables are adjusted as needed

Away at CCDH meeting (at which KS said the above was okay to merge)

---

Additional item

• increase efficiency of the QC script that reinterprets RDF as Graphviz only I use.
  • considerable speed-up from treating uri as sequence of tokens instead of single string.
  • a few base-uri in cuire maps & ingests are shifted to token boundary’s
  • additional curie prefixes for source items passing through dipper are included
  • do a better job of surfacing URI w/o curie-prefixes

• Mouse ingests have the greatest variety of labs/urls and need most fixes

  • adjusting them led to light re-factoring (they still need more)

• Wormbase ingest had an extra stutter in
  species/c_elegans/PRJNA13758/c_elegans.PRJNA13758.WS273.annotations.gff3.gz

[justaddcoffee]

Both worm and reactome ingests with DipperCache work great on my end. Haven't tested GO yet as it requires a big download (idmapping_selected.tab.gz) that I don't have locally ATM.

One question I have is, does the code to actually populate DipperCache with the files live in here somewhere? Can't find it in this PR, although maybe I'm missing it. If it's not here, maybe that should go in this repo somewhere?

[TomConlin]

The code populating the cache is in a repo temporary still owned by me. Goal is to come up with a deployment process then transfer it to Monarch The deploy is in a bit of limbo while Monarch VMs have network issues.

I am keen on hearing thoughts on how to robustly deploy a purely static site in this day and age.

the repo you are wondering about is currently https://github.com/TomConlin/DipperCache

[justaddcoffee]

the repo you are wondering about is currently https://github.com/TomConlin/DipperCache

Possibly could move that code into Dipper? Seems like it belongs here, since it is going to become part of the 'E' in our ETL pipeline. (Otherwise, we'll need to add/update URLs for file downloads in both the Dipper repo and this other repo.) Just a thought

[TomConlin]

@justaddcoffee entirely possible. Won't be determined till we see how it is actually deployed.