Drop Solr (GOlr) indices from Zenodo archive bundle

geneontology / pipeline

Declarative pipeline for the Gene Ontology.

https://build.geneontology.org/job/geneontology/job/pipeline/

BSD 3-Clause "New" or "Revised" License

5 stars 5 forks source link

Drop Solr (GOlr) indices from Zenodo archive bundle #353

Closed kltm closed 2 months ago

kltm commented 5 months ago

Drop Solr indices from the Zenodo archive bundle, to:

make the final stages a little more "stable"
help deal with recent Zenodo changes https://github.com/geneontology/pipeline/issues/345

An alternate I want to consider is silently "splitting", maybe asynchronously or after-the-fact, giving us an archive w/o slowing things down.

kltm commented 5 months ago

As I collect progress report items, I def think we want this archived--I do not want to be caught needing to regenerate.

kltm commented 5 months ago

Noting that this is a bit of a "non-starter" given that Zenodo doesn't really work anyway with https://github.com/geneontology/pipeline/issues/345

kltm commented 3 months ago

Glarf--the issue may be forced on this release, as recent additions have likely pushed us over the 50GB limit.

51532257933 Apr  5 14:06 go-release-archive.tgz

48G Apr  5 14:06 go-release-archive.tgz

Well, showdown between GB and GiB...

kltm commented 3 months ago

I can confirm we can no longer proceed with current package:

Warning!
Could not upload files.
Uploading the selected files would result in 51.53 GBbut the limit is50.00 GB.

kltm commented 3 months ago

Okay, rebuilding package without these files. I want to look at saving these separately in Zenodo.

kltm commented 2 months ago

Okay, I now have an experimental deposition here: https://zenodo.org/records/10946933, that contains the solr index and the two blazegraph builds. However, it has a "twin" in the dashboard that is marked for "Review". I don't know what that means. I'll check back on it tomorrow. If it looks good:

[x] add them to the release notes
[ ] add this process to the release SOP (sadly manual at this point)
[ ] (add to end-use docs?)

kltm commented 2 months ago

Zenodo itself seems to be broken at this point--any attempt to make new versions or edit the metadata of the new deposit is met with a vague server error.

kltm commented 2 months ago

Checking again, I seem to have the ability to update the "binaries" package. I think this can be closed out by updating my SOP for the current "semi-manual" releases.