Rename products/annotations to products/upstream_and_raw_data to help avoid user confusion

[kltm]

We sometimes field questions and have internal confusion with the products/annotations directory

In discussion with @pgaudet , we want to rename this to something a little less confusing, like products/upstream_and_raw_data.

TODO:

• [x] on friday, allow snapshot to start again
• [x] check that old S3 paths are updated/removed: current and snapshot

[kltm]

~When done, must also update: https://wiki.geneontology.org/Release_Pipeline#products~ Now https://github.com/geneontology/pipeline/issues/320

[kltm]

This looks good on master; all other main pipelines should have this now. Moving to clearing until next release.

[kltm]

@pgaudet @vanaukenk I'm realizing that we probably actually want to announce this pretty broadly and as soon as possible. This will be a breaking change. Even though we generally tell people not to pull stuff from there, people in fact do for various (good) reasons. I'm specifically thinking of @alexsign and any other downstream consumer of the /products/annotations/* files.

[kltm]

From the software call, for the moment, we're punting breaking things until the weekend by stopping snapshot. @vanaukenk @pgaudet We can either let snapshot go over the weekend, assuming we make an announcement tomorrow, or we should probably halt snapshot until at least the next annotation call (then running on the 22nd).

[vanaukenk]

I've put this item on the annotation call agenda for 2023-03-21. I checked this morning and this indeed would have broken the WB script that retrieves our Noctua annotations.

[kltm]

@pgaudet @vanaukenk Okay, I'm going to stall out snapshot until this is resolved/made public. Hopefully early next week to give everybody a chance to fix and have a breakage before the next release.

[pgaudet]

@kltm Where was this announced? Should we email go-friends?

[kltm]

@pgaudet This has not been widely announced, that's why I'm slow-walking releases for the moment. Emailing go-friends would be a good start.

[ukemi]

ping @loricorbani Lori, we will need to change our script so that we are picking up the Noctua annotations from the renamed directory.

[vanaukenk]

Plan:

• [x] email to go-friends announcing the change (@suzialeksander )
• [ ] for the next successful run starting on March 26th (snapshot, current, and release) will change to the new directory name; change is not retroactive

[leemdi]

Are we going to test this first? That is, have both running so we can make our switch and test?

from: http://snapshot.geneontology.org/products/annotations/noctua_mgi.gaf.gz http://snapshot.geneontology.org/products/annotations/mgi-prediction.gaf

to: ??

Thanks. Lori

[kltm]

@loricorbani The new paths would be: http://snapshot.geneontology.org/products/upstream_and_raw_data/noctua_mgi.gaf.gz http://snapshot.geneontology.org/products/upstream_and_raw_data/mgi-prediction.gaf That said, due to data archiving and the fact that our published URLs are not dynamic, there unfortunately can't be any testing that isn't the final thing--copies at both paths existing at the same time isn't something that we want to publish (and would likely cause even more confusion if we did).

[leemdi]

great. many thanks. we will turn off our go-load this weekend; test our changes on Monday; turn go-loads back on for Tuesday morning.

[leemdi]

http://snapshot.geneontology.org/products/upstream_and_raw_data/noctua_mgi.gaf.gz http://snapshot.geneontology.org/products/upstream_and_raw_data/mgi-prediction.gaf

I was unable to download these files this morning, Monday, 03/27. I though this change was going live today?

[leemdi]

also, I don't think I am on this email : go-friends Please add me to this email. Thanks.

[kltm]

The changes in the pipeline code went live last night. Once a pipeline run completes, the path will change. In our current setup, the snapshot is a live archive (not staged), so this cannot be 100% timed ahead. This should go live very soon.

[kltm]

@loricorbani Mailing list info is here: https://mailman.stanford.edu/mailman/listinfo/go-friends (from http://help.geneontology.org/).

[kltm]

@loricorbani I'm not sure where your message went, but the snapshot run is currently underway and, if there are no interruptions, would be complete in about 24hrs.

[leemdi]

i removed my comment because I didn't see your comments until I refreshed this page. i'll keep checking...i didn't realize there would be a delay. I saw Monday March 27 and assumed that meant the new folder would be available on the same day. i'm not too familiar with your pipeline and when things get added to the folder. in any case, i'll continue to check in the morning so we can turn our MGI GO pipeline back on.

[suzialeksander]

@loricorbani I just added you to the GO-friends mailing list, using the jax email tied to your GH account.

[leemdi]

@suzialeksander thanks!

[kltm]

Ran into a failure with PAINT files during snapshot run, looking into it.

[kltm]

Fixed applied; rerunning.

[kltm]

release clearing and snaphot cleared; next: removing old paths.

[kltm]

Okay, the old path has been removed. This should now be correct and final on snapshot, current, and release (from today).