Can no longer upload to Zenodo; due to recent Zenodo update?

[kltm]

Zenodo has apparently changed the behind-the-scenes AWS bucket API with their more recent update. (This is the one they showed us back when we were having trouble uploding large files.)

Get bucket info using script? [Y/n]
Traceback (most recent call last):
  File "./scripts/zenodo-version-update.py", line 512, in <module>
    main()
  File "./scripts/zenodo-version-update.py", line 320, in main
    for filedoc in response.json()['contents']:
KeyError: 'contents'

With this, our upload script no longer works. https://help.zenodo.org/docs/about/whats-changed/ gives no indication of this change, but it was possibly a not-for-public "hack". Hopefully this is now integrated. We shall see.

The fix here is, in order of ease and likelihood:

• Hope this is temporary
• repair script to fit available API (i.e. "old" API)
• figure out how to manually complete upload
• find non-Zenodo location for data (likely Data Dryad)

[kltm]

I'm actually a little confused--the API docs (https://developers.zenodo.org/?shell#quickstart-upload) seem so far to indicate that things have not changed. Walking through the commands curl by curl, I'm not seeing where things have gone wrong or changed (yet).

Similar to my issue is https://github.com/zenodo/zenodo/issues/2493; I'm actually thinking it may make sense to give this a moment as it may actually be an upstream issue.

[kltm]

Grossly, the steps we need to execute are:

1. create new version
2. examine new version to get files in it
3. remove files found in new version
4. add files to new version
5. publish

We seem to be getting stuck now on "2". This stopped working for us between 2023-10-09 and 2023-11-16.

Using the sandbox server, when we try and probe, using the new file API, for the deposition contents of the recently created version, we get:

curl -X GET -H "Accept: application/json" -H "Content-Type: application/json" -H "Authorization: Bearer <TOKEN>" https://sandbox.zenodo.org/api/files/XXX-YYY-ZZZ-AAA-BBB

{
    "error_id": "381670a8ddd247ceaef6a05dbfc1fac4",
    "message": "The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.",
    "status": 500
}

[kltm]

Opened issue at Zenodo: https://github.com/zenodo/zenodo/issues/2506

[kltm]

Okay, I have managed to get a 47GB upload to work locally in default FF after many many partials. I'm now looking at making a draft release through the web interface. Tagging @pgaudet on progress.

[kltm]

@pgaudet Bumping this over to stabilization project for now, as it essentially blocks #353 ; if I can't fix, we should move off Zenodo

[kltm]

Can confirm now this is ongoing.