Issues with Zenodo are preventing release completion [replacement issue]

[kltm]

While we have managed to create a correct product that is locked, issues with the archival upload and DOI generation at Zenodo are preventing us from completing the August release. I have retried over half a dozen times over several days. Errors look like:

INFO:zenodo-version-update:next command as curl: curl -X GET -H "Accept: application/json" -H "Content-Type: application/json" -H "Authorization: Bearer XXX" https://zenodo.org/api/deposit/depositions/7033769
    main()
  File "./scripts/zenodo-version-update.py", line 401, in main
    retpay = json.loads(decoded_body)
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

real    135m12.170s

We last ran into this issue on 2020-11-19; I'm trying to figure out what the cause was from context and notes. In the past, on an occurrence on 2020-03-25, it appears to have started working again after some time and several retries.

Tagging @pgaudet @dustine32

(Noting replacing previous instance of this issue, #298)

[kltm]

From @dustine32:

@kltm It might help if we could look at the values of these response variables: https://github.com/geneontology/go-site/blob/b3c282fbfa3cf1e5b8d980daa77fa89a36afd2ee/scripts/zenodo-version-update.py#L399-L400

    body = buffer.getvalue()
    decoded_body = body.decode('iso-8859-1')

Obviously, these values aren't in a nice, parsable JSON format but hopefully it's at least some helpful error message.

@kltm Do you think it's worth it for me to PR some LOG.info() lines for these to zenodo-version-update.py in go-site and you should(?) be able to restart the Archive step in Jenkins?

[kltm]

@dustine32 While trying to capture the error is worthwhile, we'll likely need to manually run the curl step to get to the bottom of thing (IIRC). I'm unlikely to be able to get to that until my return next week. I'm hoping that this is just Zenodo being weird and them having trouble with AWS (timeout, VPN, whatever); I'm afraid that we've gone over the size limit again. It may be worthwhile to see if there was any significant growth between this release attempt and the last successful release.

[kltm]

Finally passed one. For like no reason. I don't understand.