Explore/Reconsider design choices of the special remote

[adswa]

[Extending this comment as I explore the code more]

Without the original authors, the backstory of the special remote isn't fully clear, but the - at many points very convoluted/clunky - implementation suggests that there were a number of limitations on Zenodos side that were getting worked around when originally implementing it.

Some of these decisions make the special remote feel "manual" and also contribute to a very slow performance. A conscious re-evaluation could spark reimplementations that change the special remote's functionality but also standardize it more and improve it.

What I'm aware about:

• Zenodo seems to only support flat hierarchies of files. This prohibits an out-of-the-box ExportRemote. Either it would need to be a plain SpecialRemote, or manual path wrangling to represent and reassemble hierarchies
• Zenodo deposits allow only 100 files. Not sure if the special remote can do anything about this really other than documentation. We noticed because a part of git annex' testremote function tests chunking, and that exceeded the amount of files that could be uploaded - So repos with 100+ files or a small chunk size and chunking enabled someone could easily run into this.
• Once a draft has been published, its files can't be modified, added to, or removed. For additional files or modifications, the special remote creates a new version as a draft, which is in my opinion useful behavior. This doesn't happen for remove (i.e., dropping files), however. git annex drop myfile.txt -f zenodo would fail. One could debate whether this should stay like this, or whether the special remote should in this case also create a new draft and remove the file in question from it - if so, the special remote would need to store the version of the published deposit with any key.
• There are different Zenodo APIs: https://developers.zenodo.org documents one, but it also contains a reference to "the new files API":

Now, let’s upload a new file. We have recently released a new API, which is significantly more perfomant and supports much larger file sizes. While the older API supports 100MB per file, the new one has a limit of 50GB total in the record (and any given file), and up to 100 files in the record.

bucket_url = r.json()["links"]["bucket"]

To use the new files API we will do a PUT request to the bucket link. The bucket is a folder-like object storing the files of our record. Our bucket URL will look like this: https://zenodo.org/api/files/568377dd-daf8-4235-85e1-a56011ad454b and can be found under the links key in our records metadata.

The new files API isn't really documented much beyond this as far as I have seen, but the choice of API probably led to the high amount of metadata parsing in the current implementation

• Zenodo supports adding other users to drafts and giving them permissions to manage the draft, but the draft is only under the original creator's "depositions" resource. The API endpoint would fail for other users, even when they are given access permissions. The API documentation doesn't mention where or how co-authors could access the relevant deposit. This seems to prevent any multi-user use case other than consumption.

• Zenodo records can be public or restricted. When they are restricted, only zenodo users added via the webinterface have access. Using the minimal implementation from #2, taking the token from the environment rather than reusing/impersonating the original creator, this makes get operations fail for anyone unauthorized in this way, but also fail for those authorized that are not the original creator. May be a similar issue with undocumented API endpoints, and would need additional mangling in the special remote to truly support restricted records in a more-than-one-user-consumption-use-case.

[jkuhl-uni]

Yes, this is true. I also thought about this. For my use-case, this is a fairly easy fix, as I can re-package results into JSON-files, which are small enough (and not too numerous) to fit in the limitations. I didn't think about chunking. As I do not have experience with zenodo, I am not sure how welcoming they are to lift those restrictions for single projects.