minetest / contentdb

A content database for Minetest mods, games, and more
https://content.minetest.net
GNU Affero General Public License v3.0
93 stars 45 forks source link

API should provide source file hashes for file-like objects #493

Open doxygen-spammer opened 7 months ago

doxygen-spammer commented 7 months ago

Problem

It is difficult to automatically determine whether the package on ContentDB is up-to-date. Releases and screenshots are file-like objects that can be updated through the API, and easily downloaded. A release script needs to know whether a new version of these objects needs to be uploaded.

There are 3 types of information available for each object:

  1. The downloaded file does not necessarily match the uploaded file. Screenshots might be transcoded, releases might be rearchived or have additional metadata filled in.
  2. The object’s ID does not help, because it can only be compared against a persistent storage of IDs. This is inconvenient, and often impossible.
  3. The title can be set to a deterministic value, and you can reasonably assume that a release titled “v1.0.0” is outdated when you are releasing “v2.0.0”. But it doesn’t help much for screenshots, since updating a screenshot usually does not change its purpose, and therefore the title stays the same.
  4. (Only for releases) The commit hash is not unique if multiple packages share a repository. Additionally, adding commits to the repository does not necessarily change the release file.

Solutions

When a file-like object is created or updated, a hash of the uploaded file should be stored, and provided in the API. Whether this hash is Fletcher-32 or SHA-3xyz does not really matter, unless you are trying to attack your own releases. ;) I prefer SHA-1 for simplicity.

Suggested API documentation:

Alternatives

Update releases on ContentDB manually, investing time of impatient mod authors (me).

Additional context

Yes, I am finally working on a new tool that automates release uploads. It will be the third, and most complex-flexible, method to upload releases. (Unlike git update detection, it shall update description and screenshots too.)

I think the API is already quite good, I encounter only minor obstacles like this one. :)