duplicates

[myrmoteras]

@gsautter we need to remove duplicates

https://ocellus.punkish.org/?q=oskoron&page=1

how can we do this? d

[gsautter]

This looks like a hard one, at least if the "rec ID"s correspond to the deposition numbers. Hard to tell how these images got uploaded twice, especially as the deposition numbers are stored in the corresponding captions right after the upload. There are three things I could imagine going wrong in this regard:

• The saving operation failing due to a rare IO problem (which we recently had when the file allocation table ran full), resulting in the first round of deposition IDs not getting persisted on our IMF end, and another deposition being created for each figure image.
• The respective IMF getting uploaded twice, with the second upload overwriting / removing the deposition IDs from the captions, and triggering a second upload of the figures.
• Having uploaded IMFs for two different versions of the same PDF, say, one with watermarks ("downloaded from", etc.) in it and one without.

All three scenarios are fairly pathological, if not completely contrived. We might well try and catch such duplicate figures by means of a hash of the PNG data on our end. Another little improvement we need to add, I guess.

Do we have an idea of how many such duplicates there are? That would vastly help assess the extent of the problem, as well as its underlying cause.