Media of draft items is deleted automatically #164

Closed dpancic closed 1 year ago

In GitLab by @KlausIllmayer on Mar 31, 2022, 17:22

When creating a draft of an item, e. g. a tool, and applying a media, e. g. an image, and then waiting a day without publishing it, the media will be deleted by the MediaStorageGarbageCollector. Afterwards it is not possible anymore to access the draft, as it raises the error javax.persistence.EntityNotFoundException: Media with id ----here the id---- not found.

It is expected, that the MediaStorageGarbageCollector respects that a media is attached to an draft and does not delete the media.

@tparkola could you change the behaviour of the MediaStorageGarbageCollector so that it does not ignore drafts?

I can reproduce this error. I used my personal login with the role moderator on stage, create a draft with an image and then wait until the next day. It is after 24h (I'm not super sure if it is really 24h or some hours more or less) deleted and I can't access aynmore the draft item. Here the logs:

I see that the media is deleted:

29-03-2022 14:53:20.978 [scheduling-1] DEBUG e.s.m.d.m.MediaStorageGarbageCollector.lambda$deleteStaleMedia$2 - e7e5eb5f-4fe3-4423-bdd2-46354363da1d

and if I afterwards try to access the draft item, I get:

31-03-2022 15:05:09.110 [http-nio-8080-exec-8] ERROR e.s.m.c.MarketplaceExceptionHandler.handleEntityNotFoundException - No entity
javax.persistence.EntityNotFoundException: Media with id e7e5eb5f-4fe3-4423-bdd2-46354363da1d not found
    at eu.sshopencloud.marketplace.domain.media.MediaStorageService.lambda$loadMediaData$3(MediaStorageService.java:272)
    at java.base/java.util.Optional.orElseThrow(Unknown Source)
    at eu.sshopencloud.marketplace.domain.media.MediaStorageService.loadMediaData(MediaStorageService.java:272)
    at eu.sshopencloud.marketplace.domain.media.MediaStorageService.getMediaDetails(MediaStorageService.java:95)
    at eu.sshopencloud.marketplace.domain.media.MediaStorageService$$FastClassBySpringCGLIB$$7d6fbd5a.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
...

@tparkola There are already some contributors that reported back, that they got this error. Do you see any chance that I can save the draft item by deleting the reference to the image in the dabase? I would delete the entry in the items_media-table with the media-id that raises the not found error: do you think this is safe or will I need to delete something else?

Adding @laureD19 and @egray523 so that they are up to date about the progress.

I guess we can also express that this is an error that involves already some contributors, therefore finding a solution should get a high priority.

In GitLab by @KlausIllmayer on Mar 31, 2022, 18:32

@tparkola It seems, that this issue now should have urgent priority. I looked into the logfiles a little bit and deleting media by the MediaStorageGarbageCollector happens very often. It seems, that it does not have a cool-down-phase of 24h but instead immediately when the MediaStorageGarbageCollector is called which happens quite often (not sure how often this happens but at least more than once in an hour).

We also have now an item on production, that somehow managed to be published with this corrupt state. I still wonder how it happend but I have a theory:

An already published item is edited by a moderator and a new media file is attached
The item is for reasons of safety stored as a draft, creating a new version with status draft
The MediaStorageGarbageCollector runs, finds the media and deletes it, because it is only attached to the draft version
The item - which is still open in the frontend edit form - is published, thus setting the draft version to the now approved version (as the person editing the item has the role moderator)
We now have an item that is published with an media-id that is not existing anymore

Otherwise I can't explain why we have such an item on production raising the missing media error and additionally breaking currently this important API call: https://marketplace-api.sshopencloud.eu/api/tools-services/?page=5&perpage=20

But that would also imply, that approving an item does not check if the media-id is still available.

At least it is now very important - as production now involved - to find out, if deleting the entry in the items-media-table can solve this problem.

In GitLab by @KlausIllmayer on Apr 5, 2022, 14:44

Another observation: strangeley, the MediaStorageGarbageCollector does not delete the media immediately as I would have expected. The MediaStorageGarbageCollector runs every 30 minutes, but in my last experiment it deleted the media only after 2 hours (and I saw in the logs that between creating the draft including an image and deleting this image, the collector was called three times without touching the image, only the fourth time was then a deleteStaleMedia).

In GitLab by @tparkola on Apr 5, 2022, 14:52

Yes, because retention time is set to 2h. Yesterday I tried to recreate the error on local backend, but failed to do so. Tomorrow I will try again.

In GitLab by @KlausIllmayer on Apr 6, 2022, 10:01

just for the record: as a workaround i deleted the corrupt entries to the no longer existing media from the items_media-table (identified by having no correspondending media_id-item in 'media_data'). it looks like this worked well (on production we had 12 such rows), afterwards the items become accessible again (of course without the once added media). i hope there is no side-effect, based on this direct intervention in the database.

In GitLab by @KlausIllmayer on Apr 6, 2022, 13:33

I've tested a lot of use cases that come into my mind regarding the currently not reproducible case of published items (not draft ones, where we can reconstruct the problem) holding broken media-relations. But I struggle to locate how and why it happens. The five cases that appeared until now all occured on production and all were approved/published by one moderator (but they are not always new items, they were also two cases where it was an enrichment of ingested items). The moderator used the "save as draft" button many times to store the progress in filling out the data of an item. I guess that this has something to do with the observed problem. There must be a special situation where the draft items, the published item and the MediaStorageGarbage Collector somehow affect each other. If this is true, I strongly suppose that solving the problem with the draft items media problem should also solve the problem with the published items media problem. At least this is my current theory and only if the problem should occur again after implementing a bugfix for this incident, I will again look into the problem.

In GitLab by @tparkola on May 13, 2022, 08:26

The issue has been solved. Media should be saved after second draft save or publishing with changes in draft.

In GitLab by @laureD19 on Jul 22, 2022, 12:25

implemented and tested on stage:

create new item as contributor
add an image
save as draft
accessing the draft and editing it
submitting it

In GitLab by @laureD19 on Jul 22, 2022, 12:25

changed the incident status to Resolved by closing the incident

SSHOC / sshoc-marketplace-backend

Media of draft items is deleted automatically #164