The outdated fragment relation should be removed

[xdxxxdx]

Is your feature request related to a problem? Please describe. When I am executing the E2E test for the retention mechanism 6.basic-retention, I found out the relation that pointing to the soft/hard deleted fragment is not removed with the fragment soft/hard delete

Describe the solution you'd like As discussed in the slack , the relation should be removed after the fragment is deleted.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context I can see when trying to follow the (phantom) relation, there is a notification: Fragment with the following identifier has been deleted: http://localhost:8080/by-short-time?generatedAtTime=2023-02-20T18:21:52.357Z

Note: In case the LDES fragment is kept in the cache, the LDES client should also need to be able to handle the 404 error. (Not the case see (https://github.com/Informatievlaanderen/VSDS-Linked-Data-Interactions/issues/28))

[WLefever-Cegeka]

Hi @xdxxxdx , The current retention mechanism is something the team put in place to limit the amount of member data in mongoDB. This was necessary for IoW, because without this mechanism the cloud cost of VMM for storage increased significantly. This current mechanism works on TreeNodes, i.e. it will mark an entire TreeNode as 'deleted' when some duration has passed (time-based retention). A while ago I made a ticket to revise the entire retention component of the server, because the spec describes retention rather on the level of individual TreeMembers. In my point of view it would be better to for example return a 410 with partial data (event stream info, relations, non-retained tree members), when the retention policies of certain tree members have been satisfied. This would allow for an implementation closer to the spec and would avoid tampering with the relations. I described this idea in this ticket: https://vlaamseoverheid.atlassian.net/browse/VSDSPUB-422 In the road map of VSDS full-blown retention/archiving is foreseen for release 3 (https://vlaamseoverheid.atlassian.net/wiki/spaces/VSDSPUB/pages/6077583859/Roadmap#Retention/Archiving). So I'd be good to get together when this gets closer to development.

[xdxxxdx]

Hello @WLefever-Cegeka,

Thanks so much for the clarification. I have some questions:

First, regarding when requesting the fragment => return 410. GONE. I think this also requires removing the relation pointing to this GONE page?

For the no Error message solution, does this mean we gonna have some fragments with empty member inside?

Or is it a combined solution?

-Request empty fragment=> 410 GONE message + delete the phantom relation -Request not empty but not full fragment (marked) => Partial data + No error return.

Thanks XD

[WLefever-Cegeka]

Hi @xdxxxdx , Retention has been revised. Current implementation will never remove fragments/make fragments unreachable. So no more 410 and no more updating of relations. Members are removed from views if they match all retention policies and removed from collections if they match all retention policies from all views. Kind regards, Wouter