Informatievlaanderen / VSDS-LDESServer4J

Linked Data Event Streams Server for Java
https://informatievlaanderen.github.io/VSDS-LDESServer4J
European Union Public License 1.2
10 stars 10 forks source link

Version-based retention of 1 contains multiple versions per entity #900

Closed brechtvdv closed 11 months ago

brechtvdv commented 1 year ago

I want to create a LatestVersionSubset with amount: 1. However, multiple versions of the same non-versioned entity are still maintained.

I created this view:

curl -X POST ‘http://ldes-server:8080/admin/api/v1/eventstreams/touristattractions/views’
      --header ‘Accept: */*’
      --header ‘Content-Type: text/turtle’
      --data-raw ‘@prefix ldes: <https://w3id.org/ldes#> .
      @prefix ldes: <https://w3id.org/ldes#> .
      @prefix tree: <https://w3id.org/tree#>.
      @prefix example: <http://example.org/> .
      @prefix xsd:  <http://www.w3.org/2001/XMLSchema#> .
      @prefix touristattractions: <http://localhost:8080/touristattractions/> .
      @prefix latestView: <http://localhost:8080/touristattractions/latestView> .
      latestView:description a <https://w3id.org/tree#ViewDescription> ;
        tree:fragmentationStrategy
             ([ a tree:PaginationFragmentation ;
                tree:bidirectionalRelations  “false” ;
                tree:memberLimit 100
             ]) ;
        ldes:retentionPolicy [
          a ldes:LatestVersionSubset;
          ldes:amount 1 ;
        ] .
      touristattractions:latestView tree:viewDescription latestView:description .’

When counting the number of instances per non-versioned entity, I get > 1 counts:

PREFIX dcterms: <http://purl.org/dc/terms/>
PREFIX dc: <http://purl.org/dc/elements/1.1/>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX prov: <http://www.w3.org/ns/prov#>
select ?ivo (COUNT(?ivo) AS ?c)
where {
  ?test a ?klasse ;
        prov:generatedAtTime ?gat ;
        dcterms:isVersionOf ?ivo .
}
GROUP BY ?ivo

The result:

https://westtoer.be/id/product/d740ecf8-887c-4c44-adbf-f230a9eecffb
?c
“1”^^http://www.w3.org/2001/XMLSchema#integer
?ivo
https://westtoer.be/id/product/5498dfed-9f67-4aff-b168-10505bec7a83
?c
“1"^^http://www.w3.org/2001/XMLSchema#integer
?ivo
https://westtoer.be/id/product/bea59da5-35ac-4c2f-9a94-3f6b37029a37
?c
“2”^^http://www.w3.org/2001/XMLSchema#integer
?ivo
https://westtoer.be/id/product/380f30f1-0163-4a50-bdfb-d8cf0022e488
?c
“1"^^http://www.w3.org/2001/XMLSchema#integer
?ivo
https://westtoer.be/id/product/c2d61e21-3665-4bf6-9ba5-9b5e5b29af86
?c
“1”^^http://www.w3.org/2001/XMLSchema#integer
?ivo
https://westtoer.be/id/product/a1735883-6271-43db-b97c-b4571122ac89
?c
“2"^^http://www.w3.org/2001/XMLSchema#integer
Tomvbe commented 1 year ago

Hi @brechtvdv this is likely due to one of two reasons.

1. Default view prevents deletion of members A known issue in the server is that a default view is created for every eventstream. This means that members are added to your custom view with version based retention AND the default view, which has no retention. This default view then blocks the removal of all members. You can check this with:

curl -X 'GET' \
  'http://localhost:8080/admin/api/v1/eventstreams/touristattractions/views' \
  -H 'accept: text/turtle'

If the default view is there, you have to remove it first. We removed the default view functionality just this week as part of https://github.com/Informatievlaanderen/VSDS-LDESServer4J/issues/893. However if you are not using the latest develop image, you will have to delete the default view manually after creating an eventstream.

2. Missing versionOf or timestamp property Can you also share the ttl you use to create the eventstream and a ttl of one of the members? Version based retention requires both a timestamp and a versionOf. This is configured on the eventstream and needs to be present on the members to be eligible for deletion.