search

Informatievlaanderen / VSDS-LDESServer4J

Linked Data Event Streams Server for Java
https://informatievlaanderen.github.io/VSDS-LDESServer4J
European Union Public License 1.2
10 stars 11 forks source link

Ingestion during refragmentation #583

Closed WLefever-Cegeka closed 1 year ago

WLefever-Cegeka commented 1 year ago

When we add a view and refragment the existing members (a process that might take a while), there's a possibility that new members for that collection (and thus for the view) arrive. However, these will not be fragmented since refragmentation is still ongoing and the views are only updated after refragmentation.

Example:

  1. Collection A has 10 members
  2. We add a new view to Collection A
  3. The refragmentation process is started and members 1 until 7 are fragmented in the new view
  4. Member 11 is ingested, but not fragmented (nor queued for fragmentation)
  5. the refragmentation process fragments members 8, 9 and 10
  6. the new fragmentation process of the new view is added to the fragmentation process
  7. Member 12 is ingested and fragmented according to the new view

The new view will not have member 11. The issue arises between step 3 and step 5.

Tomvbe commented 1 year ago

Fixed in https://github.com/Informatievlaanderen/VSDS-LDESServer4J/pull/861 . Fragmentizers now follow the sequenceNr of the members to determine order of fragmenting. New members being ingested during refragmentation will always receive an incremental sequence and won't be ingested too soon.