bbcarchdev / spindle

RES Linked Open Data aggregation engine
https://bbcarchdev.github.io/spindle/
Apache License 2.0
2 stars 1 forks source link

Correlate: Updating the spindle queue state needs to happen AFTER sparql-put #126

Open nevali opened 6 years ago

nevali commented 6 years ago

There's a race condition in the ingest pipeline which means that spindle-generate may be invoked before updated graphs are available in the quad-store or cache.

Unfortunately, fixing this is likely going to involve splitting spindle-correlate into two steps: one which performs correlation, and another which updates the index state to mark items as dirty, i.e.:

Alternatively, it may be possible to perform a sparql-put immediately after the sparql-get process has retrieved the previous state of the graph.