search

AtomGraph / LinkedDataHub

The low-code Knowledge Graph application platform. Apache license.
https://atomgraph.github.io/LinkedDataHub/
Apache License 2.0
493 stars 124 forks source link

Imports can cause concurrent write requests to the triplestore #47

Closed namedgraph closed 3 years ago

namedgraph commented 3 years ago

LinkedDataHub has no problem sending concurrent streams of outbound RDF data, but it turns out the Graph Store Protocol fails on concurrent write (POST, PUT, DELETE) requests. At least Dydra fails, Fuseki needs to be further tested.

The solution would be to queue outbound requests in ImportListener. Possibly as easy as turning down the number of Executor threads to 1.

namedgraph commented 3 years ago

ImportListener queues GSP requests to the triplestore -- Fuseki does so by default, Dydra using the asynchronous mode.

However, applications should take case to create queries and upload files before starting the (potentially long-running) imports, because the queued requests shouldn't interleave with normal Linked Data requests. Demo app scripts have been refactored accordingly: https://github.com/AtomGraph/LinkedDataHub-Apps/blob/master/demo/northwind-traders/import-csv.sh#L111