eclipse-rdf4j / rdf4j

Eclipse RDF4J: scalable RDF for Java
https://rdf4j.org/
BSD 3-Clause "New" or "Revised" License
367 stars 164 forks source link

GH-5197: preparation for supporting fair sub-query execution in FedX #5198

Closed aschwarte10 closed 1 week ago

aschwarte10 commented 2 weeks ago

GitHub issue resolved: #5197

This change adds preparational infrastructure for having different implementations of schedulers. Configuration is here prepared by means of defining a "SchedulerFactory" interface with a default implementation aside (which essentially mimics the current behavior).

Note that for ease of development some aspects of ControlledWorkerScheduler are made accessible to sub-classes. The idea is that in the end version there is an abstract scheduler class providing shared functionality and different implementation (e.g. the current FIFO one and a fair implementation)


Note: I would like to see this preparation merged in RDF4J 5.1 such that we can prepare (and test) an implementation of a fair scheduler asynchronously already. The final target version for the implementation is RDF4J 5.2


PR Author Checklist (see the contributor guidelines for more details):

aschwarte10 commented 1 week ago

I have a more general question that isn't really about this PR. How do we make sure that all the various thread pools (executors) and shutdown in a timely manner? Are they just for the duration of the query, or are they for the duration of the federation sail?

The thread pools are alive for the life-time of the repository / sail (i.e. #init / #shutdown). Please note that the thread pools themselves are configured to terminate idle threads after 60s. So if there is no load, there are also no threads dangling around.

aschwarte10 commented 1 week ago

I think this PR should be good to go now. All comments above are addressed and we have a good basis for the next development steps

aschwarte10 commented 1 week ago

@hmottestad any further comments (or time for review) from your side? Otherwise I suggest to merge this one