support multiple LDES sources for one vocab (aka dataset)

[marc-portier]

The current system already supports adding extra sources to a certain vocab via the "add source" button in the sources tab:

However these can only be of type "dump" not of type "ldes feed"

The current support seems to be intended for usage with multiple dumps all collaborating to offer the content needed for one vocabulary. Even if the original driver behind this was introducing multilingual support over different dumps, there seems to be no assumptions about how the triples over these separate sources are expected to be distributed: i.e. they could either be fragmenting the available entities of the total vocab, either be providing different properties, either be offering translated value-strings, either some combination of the previous.

Apart from the free fragmentation support, an extra benefit of this grouping of «multiple sources per vocab» is that it reuses (i.e. avoids duplication) the same mapping config, as well as the running of analysis and unification. Which makes it superior to the alternative of defining multiple vocabs to be search together through the widget-config.

The argument is that this feature is very useful, but should just be extended to be able to support multiple mixed sources per vocab: adding both ldes and dump type sources to a vocab to further benefit from this.

Also note: this option is available if one starts off with an LDES-feed type vocabulary -- but to what effect?

Concrete use cases for this support are:

• a similar multilingual support with triples coming in through LDES feeds that are organised per language (NOTE: possibly one main language would be providing the full entity graph and state properties like owl:deprecated to be applied to all languages )
• adding other derived (i.e. calculated on top of some raw-ingest feed) entities/properties that are provided by a secundary LDES feed but need to be available in search-index or labeling
• ingesting a basic dump entity to kickstart (speed up) the subsequent change-sync via LDES

[nvdk]

As we understand this issue: When adding an additional source for a vocabulary vocabsearch will support the extra dataset to be either a LDES feed or a file. The challenge of having multiple LDES feeds as sources is to determine when it's a good time to update search set (e.g. "unification", so applying the mapping ). Given automatic updates (mapping already defined and no user interaction) we propose doing this on a set interval:

• once an hour while any LDES source of a vocabulary is in "consuming mode" (that is, still receiving new information).
• on finishing consumption if no other LDES source are in consuming mode

It will also be possible to trigger unification manually via the interface.

[nvdk]

addressed in 1.1 release