search

Informatievlaanderen / VSDS-Linked-Data-Interactions

https://informatievlaanderen.github.io/VSDS-Linked-Data-Interactions/
European Union Public License 1.2
4 stars 6 forks source link

Allow polling multiple URL endpoints in LdioHttpInPoller #324

Closed rorlic closed 1 year ago

rorlic commented 1 year ago

A Data Publisher can have different data sets that are similar and belong to the same Data Owner. When publishing those data sets to the same LDES server using the same LDIO (or NiFi) workbench he/she needs to create multiple nearly identical pipelines which only differ in the source URL that is polled.

To ease maintenance of these pipelines and to keep the configuration simple and compact, it would be better if the LdioHttpInPoller component could handle one or more source URLs. Currently there is no need for a different polling interval if multiple URLs are specified.

The configuration would ideally allow specifying one URL as a string for backwards compatibility as well as an set of URLs as an array, i.e.:

- name: be.vlaanderen.informatievlaanderen.ldes.ldio.LdioHttpInPoller
  config:
    url: www.example.com
    interval: PT1M

and:

- name: be.vlaanderen.informatievlaanderen.ldes.ldio.LdioHttpInPoller
  config:
    url: 
    - www.example1.com
    - www.example2.com
    interval: PT1M
Tomvbe commented 1 year ago

This feature was delivered in https://github.com/Informatievlaanderen/VSDS-Linked-Data-Interactions/pull/317