RSS as a Source

[stockholmux]

Is your feature request related to a problem? Please describe. I have a source of data I'd like to ingest into OpenSearch via Data Prepper. This data source is provided-as-a-service and my only machine readable output is an RSS/Atom feed. Data Prepper has no way to natively ingest this type of data.

Describe the solution you'd like I'd like an entry-pipeline source type for RSS. I should be able to supply a URL and a polling frequency and Data Prepper will grab the data from the RSS URL every n seconds. Each <item> in the feed would be a document and the tags inside would be fields.

Describe alternatives you've considered (Optional)

• This can be accomplished with Logstash and the RSS Input plugin, but I'd like to NOT introduce logstash into my architecture.

Additional context Aside from this specific context, RSS is a surprisingly common output format for a rich variety of different types of tools. This would really allow OpenSearch + Data Prepper to ingest a large variety of different data without any extra coding.

[dlvenable]

This is an interesting feature for Data Prepper. It can be a good starting point for supporting full-text search use-cases within Data Prepper.

I'd like to propose that Data Prepper include this functionality with some limited scope to begin. Here are some limitations:

• Target single-node architectures to start. Since this is a polling mechanism, Data Prepper would need to sync the last RSS item across all nodes to not re-read them. For the initial iteration, Data Prepper can assume one node. It would be possible in this iteration to use the new aggregate processor and de-duplication capability to support multiple nodes. All the nodes would read the same items and then the aggregate processor de-duplicates them.
• Read from public URLs only. Data Prepper will not support authentication in the first iteration.
• Target RSS 2.0 only. No older versions and no Atom.

@stockholmux, Would these limitations work?

[sshivanii]

Use Case

Several customers use RSS(Really Simple Syndication/Rich Site Summary) feeds, which fetch latest news and events from websites in real-time and updates them into one convenient dashboard. These customers want to read the RSS feeds and send them to OpenSearch in a document type. Data Prepper can be leveraged to support this full-text search use case by providing an RSS Source Plugin.

This proposal provides details for supporting RSS as a source for Data Prepper.

Configuration

Here's what a basic pipeline configuration with RSS Source plugin would look like:

source:
  rss:
    url: "https://forum.opensearch.org/latest.rss"
    polling_frequency: PT5M

Configuration Options

Option	Type	Default	Required	Description
url	String	empty	Yes	The RSS feed URL
polling_frequency	Duration	5 mins	No	Frequency to retrieve the RSS feed data

Out of Scope

• This proposal will be supported for single-node architecture of Data Prepper as a start. Using multiple nodes of Data Prepper is not part of this scope.
• Only public RSS feed URLs are supported for the initial iteration of the RSS plugin. Support for authentication is not covered.

Tasks

• [x] #2041
• [x] #2059
• [x] #2072
• [x] #2181
• [x] #2182
• [ ] #2183
• [ ] https://github.com/opensearch-project/data-prepper/issues/2349
• [ ] #2081

[dlvenable]

@sshivanii , Thank you for this proposal. I think the default for polling_frequency should be larger. I'd say on the order of minutes. Maybe 5 minutes - PT5M. Also, I don't think we need to require it since we have a default value.

[sshivanii]

@dlvenable Considering how often the feeds will get refreshed, I agree with keeping it longer, 5 minutes. Good catch, I'll make the polling_frequency optional.