[BUG] New Item from Multiple RSS Feeds not working

[nkav]

Describe the bug Having issues with Pipedream reliably triggering when I am tracking multiple RSS feeds. Often times no new triggers will happen even as I know the RSS feeds are being updated with new items.

To Reproduce Steps to reproduce the behavior:

1. Go to Pipedream and set up the "New Item from Multiple RSS Feeds" Trigger
2. Add multiple feeds (ideally 8-10)
3. Set up the feed to trigger on a recurring basis (e.g. daily)
4. Monitor events and notice nothing new is coming in

Expected behavior When I only had 2 feeds on the trigger (https://oversharing.substack.com/feed.xml and https://noahpinion.substack.com/feed.xml), everything triggered on a daily basis as expected without any problems. But then I added multiple more feeds (see below) and then the events stopped coming through, nothing was in the logs, etc.

Feeds: https://mattstoller.substack.com/feed.xml https://modernpower.substack.com/feed.xml https://cityobservatory.org/feed/ https://www.apricitas.io/feed.xml https://herbsundays.substack.com/feed.xml https://erikhoel.substack.com/feed.xml https://theconnector.substack.com/feed.xml https://alltechishuman.substack.com/feed.xml

Screenshots Before the 30th of August, I only had 2 feeds, and then I updated to multiple more feeds (see above). As you can see in the logs, something weird starting happening once I added the additional feeds, and no events were available for me to investigate.

Desktop:

• OS: Mac OS
• Browser: Chrome
• Version 104

Additional context I enabled the option for Pipedream Support to access/update my workflow: https://pipedream.com/@nkav/fewer-rss-feeds-transform-to-markdown-in-todoist-p_mkC9JAl/build

[vunguyenhung]

Hello @nkav, I'm able to reproduce your issue.

Adding the RSS multiple feeds source with 2 sources works correctly

But the RSS multiple feeds source with 7 sources does not emit any event

Expected behavior: the RSS multiple feeds source with 7 sources should emit events from all sources, sorted by published date

I have added this to our Component backlog 🙏

[dylburger]

@vunguyenhung Did you see anything in the source logs that would suggest a specific error? e.g. did it appear to timeout trying to fetch records from all 7 feeds?

[vunguyenhung]

Hello @dylburger, here's the log for RSS multiple feeds source with 7 sources

It does not throw error, but it does not log the suggest log either.

Here's the log for RSS multiple feeds source with 2 sources

[dylburger]

Thanks!

[nkav]

Awesome, thank you for investigating here! Is there by any chance an ETA?

[dylburger]

@nkav I just moved it to the top of our backlog. Generally we can get any change out within 1-2 weeks. We track the status of the ticket at each stage in time (e.g. working on it, in QA, etc.) so you'll be able to follow here for updates. If you feel comfortable using the Pipedream CLI, you can even check out the PR once it's code-complete, and deploy that version of the source before it's released to production to make sure the code works with your RSS feeds.

[ghost]

@michelle0927 as leo mentioned previously iam facing the same issue still the rSS Feed works fine on inputting 2 urls but not when its 7

Expected behavior: the RSS multiple feeds source with 7 sources should emit events from all sources, sorted by published date

[andrewjschuang]

@Hassan-Abbas-SQA I don't think currently we can have too many urls currently, since it exceeds the maximum time when fetching all of them. That's why the description was added.

I've tested without the sorting part, but the result is the same. Hence the problem is when there are too many URLs.

[ghost]

@andrewjschuang can you plaese move this ticket to ready for QA so i can ship it to ready for release

[michelle0927]

@dannyroosevelt if there is no need of many urls right now can we ship this to ready for release and @andrewjschuang will create another issue for tracking this issue later?

@Hassan-Abbas-SQA Did you try making use of the max prop? If you set it to a lower number (like 10), it will work for many urls. That's the reason it was added in this PR.

[andrewjschuang]

Wait I just checked the code and we're running it synchronously. @michelle0927 can you change it to parallel and test it again to see if it helps? With and without the sorting too?

[michelle0927]

@andrewjschuang Is this what you mean? I tried changing it to the following, and am seeing the same results.

  hooks: {
    async activate() {
      const promises = this.urls.map(url => this.rss.fetchAndParseFeed(url));
      await Promise.all(promises);
    },
  },
  async run() {
    const promises = this.urls.map(url => this.rss.fetchAndParseFeed(url));
    const items = await Promise.all(promises);
    for (const item of items) {
      const meta = this.generateMeta(item);
      this.$emit(item, meta);
    }
  },

[ghost]

@dannyroosevelt if there is no need of many urls right now can we ship this to ready for release and @andrewjschuang will create another issue for tracking this issue later?

@Hassan-Abbas-SQA Did you try making use of the max prop? If you set it to a lower number (like 10), it will work for many urls. That's the reason it was added in this PR.

yes it worked

[andrewjschuang]

@andrewjschuang Is this what you mean? I tried changing it to the following, and am seeing the same results.

  hooks: {
    async activate() {
      const promises = this.urls.map(url => this.rss.fetchAndParseFeed(url));
      await Promise.all(promises);
    },
  },
  async run() {
    const promises = this.urls.map(url => this.rss.fetchAndParseFeed(url));
    const items = await Promise.all(promises);
    for (const item of items) {
      const meta = this.generateMeta(item);
      this.$emit(item, meta);
    }
  },

Yes, but can you remove it from activate? Because it will run twice when deployed

[michelle0927]

@andrewjschuang I tried removing the activate hook, and it's still freezing up for me.

[andrewjschuang]

@andrewjschuang I tried removing the activate hook, and it's still freezing up for me.

Oh then I guess there's no way, thanks for testing