fetch_url_error: This URL returned a 524 status code

[alek-p]

Thanks for the great library, I've been using it for years and after upgrading to v3.1.0 I'm running into problems so I had to revert to using v2.0.1

This is the trace that I keep seeing:

      throw er; // Unhandled 'error' event
      ^

fetch_url_error: This URL returned a 524 status code
    at Request.request.get.on (/code/rssalerts/node_modules/rss-feed-emitter/src/Feed.js:128:28)
    at emitOne (events.js:115:13)
    at Request.emit (events.js:210:7)
    at Request.onRequestResponse (/code/rssalerts/node_modules/request/request.js:1066:10)
    at emitOne (events.js:115:13)
    at ClientRequest.emit (events.js:210:7)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:565:21)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:116:23)
    at Socket.socketOnData (_http_client.js:454:20)
    at emitOne (events.js:115:13)
    at Socket.emit (events.js:210:7)
    at addChunk (_stream_readable.js:266:12)
    at readableAddChunk (_stream_readable.js:253:11)
    at Socket.Readable.push (_stream_readable.js:211:10)
    at TCP.onread (net.js:587:20)

[TobiTenno]

The library hasn't changed how it's retrieving data, so you may want to check your node version...

If you could give us the URL that is giving you an error, we can add it to our integration suite and figure out what's up, as a 524 is a connection timeout and not likely to be on the caller's end.

Could you provide the following:

Node.js version:

Feed url:

Any other custom config you're providing:

[alek-p]

I noticed the error handling has changed so perhaps the timeouts were just getting ignored before instead of throwing the error?

Node.js version: v0.10.25

Feed url: I have ~10 feeds connected, I'll find which one is giving the error and will update this ticket.

Any other custom config you're providing: No custom configs

[alek-p]

Here is the partial URL of the feed I'm using: http://api.prnewswire.com/releases/version01/getRSS?query=language:en&pageSize=100&getRecent=true&timeInterval=60&apikey=xxxxxxxx unfortionatley I can't provide the full URL without revealing my API key. You could sign up for your own API key here: http://api.prnewswire.com/user/jsp/register.jsp

[TobiTenno]

yeah, we don't support node v0.10 anymore. Please upgrade to a minimum of node 8.

[TobiTenno]

You'll get a lot of improved performance and syntax upgrades to be on par with modern javascript.

[midzer]

I am experiencing the same since upgrade. node: v12.16.1

example URL: https://www.oedp-dachau.de/service/rss/feeds/oedp-blog-rss-feed/

[TobiTenno]

The same what? Precisely the same error?

Have you tried moving where you're running it, as, again, 524 is a timeout, not a client end failure

[TobiTenno]

@alek-p look at your feed url, is it possible your page size is just too large? 🤔

[TobiTenno]

@midzer i tested your url and i'm getting nothing from it because there's no items in it....

const FeedEmitter = require('rss-feed-emitter');
const feeder = new FeedEmitter();

const testUrls = ['https://www.oedp-dachau.de/service/rss/feeds/oedp-blog-rss-feed/'];

const feeds = feeder.add({
  url: testUrls,
  timeout: 10000,
  skipFirstLoad: false,
});

feeder.on(`initial-load:${testUrls[0]}`, console.log);

console.log('starting....');
feeder.on('new-item', (i) => {
  console.log(i);
  console.log(feeds.length);
  console.log(feeds[0].items.length);
});
feeder.on('error', console.error);

λ node test.js
starting....
{
  url: 'https://www.oedp-dachau.de/service/rss/feeds/oedp-blog-rss-feed/',
  items: []
}

<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:content="http://purl.org/rss/1.0/modules/content/" version="2.0">
   <channel>
      <title>ÖDP Dachau: Blog</title>
      <link>https://www.oedp-dachau.de/</link>
      <description>Unsere Kolumne zu aktuellen politischen Themen - jeden Freitag neu.</description>
      <language>en</language>
      <image>
         <title>ÖDP Dachau: Blog</title>
         <url>https://www.oedp-dachau.de/fileadmin/templates/images/rss-icon.gif</url>
         <link>https://www.oedp-dachau.de/</link>
         <width>15</width>
         <height>15</height>
         <description>Unsere Kolumne zu aktuellen politischen Themen - jeden Freitag neu.</description>
      </image>
      <generator>TYPO3 - get.content.right</generator>
      <docs>http://blogs.law.harvard.edu/tech/rss</docs>
      <lastBuildDate>Thu, 01 Jan 1970 01:00:00 +0100</lastBuildDate>
   </channel>
</rss>

[alek-p]

100 entries is not all that big and going smaller but more frequent risks me missing items as they update in bulk. That would probably be worse than fetching all items but later. Also 100 entries is currently working without throwing up an uncaught exception with the older version of the library. That might mean it just silently fails when there is a feed timeout but that's ok for me right now.

[midzer]

@TobiTenno Thanks for investigating!

Strangly, rss-feed-emitter v3 crashed because of this url after a while of running. Same with 2 or 3 other RSS feeds which I have disabled currently. This did not happen with v2. Meanwhile I have downgraded back to v2 on my production site.

[TobiTenno]

@alek-p okiedoke. i'm not familiar with how entries are sized in rss.

@midzer i'll keep my little local one running and if you have any others you want me to test, i'll happily test them all locally. i'd love to see what we can do to figure out the crash.

[TobiTenno]

@midzer are you, by chance, not creating an 'error' handler for the feeder? I realized I have one which may be why my process isn't crashing. I did end up getting an error when I hibernated my computer and re-awoke it, but those were due to actual disconnections from the internet.

[midzer]

Not that I know of (same as v2)

Here's my current v3 code https://github.com/midzer/feed-websocket/blob/push/index.js (for v2 is one commit earlier)

[TobiTenno]

yeah, so in general, i'd recommend adding a feeder.on('error', () => {}) handler at minimum

[TobiTenno]

This is pretty much standard with any emitter (so far as i've seen), since it can cause unhandled exception exceptions if you don't handle potential errors from the error event

Like even if it is a specific issue to v3 that we're exposing the errors like this, it's better to code defensively like this

[filipedeschamps]

:tada: This issue has been resolved in version 3.2.0 :tada:

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot :package::rocket: