Is the feed size limit too low?

maddyblue / goread

RSS reader in go on app engine; formerly goread.io

ISC License

2.38k stars 294 forks source link

Is the feed size limit too low? #343

Open olof opened 8 years ago

olof commented 8 years ago

Hello,

Question rather than a bug report: I'm trying out goread and found two podcasts that are both larger than the 2097152 byte limit (1<<21, hardcoded in tasks.go: fetchFeed).

$ curl -s http://api.sr.se/api/rss/pod/4886 | wc -c
2307379

$ curl -s http://api.sr.se/api/rss/pod/4023 | wc -c
4117725

Both of the podcast feeds are from Sweden's public service radio broadcaster.

Is the 1<<21 byte limit based on real limitations, or would increasing this be possible?

Thanks,

HaXXoR75 commented 8 years ago

According to the commit where this limit was added ( f58356a ), it was done because extremely large feeds cause the instance to crash, without incrementing the error counter, causing it to constantly retry and crash over and over. The example feed linked there is currently 6076129 bytes, so it can be safely assumed that size is a real limitation. whether or not you can increase the limit to 1 < 22 or 23 safely would require trying it and seeing, while im quite sure the first feed you mention would work, I'm really not sure on the second

olof commented 8 years ago

Thanks. I've had issues with other, desktop RSS managers handling them badly, but this is the first time a tool outright refuses to parse it. If you don't consider this a bug, please close this issue and I'll have to find some other solution.