Script skips everything (too old)

[Chiharuuu]

Not sure why this keeps happening, but every time I start the feed2discord.py, it loads up fine and connects to the bot account, and every time it reads the RSS URLs (just Reddit so far), it keeps throwing back similar line to this:

INFO:main:kpopreddit:item t3_43oas1 unseen, processing: INFO:main:kpopreddit:too old; skipping

I have no idea why it keeps saying that all the posts are old, because one was made only minutes before on /r/kpop. The bot has all the required permissions to connect and post messages in the Discord channel too. It doesn't post anything from the feed in any of the channels and eventually it closes with:

/usr/lib/python3.5/asyncio/selector_events.py:577: ResourceWarning: unclosed transport <_SelectorSocketTransport fd=10>

If it's of anything, this is how I've got the feed in feed2discord.ini:

[kpopreddit] channels = rkpop feed_url = https://www.reddit.com/r/kpop/.rss fields = link max_age = 7200

As well as the Channels section:

[CHANNELS] rkpop = 164987645401890816

[DEFAULT] rss_refresh_time = 500 max_age = 86400 send_typing = 1

It does the same thing for all 3 entries I have in the CHANNELS section, which are all formatted similarly (and I figure it's correct, hopefully, with the last number in the channel's URL).

This is all running on a Ubuntu 14.04 server with Python 3.5.0 installed, for reference.

Sorry if this is long, but I'm sorta new to Github and needed some help with this. Thanks!

[freiheit]

The "too old; skipping" means that the item (post) in the feed is older than the "max_age" for that feed. Doesn't look like any kind of permissions problem for the channel (or if there is, that's a separate issue)

When I examine that RSS feed by hand, I see that the first item in the feed has a date of March 25... Try this feed_url instead: https://www.reddit.com/r/kpop/new/.rss

[Chiharuuu]

With the new feed URL, I'm still getting the same "too old; skipping" error. Really have no idea why...

With debug set to 3, these are the last lines. All previous ones say "too old" as well, even though I'm using the URL for the new feed rather than default.

DEBUG:__main__:kpopreddit:item:checking database history for this item INFO:__main__:kpopreddit:item t3_4cqjh6 unseen, processing: INFO:__main__:kpopreddit:too old; skipping DEBUG:__main__:kpopreddit:sleeping for 500 seconds WARNING:asyncio:Executing <Task pending coro=<background_check_feed() running at feed2discord.py:320> wait_for=<Future pending cb=[Task._wakeup()] created at /usr/lib/python3.5/asyncio/tasks.py:503> created at feed2discord.py:333> took 0.247 seconds

[freiheit]

What date/time does your system think it is? (run date -u to see)

[Chiharuuu]

Server's time/date is Fri Apr 1 01:52:57 UTC 2016 should be fine, right?

Also on a potentially unrelated note, I tried this same thing replaced with an Instagram tag feed url (http://widget.websta.me/rss/tag/taghere) and it worked provided I set the max_age to a very high number, and started sending them into my Discord channel, but then when I tried a feed url for just one user (http://widget.websta.me/rss/n/usernamehere), it gave back this:

DEBUG:__main__:instagramtest:processing entries DEBUG:__main__:instagramtest:item:processing this entry DEBUG:__main__:instagramtest:item:checking database history for this item INFO:__main__:instagramtest:item http://websta.me/p/1214601160999041871_329452045 $ unseen, processing: ERROR:__main__:instagramtest:Unexpected error: ERROR:__main__:(<class 'TypeError'>, TypeError('Tuple or struct_time argument r$ quired',), <traceback object at 0x7f3c364d6ec8>) ERROR:__main__:instagramtest:giving up

So now I've got this and the Reddit error too... I've installed NTP and synced servers properly, but still the same errors for "too old" as well.

[freiheit]

Yes, that date/time looks about right...

Please do a "git pull", set debug to 4 and try again with the reddit feed. I've added some additional debugging that will output all the relevant dates...

That TypeError on the instagram feed is definitely a bug, but a different bug. Can you put that into a new ticket?

[freiheit]

I think I have an error with accidentally treating an item's date as if it's in the local timezone (but without converting it)... I'll have to look at that more carefully.

[Chiharuuu]

I can't figure out how to post the log because it's so long, so I've uploaded the text file with log here in it's entirety. Debug was set to 4 for this. Will make a ticket for the TypeError too. Also, my server's time is currently set to KST, if that makes a difference. Can be changed if needed.

[freiheit]

This is the bit I needed:

DEBUG:__main__:kpopreddit:item:processing this entry
DEBUG:__main__:kpopreddit:item:id:t3_4czg2d
DEBUG:__main__:kpopreddit:item:checking database history for this item
INFO:__main__:kpopreddit:item t3_4czg2d unseen, processing:
INFO:__main__:kpopreddit:too old; skipping
DEBUG:__main__:kpopreddit:now:1459561548.9034908
DEBUG:__main__:kpopreddit:now:gmtime:time.struct_time(tm_year=2016, tm_mon=4, tm_mday=2, tm_hour=1, tm_min=45, tm_sec=48, tm_wday=5, tm_yday=93, tm_isdst=0)
DEBUG:__main__:kpopreddit:now:localtime:time.struct_time(tm_year=2016, tm_mon=4, tm_mday=2, tm_hour=10, tm_min=45, tm_sec=48, tm_wday=5, tm_yday=93, tm_isdst=0)
DEBUG:__main__:kpopreddit:pubDate:2016-04-02T01:45:32+00:00
DEBUG:__main__:kpopreddit:pubDate_parsed:time.struct_time(tm_year=2016, tm_mon=4, tm_mday=2, tm_hour=1, tm_min=45, tm_sec=32, tm_wday=5, tm_yday=93, tm_isdst=0)
DEBUG:__main__:{'id': 't3_4czg2d', 'link': 'https://www.reddit.com/r/kpop/comments/4czg2d/what_song_do_you_have_on_repeat_right_now/', 'guidislink': True, 'author': '/u/YeBeAWitch', 'updated': '2016-04-02T01:45:32+00:00', 'content': [{'type': 'text/html', 'base': '', 'value': '<!-- SC_OFF --><div class="md"><p>I&#39;m normally into more upbeat stuff but for me it&#39;s <a href="https://www.youtube.com/watch?v=2fBODWRrnWM">Sunggyu&#39;s kontrol</a>. For the last week I&#39;ve been playing it first no matter what I&#39;m listening to.</p> </div><!-- SC_ON --> &#32; submitted by &#32; <a href="https://www.reddit.com/user/YeBeAWitch"> /u/YeBeAWitch </a> <br/> <span><a href="https://www.reddit.com/r/kpop/comments/4czg2d/what_song_do_you_have_on_repeat_right_now/">[link]</a></span> &#32; <span><a href="https://www.reddit.com/r/kpop/comments/4czg2d/what_song_do_you_have_on_repeat_right_now/">[comments]</a></span>', 'language': None}], 'href': 'https://www.reddit.com/user/YeBeAWitch', 'authors': [{'name': '/u/YeBeAWitch', 'href': 'https://www.reddit.com/user/YeBeAWitch'}], 'updated_parsed': time.struct_time(tm_year=2016, tm_mon=4, tm_mday=2, tm_hour=1, tm_min=45, tm_sec=32, tm_wday=5, tm_yday=93, tm_isdst=0), 'tags': [{'scheme': None, 'term': 'kpop', 'label': '/r/kpop'}], 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://www.reddit.com/r/kpop/comments/4czg2d/what_song_do_you_have_on_repeat_right_now/'}], 'title': 'What song do you have on repeat right now?', 'summary': '<!-- SC_OFF --><div class="md"><p>I&#39;m normally into more upbeat stuff but for me it&#39;s <a href="https://www.youtube.com/watch?v=2fBODWRrnWM">Sunggyu&#39;s kontrol</a>. For the last week I&#39;ve been playing it first no matter what I&#39;m listening to.</p> </div><!-- SC_ON --> &#32; submitted by &#32; <a href="https://www.reddit.com/user/YeBeAWitch"> /u/YeBeAWitch </a> <br/> <span><a href="https://www.reddit.com/r/kpop/comments/4czg2d/what_song_do_you_have_on_repeat_right_now/">[link]</a></span> &#32; <span><a href="https://www.reddit.com/r/kpop/comments/4czg2d/what_song_do_you_have_on_repeat_right_now/">[comments]</a></span>', 'title_detail': {'type': 'text/plain', 'base': '', 'value': 'What song do you have on repeat right now?', 'language': None}, 'author_detail': {'name': '/u/YeBeAWitch', 'href': 'https://www.reddit.com/user/YeBeAWitch'}}

[freiheit]

Yeah, I think this is likely a timezone bug... Probably have to change how I do those date comparisons.

In the meantime, I think you can work around this by setting the "TZ" environment variable to "UTC" when you run the script. Like: TZ=UTC python feed2discord.py

[Chiharuuu]

TZ=UTC python feed2discord.py did the trick, posts fine now and I've set it up to posts 3 subreddits so far, works beautifully. Thanks!

[freiheit]

I just pushed a change that should force it to do the equivalent on its own. If you git pull you should be able to run without manually doing the "TZ=UTC" thing anymore.

[Chiharuuu]

Runs fine with just python feed2discord.py now, much better.

[freiheit]

FYI, there's some major improvements to the timezone handling. Should work on Windows or Linux, and does things even smarter than that workaround. A "git pull" to get the latest should work, or you can look at #24 to see the details.