fake-name / xA-Scraper

69 stars 8 forks source link

Don't update retrieval data if older #85

Closed God-damnit-all closed 4 years ago

God-damnit-all commented 4 years ago

Old ground is being retreaded due to the order in which update_last_fetched was being called, this adds a simple check to make sure the new data is ahead of the old date.

God-damnit-all commented 4 years ago

I just realized I'd get a not iterable error if I didn't check for None first, which leads to redundant code but I honestly don't know a better way around it, hence what I just force-pushed.

Feel free to update it to something better because my searches on Stack Overflow didn't yield a more elegant solution.

Good god do I miss switch-case statements.

fake-name commented 4 years ago

Huh, I assumed that the dates would be strictly monotonic. I wonder if this is only caused by re-tweets.

God-damnit-all commented 4 years ago

@fake-name Nah, it's just because xA-Scraper is processing update_last_fetched events in reverse order for each batch of 20 for some reason. It's pretty apparent when you look at the console, how the date keeps descending as it procs 20 times in a row.

Not sure if it's actually worth fixing that behavior since an if statement is probably going to require less processing power than whatever it'll take to make it call update_last_fetched in the correct order, but seeing as I don't know why it does that I could be wrong.

fake-name commented 4 years ago

So I wound up snagging this and changing a bunch of stuff (you can now reset the fetch times, too).