search

vprnet / twitter-bot-sewage

Automated tweets when there’s an update to the State of Vermont’s database of reported sewage overflows.
https://twitter.com/dirtywatervt
3 stars 0 forks source link

Assign unique IDs to spills? #2

Open smbsimon opened 9 years ago

smbsimon commented 9 years ago

Today, an overflow was reported, the bot tweeted, then new information about the spill was discovered. When the bot ran again, it saw new data and treated this new data as a new overflow entirely. The bot tweeted this new information, so the logs now show two similar reports for the same spill.

It's an interesting question: Does an update to an overflow justify a new tweet?

The goal of the bot is simply to report the information on the water waste inventory site, and so in order to keep with the desired functionality, I've decided to have to bot continue to tweet updates. What might be nice, though, would be for it to add an "UPDATE" tag to the tweet if it's clearly an update. Would assigning unique IDs and then checking a tweet against an ID solve the puzzle? Is there a better way to skirt around this issue?

screen shot 2015-09-03 at 3 44 08 pm
akvamme commented 9 years ago

if the url for the specific spill (from the view link on the index page) stays the same after an update (hopefully it does), you should be able make that the unique id, and check accordingly.

however: searching a log file is going to start to be a performance problem as it grows.

smbsimon commented 9 years ago

Totally get the performance problem issue. I'm thinking I might have it delete way old logs after a month or so... tbd. Not sure about the view link on the index page, but that's an A+ idea. They actually keep changing the way the site's organized, and I think they might be doing something tricky with the submission number column that they just added last week, so that might solve all my problems. Thanks, Andy!