search

mathcolo / mbtapuller

Train stuff
MIT License
2 stars 0 forks source link

Catch all pull exceptions #56

Closed mathcolo closed 8 years ago

mathcolo commented 8 years ago

pull_data is failing due to regular API oddities in the middle of the night, this is an error catch-all so that it doesn't terminate. Not expecting this to have any adverse effects on data consistency or day-to-day trip turnover.

tinzerillo commented 8 years ago

So it just retries every minute until it works again? Should we have a limit for when it stop it if it's failed x times? Also, I have been meaning to ask..what is the 'once' param for? I don't see it being set to true ever (am I missing something?)

mathcolo commented 8 years ago

There should probably be a limit—I agree and thought about this too a bit. What exactly the logic should be behind that is perhaps "coming soon" since it's unclear to me exactly how long these glitches last. If they're short-lived then we don't really need to back off, otherwise we should. In theory would merge this in and then do some log file analysis to investigate.

Re: the --once argument, it is set by the command line input with pull_data.py:34. Allows you to pass --once in the CLI so you can get one set of data in the db, then no more! 👍

tinzerillo commented 8 years ago

Ok! I understand. I think fine to merge for now then and then make an issue to figure out the limit.