transitmatters / gobble

🦃 Process MBTA events into a format that can be consumed by the Data Dashboard
MIT License
2 stars 3 forks source link

Keep trip state in one JSON file per route #60

Closed idreyn closed 10 months ago

idreyn commented 10 months ago

Resolves #59 (see that issue for a detailed description of the rationale)

This PR adds a trip_state.py that is responsible for keeping track of the current state of a given trip (i.e. the previous event that it recorded). The diff between the last event and the new one is used by process_event().

Instead of a single state.json file, we now write one trip_states/{route_id}.json file per route. This should increase disk write performance, with smaller files and no write contention between different event processing threads.

Test plan:

I ran python -m gobble and verified that:

devinmatte commented 10 months ago

Also if you want to use datadog locally, you can set up a local agent with our account API keys and send profiles from local. Just set env=local or something

idreyn commented 10 months ago

(Sorry for the crazy number of commits from my other branch, they'll all get squashed down though)