search

m-lab / etl

M-Lab ingestion pipeline
Apache License 2.0
22 stars 7 forks source link

Retire daily parsers and associated configurations #888

Open stephen-soltesz opened 4 years ago

stephen-soltesz commented 4 years ago

We have three parser configs: daily, batch, and universal. Universal gardener will be deployed to production soon. The daily and batch parser configs are implicated in parser failures for traceroute datatype.

From the 2020-05-20 monitoring discussion it sounds like we can retire the legacy daily parsers now. Removing them sooner rather than later could help eliminate that failure mode and simplify over all system complexity.

yachang commented 4 years ago

https://github.com/m-lab/etl/pull/886

stephen-soltesz commented 4 years ago

If Cloud Functions were only used for the daily parsers they should also be removed.

stephen-soltesz commented 4 years ago

Deployed services in AppEngine for the daily parsers (now orphaned) should be deleted.