m-lab / etl

M-Lab ingestion pipeline
Apache License 2.0
22 stars 7 forks source link

Add handler for local output mode #994

Closed stephen-soltesz closed 3 years ago

stephen-soltesz commented 3 years ago

This change adds a new handler to the passive mode ETL worker to support v2 datatypes including during local development.

This change adds a new resource, /v2/worker, that is unconditionally available when the worker is started. The /v2/worker handler accepts a filename= parameter (like the existing /worker handler used by the legacy pipeline). However, the /v2/worker only supports v2 datatypes.

With this handler, a developer can run the worker in "local" output mode to test behavior of a new parser without depending on BigQuery or other GCP services (other than the source archive in GCS). This handler also allows adhoc tests in GKE.

Instructions for use are now included in the README.md

This change is Reviewable

coveralls commented 3 years ago

Pull Request Test Coverage Report for Build 6470


Changes Missing Coverage Covered Lines Changed/Added Lines %
cmd/etl_worker/etl_worker.go 29 37 78.38%
<!-- Total: 29 37 78.38% -->
Totals Coverage Status
Change from base Build 6456: 0.6%
Covered Lines: 3572
Relevant Lines: 5640

💛 - Coveralls