coveralls
commented
3 years ago Pull Request Test Coverage Report for Build 6444
- 55 of 57 (96.49%) changed or added relevant lines in 2 files are covered.
- No unchanged relevant lines lost coverage.
- Overall coverage increased (+0.3%) to 62.743%
| Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | ||
|---|---|---|---|---|---|
| cmd/etl_worker/etl_worker.go | 0 | 2 | 0.0% | ||
| <!-- | Total: | 55 | 57 | 96.49% | --> |
| Totals | |
|---|---|
| Change from base Build 6436: | 0.3% |
| Covered Lines: | 3523 |
| Relevant Lines: | 5615 |
This change adds a new output mode to the ETL worker (aka parser) to support writing to local files. The
storage.LocalWritertype implements therow.Sinkinterface. With this change there are now three output modes: "bigquery" (direct, original mode), "gcs" (for bq loads from gcs), "local" (new, added for local development).The "local" output mode wites files like "gcs" mode (i.e. JSONL named after the archive) but to a local directory specified with the new
-output_dirflag.The LocalWriter is expected to help simplify local development of the etl worker, e.g. for a new datatype by removing some dependencies on BigQuery, table schemas, and updates.
Tested using unit tests and running locally using:
This invocation is not included in the README yet b/c it is not repeatable. Because this is an exclusive option with other default, this should not impact current deployments.
This change is