search

mozilla / gcp-ingestion

Documentation and implementation of telemetry ingestion on Google Cloud Platform
https://mozilla.github.io/gcp-ingestion/
Mozilla Public License 2.0
75 stars 31 forks source link

Make bq_ properties in PipelineMetadataStore nullable #2548

Closed akkomar closed 6 months ago

akkomar commented 6 months ago

In https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/806 we added json_object_path_regex and removed bq_dataset_family, bq_metadata_format, and bq_table from base Glean schema metadata. This caused schema loading to fail with:

ValueInstantiationException: Cannot construct instance of `com.mozilla.telemetry.ingestion.core.schema.AutoValue_PipelineMetadataStore_PipelineMetadata$Builder`, problem: Missing required properties: bq_dataset_family bq_table bq_metadata_format

To unblock schema loading we'll set these fields as @Nullable so PipelineMetadata for the base Glean schema can be instantiated. This is a safe change because these fields are not used anywhere in the ingestion logic (proposal that brought them in hasn't been fully implemented).

codecov-commenter commented 6 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (3d56691) 85.23% compared to head (cfab387) 85.23%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2548 +/- ## ========================================= Coverage 85.23% 85.23% Complexity 911 911 ========================================= Files 123 123 Lines 5254 5254 Branches 536 536 ========================================= Hits 4478 4478 Misses 607 607 Partials 169 169 ``` | [Flag](https://app.codecov.io/gh/mozilla/gcp-ingestion/pull/2548/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mozilla) | Coverage Δ | | |---|---|---| | [ingestion_beam](https://app.codecov.io/gh/mozilla/gcp-ingestion/pull/2548/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mozilla) | `82.71% <ø> (ø)` | | | [ingestion_core](https://app.codecov.io/gh/mozilla/gcp-ingestion/pull/2548/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mozilla) | `75.07% <ø> (ø)` | | | [ingestion_sink](https://app.codecov.io/gh/mozilla/gcp-ingestion/pull/2548/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mozilla) | `84.88% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mozilla#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.