Closed benkonz closed 6 months ago
Attention: Patch coverage is 25.00000%
with 6 lines
in your changes are missing coverage. Please review.
Project coverage is 70.91%. Comparing base (
f637e88
) to head (6ed1bbf
). Report is 8 commits behind head on master.
Files | Patch % | Lines |
---|---|---|
...ala/com/spotify/ratatool/samplers/BigSampler.scala | 33.33% | 4 Missing :warning: |
...spotify/ratatool/samplers/BigSamplerBigQuery.scala | 0.00% | 2 Missing :warning: |
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
adds a new arg to BigSampler called
bigqueryPartitioning
, defaults to "DAY", which should maintain the same behavior as before. Users can pass in "DAY|HOUR|MONTH|YEAR", as well as NULL if no table partitioning is desired.Making this change so that Ratatool works better with Spotify's internal Luigi BigQuery tasks, which use table sharding as partitioning, and when ratatool sets the partitioning to ingestion day, it causes problems with retention.
Tested by outputting this table via this workflow:
the
43ea5c916cd5a85623bf0de598da15982c29d8952dbf63a068d10e5b56466e61
docker image is using my local ratatool PR's code viasbt publishM2
the linked table has to partitioning and uses the sharding generated by the BigQueryTarget in Luigi
here is another table that is using the
--bigquery-partitioning
arg to set the partitioning to "MONTH".