Closed rviscomi closed 3 years ago
To get around the 15 TB limitation of Dataflow, load the response_bodies tables in halves.
response_bodies
This change also upgrades to Apache Beam 2.31 and removes the old experimental use_beam_bq_sink flag.
use_beam_bq_sink
Tested successfully on 2021_07_01 for both desktop and mobile.
To get around the 15 TB limitation of Dataflow, load the
response_bodiestables in halves.This change also upgrades to Apache Beam 2.31 and removes the old experimental
use_beam_bq_sinkflag.Tested successfully on 2021_07_01 for both desktop and mobile.