Open aniketsinghrawat opened 1 year ago
fixes: #311 As mentioned in this pr https://github.com/google/weather-tools/pull/250#issuecomment-1277339173 skipping file prefix validation does not work in case of big query as it was dependent on first_uri to infer schema for big query.
Fixed this issue by creating a pipeline stage that infers the schema and creates the bigquery table using the first uri. This stage is skipped for all subsequent uris.
fixes: #311 As mentioned in this pr https://github.com/google/weather-tools/pull/250#issuecomment-1277339173 skipping file prefix validation does not work in case of big query as it was dependent on first_uri to infer schema for big query.
Fixed this issue by creating a pipeline stage that infers the schema and creates the bigquery table using the first uri. This stage is skipped for all subsequent uris.