z3z1ma / target-bigquery

target-bigquery is a Singer target for BigQuery. It supports storage write, GCS, streaming, and batch load methods. Built with the Meltano SDK.
MIT License
28 stars 37 forks source link

Error logs when inserting data into bigquery tables #83

Closed blancaruano closed 4 months ago

blancaruano commented 7 months ago

When I run meltano with tap-intercom target-bigquery, it extracts and loads the data correctly to BQ. However, in the logs I see an error for every batch that says that dataset and tables already exist in BQ. Is there any way to deactivate these logs? Or set them as info instead of error because the process works correctly. The methods that fail are: datasetservice.insert and tableservice.insert. The logs say: Already Exists: Dataset x Already Exists: Table x Maybe I am missing something in the setup?

This is my setup:

- name: target-bigquery
    variant: z3z1ma
    pip_url: git+https://github.com/z3z1ma/target-bigquery.git
    config:
      method: batch_job
      location: EU
      add_metadata_columns: true
      flattening_enabled: false
      partition_granularity: day
      batch_size: 1000
      column_name_transforms:
        snake_case: true
      overwrite: true
AlejandroUPC commented 4 months ago

Same, in the bigquery API logs we see plenty of those errors, is there a way to disable or have it check if it exists? Note that this might be considered a bug given that the behavior defined in def create_table docstring says:

"""Creates a dataset and table for this table.

        This is a convenience method that wraps the creation of a dataset and
        table in a single method call. It is idempotent and will not create
        a new table if one already exists."""
z3z1ma commented 4 months ago

This is fixed in main