search

z3z1ma / target-bigquery

target-bigquery is a Singer target for BigQuery. It supports storage write, GCS, streaming, and batch load methods. Built with the Meltano SDK.
MIT License
27 stars 34 forks source link

Unable to load raw JSON fields into bigquery #82

Closed melgazar9 closed 3 months ago

melgazar9 commented 5 months ago

When I run meltano tap-coingecko target-jsonl I am able to load raw json fields with th.AnyType() or th.CustomType({"anyOf": [{"type": "object"}, {"type": "array"}, {}]}). However when using target-bigquery (target-snowflake fails as well) the loader fails. Maybe there's a simple parameter I'm missing / misunderstanding? Here's my setup:

    - name: target-bigquery
      variant: z3z1ma
      pip_url: git+https://github.com/z3z1ma/target-bigquery.git
      config:
        method: gcs_stage
        project: ${GCP_PROJECT_ID}
        credentials_path: ${GOOGLE_APPLICATION_CREDENTIALS}
        dataset: ${GCP_DATASET}
        batch_size: 100000
        denormalized: true
        bucket: ${GCP_BUCKET_NAME}

When using th.AnyType() defined as my schema I can see the jsonl file successfully loaded into the gcs bucket. The problem is with the below "platforms" json field. Downloading the file here's the last record:

{"id":"zzz","symbol":"zzz","name":"GoSleep ZZZ","platforms":{"arbitrum-one":"0x7a2c1b8e26c48a5b73816b7ec826fd4053f5f34b","binance-smart-chain":"0x0b9bdcc696efa768cafe0e675525eaf42e32d108"},"_sdc_extracted_at":"2024-03-21T17:00:38.332157+00:00","_sdc_received_at":"2024-03-21T12:01:01.052616","_sdc_batched_at":"2024-03-21T12:01:00.844510","_sdc_deleted_at":null,"_sdc_sequence":1711040461053,"_sdc_table_version":null}

The error with th.AnyType() is

Traceback (most recent call last): cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.631997Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/bin/target-bigquery", line 8, in <module> cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.633221Z [info     ]     sys.exit(TargetBigQuery.cli()) cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.633634Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/click/core.py", line 1157, in __call__ cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.634096Z [info     ]     return self.main(*args, **kwargs) cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.634305Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/click/core.py", line 1078, in main cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.634497Z [info     ]     rv = self.invoke(ctx)      cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.635373Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/click/core.py", line 1434, in invoke cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.635594Z [info     ]     return ctx.invoke(self.callback, **ctx.params) cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.635706Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/click/core.py", line 783, in invoke cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.635809Z [info     ]     return __callback(*args, **kwargs) cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.635926Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/singer_sdk/target_base.py", line 578, in cli cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.636006Z [info     ]     target.listen(file_input)  cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.636072Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/singer_sdk/io_base.py", line 35, in listen cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.636135Z [info     ]     self._process_endofpipe()  cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.636199Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/singer_sdk/target_base.py", line 293, in _process_endofpipe cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.636260Z [info     ]     self.drain_all(is_endofpipe=True) cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.636318Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/target.py", line 534, in drain_all cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.636377Z [info     ]     sink.clean_up()            cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.636439Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/gcs_stage.py", line 195, in clean_up cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.636499Z [info     ]     client.load_table_from_uri( cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.636588Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/google/cloud/bigquery/job/base.py", line 971, in result cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.636734Z [info     ]     return super(_AsyncJob, self).result(timeout=timeout, **kwargs) cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.636817Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/google/api_core/future/polling.py", line 261, in result cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.636895Z [info     ]     raise self._exception      cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.636958Z [info     ] google.api_core.exceptions.BadRequest: 400 Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details. File: gs://<bucket>/target_bigquery/<bucket>/coin_list/extracted_date=2024-03-21/d9bbd335-d9f1-4b03-a911-9caac12f3870.jsonl.gz; reason: invalid, location: gs://<bucket>/target_bigquery/<bucket>/coin_list/extracted_date=2024-03-21/d9bbd335-d9f1-4b03-a911-9caac12f3870.jsonl.gz, message: Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details. File: gs://<bucket>/target_bigquery/<bucket>/coin_list/extracted_date=2024-03-21/d9bbd335-d9f1-4b03-a911-9caac12f3870.jsonl.gz; reason: invalid, message: Error while reading data, error message: JSON processing encountered too many errors, giving up. Rows: 1; errors: 1; max bad: 0; error percent: 0; reason: invalid, location: gs://<bucket>/target_bigquery/<bucket>/coin_list/extracted_date=2024-03-21/d9bbd335-d9f1-4b03-a911-9caac12f3870.jsonl.gz, message: Error while reading data, error message: JSON parsing error in row starting at position 0: JSON object specified for non-record field: platforms File: gs://<bucket>/target_bigquery/<bucket>/coin_list/extracted_date=2024-03-21/d9bbd335-d9f1-4b03-a911-9caac12f3870.jsonl.gz cmd_type=loader name=target-bigquery run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr
2024-03-21T17:01:03.730429Z [error    ] Loading failed                 code=1 message=google.api_core.exceptions.BadRequest: 400 Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details. File: gs://<bucket>/target_bigquery/<bucket>/coin_list/extracted_date=2024-03-21/d9bbd335-d9f1-4b03-a911-9caac12f3870.jsonl.gz; reason: invalid, location: gs://<bucket>/target_bigquery/<bucket>/coin_list/extracted_date=2024-03-21/d9bbd335-d9f1-4b03-a911-9caac12f3870.jsonl.gz, message: Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details. File: gs://<bucket>/target_bigquery/<bucket>/coin_list/extracted_date=2024-03-21/d9bbd335-d9f1-4b03-a911-9caac12f3870.jsonl.gz; reason: invalid, message: Error while reading data, error message: JSON processing encountered too many errors, giving up. Rows: 1; errors: 1; max bad: 0; error percent: 0; reason: invalid, location: gs://<bucket>/target_bigquery/<bucket>/coin_list/extracted_date=2024-03-21/d9bbd335-d9f1-4b03-a911-9caac12f3870.jsonl.gz, message: Error while reading data, error message: JSON parsing error in row starting at position 0: JSON object specified for non-record field: platforms File: gs://<bucket>/target_bigquery/<bucket>/coin_list/extracted_date=2024-03-21/d9bbd335-d9f1-4b03-a911-9caac12f3870.jsonl.gz name=meltano run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1
2024-03-21T17:01:03.730885Z [info     ] ELT could not be completed: Loader failed. cmd_type=elt name=meltano run_id=ec6959fd-26da-4c3f-b9c5-8a5c1b8a6977 state_id=tmp1 stdio=stderr

The error with th.CustomType({"anyOf": [{"type": "object"}, {"type": "array"}, {}]}) is:

Traceback (most recent call last): cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.966724Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/bin/target-bigquery", line 8, in <module> cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.966804Z [info     ]     sys.exit(TargetBigQuery.cli()) cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.966869Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/click/core.py", line 1157, in __call__ cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.967063Z [info     ]     return self.main(*args, **kwargs) cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.967175Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/click/core.py", line 1078, in main cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.967255Z [info     ]     rv = self.invoke(ctx)      cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.967326Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/click/core.py", line 1434, in invoke cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.967392Z [info     ]     return ctx.invoke(self.callback, **ctx.params) cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.967458Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/click/core.py", line 783, in invoke cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.967519Z [info     ]     return __callback(*args, **kwargs) cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.967580Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/singer_sdk/target_base.py", line 578, in cli cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.967638Z [info     ]     target.listen(file_input)  cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.967699Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/singer_sdk/io_base.py", line 34, in listen cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.967771Z [info     ]     self._process_lines(file_input) cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.967833Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/singer_sdk/target_base.py", line 278, in _process_lines cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.967891Z [info     ]     counter = super()._process_lines(file_input) cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.967949Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/singer_sdk/io_base.py", line 78, in _process_lines cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.968008Z [info     ]     self._process_schema_message(line_dict) cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.968066Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/singer_sdk/target_base.py", line 378, in _process_schema_message cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.968126Z [info     ]     _ = self.get_sink(         cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.968183Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/target.py", line 487, in get_sink cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.968245Z [info     ]     return self.add_sink(stream_name, schema, key_properties) cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.968304Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/singer_sdk/target_base.py", line 240, in add_sink cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.968366Z [info     ]     sink = sink_class(         cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.968428Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/gcs_stage.py", line 136, in __init__ cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.968486Z [info     ]     super().__init__(target, stream_name, schema, key_properties) cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.968545Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/core.py", line 306, in __init__ cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.968605Z [info     ]     self.create_target(key_properties=key_properties) cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.968666Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/tenacity/__init__.py", line 289, in wrapped_f cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.968735Z [info     ]     return self(f, *args, **kw) cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.968795Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/tenacity/__init__.py", line 379, in __call__ cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.968857Z [info     ]     do = self.iter(retry_state=retry_state) cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.968922Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/tenacity/__init__.py", line 314, in iter cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.969111Z [info     ]     return fut.result()        cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.969313Z [info     ]   File "/opt/homebrew/Cellar/python@3.9/3.9.18_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 439, in result cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.969392Z [info     ]     return self.__get_result() cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.969459Z [info     ]   File "/opt/homebrew/Cellar/python@3.9/3.9.18_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.969521Z [info     ]     raise self._exception      cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.969582Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/tenacity/__init__.py", line 382, in __call__ cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.969643Z [info     ]     result = fn(*args, **kwargs) cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.969706Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/core.py", line 474, in create_target cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.969768Z [info     ]     self.table.create_table(self.client, self.apply_transforms, **kwargs) cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.969828Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/core.py", line 185, in create_table cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.969885Z [info     ]     self._table = client.create_table( cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.969942Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 797, in create_table cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.970404Z [info     ]     api_response = self._call_api( cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.970479Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 831, in _call_api cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.970536Z [info     ]     return call()              cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.970735Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 293, in retry_wrapped_func cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.970833Z [info     ]     return retry_target(       cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.970909Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 153, in retry_target cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.970976Z [info     ]     _retry_error_helper(       cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.971041Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/google/api_core/retry/retry_base.py", line 212, in _retry_error_helper cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.971104Z [info     ]     raise final_exc from source_exc cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.971164Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 144, in retry_target cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.971226Z [info     ]     result = target()          cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.971288Z [info     ]   File "/Users/melgazar9/scripts/github/personal/tap-coingecko/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/google/cloud/_http/__init__.py", line 494, in api_request cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.971481Z [info     ]     raise exceptions.from_http_response(response) cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
2024-03-21T17:07:14.971643Z [info     ] google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/<project-id>/datasets/<bucket>/tables?prettyPrint=false: Field platforms is type RECORD but has no schema cmd_type=loader name=target-bigquery run_id=bbce8cf8-ebbc-4eff-a4af-b0b42706ca62 state_id=tmp1 stdio=stderr
z3z1ma commented 3 months ago

fixed by #86 I think