search

z3z1ma / target-bigquery

target-bigquery is a Singer target for BigQuery. It supports storage write, GCS, streaming, and batch load methods. Built with the Meltano SDK.
MIT License
27 stars 34 forks source link

Interactive CLI for `upsert` option saves as a string, but needs to be a boolean #93

Open JamesStratford opened 1 month ago

JamesStratford commented 1 month ago

When using meltano config tap-bigquery set --interactive, if you set upsert to true, it will save as a string and the following error occurs

2024-07-15T03:31:26.531723Z [info     ]   File "/workspaces/meltano/meltano-xxx/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/referencing/jsonschema.py", line 56, in _legacy_dollar_id cmd_type=elb consumer=True job_name=dev:tap-tiktok-to-target-bigquery name=target-bigquery producer=False run_id=b0731a91-d41a-4027-9540-c1ac834ea862 stdio=stderr string_id=target-bigquery
2024-07-15T03:31:26.531817Z [info     ]     id = contents.get("$id")   cmd_type=elb consumer=True job_name=dev:tap-tiktok-to-target-bigquery name=target-bigquery producer=False run_id=b0731a91-d41a-4027-9540-c1ac834ea862 stdio=stderr string_id=target-bigquery
2024-07-15T03:31:26.531901Z [info     ]          ^^^^^^^^^^^^          cmd_type=elb consumer=True job_name=dev:tap-tiktok-to-target-bigquery name=target-bigquery producer=False run_id=b0731a91-d41a-4027-9540-c1ac834ea862 stdio=stderr string_id=target-bigquery
2024-07-15T03:31:26.532010Z [info     ] AttributeError: 'str' object has no attribute 'get' cmd_type=elb consumer=True job_name=dev:tap-tiktok-to-target-bigquery name=target-bigquery producer=False run_id=b0731a91-d41a-4027-9540-c1ac834ea862 stdio=stderr string_id=target-bigquery

This is what the YAML produces

  - name: target-bigquery
    variant: z3z1ma
    pip_url: git+https://github.com/z3z1ma/target-bigquery.git
    config:
      dataset: '${BQ_DATASET}'
      project: REDACTED
      location: REDACTED
      denormalized: true
      options:
        storage_write_batch_mode: true
      method: batch_job
      upsert: 'True'

Manually changing it to just the boolean true fixes the error

AlejandroUPC commented 1 month ago

Mmm interesting, I have not taken a look or debugged my self but it looks like it just gets the value from a dictionary, if not found is False otherwise uses the value, which should be evaluated as true in Python?

    def _is_upsert_candidate(self) -> bool:
        """Determine if this stream is an upsert candidate based on user configuration."""
        upsert_selection = self.config.get("upsert", False)
        upsert_candidate = False
        if isinstance(upsert_selection, list):
            selection: str
            for selection in upsert_selection:
                invert = selection.startswith("!")
                if invert:
                    selection = selection[1:]
                if fnmatch(self.stream_name, selection):
                    upsert_candidate = True ^ invert
        elif upsert_selection:
            upsert_candidate = True
        return upsert_candidate

https://github.com/z3z1ma/target-bigquery/blob/main/target_bigquery/core.py#L383C9-L383C25