Gemma-Analytics / ewah

ELT With Airflow Helper - Classes and functions to make apache airflow life easier
MIT License
12 stars 2 forks source link

Add UUID to str conversion + add UUID to QBC type mappings #246

Closed lpillmann closed 9 months ago

lpillmann commented 9 months ago

Context

Postgres ingestion was failing with error

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/ewah/operators/base.py", line 502, in execute
    self.ewah_execute(context)
  File "/home/airflow/.local/lib/python3.8/site-packages/ewah/operators/sql_base.py", line 116, in ewah_execute
    self.upload_data(batch)
  File "/home/airflow/.local/lib/python3.8/site-packages/ewah/operators/base.py", line 598, in upload_data
    return self.uploader.upload_data(data, metadata)
  File "/home/airflow/.local/lib/python3.8/site-packages/ewah/uploaders/base.py", line 128, in upload_data
    if not pk_name in self.columns_definition.keys():
  File "/home/airflow/.local/lib/python3.8/site-packages/ewah/uploaders/base.py", line 105, in columns_definition
    return self.cleaner.get_columns_definition(dwh_engine=self.dwh_engine)
  File "/home/airflow/.local/lib/python3.8/site-packages/ewah/cleaner.py", line 276, in get_columns_definition
    raise Exception(
Exception: Field 'uuid' has an invalid data type '<class 'uuid.UUID'>'!

Which meant UUID type is not currently supported.

Changes