googleapis / python-bigquery-sqlalchemy

SQLAlchemy dialect for BigQuery
MIT License
436 stars 130 forks source link

Add JSON type compiler support #1130

Open SandeepAkode opened 2 weeks ago

SandeepAkode commented 2 weeks ago

Fixes #399 🦕

This commit handles JSON as column type. Currently it throws error for that and we can't use str() method on that column like column.type.

chalmerlowe commented 2 weeks ago

Would it be possible for you to provide an example output based on having this new code?

i.e. being able to write sqlalchemy-based Python code with this new capability would allow us to generate SQL that looks like what?

A simple example would suffice so that I can quickly picture what we are trying to achieve that isn't currently possible?

I am in the middle of a couple of hot tasks, so I won't be able to come back to this for a day or two, but I am interested in getting this pushed through.

SandeepAkode commented 2 weeks ago

Here's an example. I created a customers_info column

ALTER TABLE
  `test_db.test_table` ADD COLUMN customer_info STRUCT<preferences JSON>; 

I created a simple sqlalchemy engine and inspector. Then I got columns using inspector.get_columns('test_db.test_table') When we check the type of the column we added

>>> column['type']
STRUCT(preferences=NullType())

Ideally it should be STRUCT(preferences=JSON())

Also, if we try to use str(column['type']) it will throw an error

sqlalchemy.exc.CompileError: Can't generate DDL for NullType(); did you forget to specify a type on this Column?

This is required as we might need to store column type as a string.

Thanks.

chalmerlowe commented 3 days ago

@SandeepAkode

Thank you very much for your interest in this project and for your suggested change. Before we can approve and merge this PR, we will need to add tests. Is that something you are comfortable doing? If not that is fine, we have several options. If you want to add the tests but are not sure what that entails, we can provide guidance along the way. If you really don't feel comfortable, then let us know and we can put this into our backlog of tasks.

chalmerlowe commented 3 days ago

This is failing this test:

File "/tmpfs/src/github/python-bigquery-sqlalchemy/tests/unit/test_dialect_types.py", line 30, in test_types_import
    assert getattr(dialect_module, type_name) == type_value
AttributeError: module 'sqlalchemy_bigquery' has no attribute 'JSON'|