search

move-coop / parsons

A python library of connectors for the progressive community.
Other
255 stars 125 forks source link

Fix nullable column type parsing by BigQuery connector #958

Closed austinweisgrau closed 6 months ago

austinweisgrau commented 6 months ago

If a Parsons Table column has values like [None, None, True, False], the BigQuery connector will infer that the appropriate type for this column is NoneType, which it will translate into a STRING type.

This change ensures that types returned by petl.typecheck() will choose the first available type that isnt NoneType if that is available.

Adds a test that checks if the connector is able to successfully parse a column type when nulls are included. First commit here will allow test to run and fail, second commit will make the fix and demonstrate the test succeeds.

austinweisgrau commented 6 months ago

Closing this in favor of consolidated fixes in #959