Fixed Error with Pandas Update

JohnOmernik / sqlalchemy-drill

Apache Drill Dialect for SQL Alchemy

Other

53 stars 35 forks source link

Fixed Error with Pandas Update #39

Closed cgivre closed 4 years ago

cgivre commented 4 years ago

This PR addresses some errors that occur with the latest versions of Superset and Pandas. In a nutshell, the dialect was attempting to infer datatypes from JSON when datatypes were available via column metadata.

This PR removes the inference and passes the column metadata directly from Drill.

JohnOmernik commented 4 years ago

So this is due to changes in Pandas correct? Should we set/update the requirements to be pandas >=1.0?

cgivre commented 4 years ago

I don't think we should update the requirements to use Pandas 1.0 yet. I think Superset was causing some errors here, but I'm not really sure. I was getting ready to demo it for someone tomorrow and foolishly decided to update...

As an FYI, the intent for Superset is to move away from Pandas entirely and use PyArrow as a back end. Which will be interesting in that Drill is also working on adding some support for Arrow.

JohnOmernik commented 4 years ago

No worries, I had some issues @dayjob with Pandas moving to 1.0. there are some deprecated functions that got removed, so ya, I am good with merging this, I just wanted to know if we should pin it.