search

Snowflake-Labs / django-snowflake

MIT License
59 stars 15 forks source link

Add support for JSONField #23

Closed timgraham closed 1 year ago

timgraham commented 2 years ago

I believe VARIANT is the data type to use with Django's JSONField, however, Snowflake requires using INSERT INTO ... SELECT to insert data, which I'm not sure Django has hooks to handle.

There's a snowflake-connector-python issue (https://github.com/snowflakedb/snowflake-connector-python/issues/244) to add support for INSERT INTO... (without SELECT).

javedsyedgithub commented 1 year ago

@timgraham we need support added for variant, array list and last insert object for Snowflake in the django project. Our deployment is at standstill without these changes. Could you take this on priority?

javedsyedgithub commented 1 year ago

@timgraham we need this support added to build our connectors for Snowpark and to help our customers moved to Snowflake who are using Django. We can't move forward until this ticket is resolved and django community is a wide community that Snowflake can't afford to loose.

javedsyedgithub commented 1 year ago

@timgraham do you have any update on this request? We and our customers can't move forward with Snowflake onboarding as this is the blocker. We need this issue fix ASAP for us to keep moving for building our app on the Snowflake platform. We are building our connectors on Snowpark and need to host our app through django with Snowflake as a database and this will help our customers have Snowflake as the underline database for all of our subscription. We really need to get this done sooner than later. Would you able to provide an ETA on this? We need this by next week otherwise unfortunately we need to switch our tech stack from Snowflake as it seems we are not getting the support that we should have.

timgraham commented 1 year ago

I'm not sure it's possible to implement without https://github.com/snowflakedb/snowflake-connector-python/issues/244 which requires non-trivial server-side changes according to @sfc-gh-mkeller. I'll post an update if I find a workaround for this project or hear an ETA on https://github.com/snowflakedb/snowflake-connector-python/issues/244.

timgraham commented 1 year ago

There appears to be hope to implement this without changes in Snowflake and Snowflake Connector Python.

As suggested in https://github.com/snowflakedb/snowflake-connector-python/issues/244#issuecomment-561462414, I'm overriding Django's SQLInsertCompiler to generate queries of the form insert into t(c1,c2,c3) select column1, column2, parse_json(column3) from values(%s, %s, %s) when JSONField is involved.

Now I'm working through Django's model_fields.test_jsonfield, implementing the various query lookups. I'll update on the progress sometime next week.

javedsyedgithub commented 1 year ago

@timgraham thanks for the information, I am looking forward for a fix on this

javedsyedgithub commented 1 year ago

@timgraham do you have any update on this?

timgraham commented 1 year ago

The pull request linked above your comment is mostly completed. I've been waiting for some feedback from Snowflake engineering on a few remaining questions.

javedsyedgithub commented 1 year ago

Hello Cedar-Team/Django-Snowflake,

Is this completed?