snowflakedb / snowflake-sqlalchemy

Snowflake SQLAlchemy
https://pypi.python.org/pypi/snowflake-sqlalchemy/
Apache License 2.0
231 stars 152 forks source link

SNOW-721174: Support `COPY INTO <location>` with storage integration and gcs bucket #368

Open dahlbaek opened 1 year ago

dahlbaek commented 1 year ago

What is the current behavior?

The COPY INTO <location> statement support gcs buckets, but there is no GCSBucket ClauseElement in https://github.com/snowflakedb/snowflake-sqlalchemy/blob/a84fa77ae4387ed60db036db61dc7d8f82849a98/src/snowflake/sqlalchemy/custom_commands.py. Similarly, the statement supports using a STORAGE_INTEGRATION option, but that option is not supported by CopyInto https://github.com/snowflakedb/snowflake-sqlalchemy/blob/a84fa77ae4387ed60db036db61dc7d8f82849a98/src/snowflake/sqlalchemy/custom_commands.py#L110-L157

What is the desired behavior?

The CopyInto class supports Copy Into <location> statements of the type

COPY INTO '<gcs_bucket>'
FROM (<select>)
FILE_FORMAT = ( TYPE = PARQUET )
STORAGE_INTEGRATION = <gcs_storage_integration>
HEADER = TRUE;

How would this improve snowflake-connector-python?

This would make it possible to use the CopyInto class on order to execute such variants of the COPY INTO <location> statement.

References, Other Background

We're a small company that make use of the snowflake-sqlalchemy library. If someone will give us pointers as to what we should add in terms of classes and tests, we can allocate time to work on a pr.

sfc-gh-dszmolka commented 6 months ago

thank you for submitting this enhancement idea, we'll consider it for future planning. Of course a PR is more than welcome, if there's such possibility from your end.