CartoDB / raster-loader

https://raster-loader.readthedocs.io
Other
15 stars 4 forks source link

Feature/sc 377004/add raster loader support for snowflake #127

Closed volaya closed 9 months ago

volaya commented 10 months ago

Proposed Changes

This PR adds support for Snowflake

A refactoring of the main scripts has also been made, to leave a cleaner and more extensible code, which allows to reuse code between platforms.

shortcut-integration[bot] commented 10 months ago

This pull request has been linked to Shortcut Story #377004: Add raster-loader support for Snowflake.

vdelacruzb commented 10 months ago

I'm struggling still to achieve loading a raster into snowflake. When executing the upload command I receive a crash because of the snowflake python connector not being installed. I've looked for related threads and found this it seems that we shouldn't have any file called snowflake.py. I've tried locally to replace file or class name called snowflake by other names and I don't get that error.

*UPDATE: I've tried removing my previous builds and so on and it seems that even keeping the name snowflake.py the installation is detected.

Also got the next error when deleting a partially uploaded table:

Error: Error uploading to Snowflake: 090105 (22000): Cannot perform CREATE TEMPSTAGE. This session does not have a current database. Call 'USE DATABASE', or use a qualified name.

So we might have to set the database as default when uploading the table.

*Finally managed to upload a raster it required setting a default DATABASE and SCHEMA in the client constructor. Also in my case it required to write the destination table name in upper_case.

vdelacruzb commented 10 months ago

I've just noticed that in the ticket the parameter --role was included:

carto snowflake upload \
  --file_path /path/to/my/raster/file.tif \
  --account my-account \
  --database my-database \
  --schema my-schema \
  --table my-table \
  --role my-role \
  --user my-user \
  --password $MY_PASSWORD \
  --overwrite

It would be nice to include it as it will define who is the owner of the table in snowflake once created.

vdelacruzb commented 9 months ago

I've done some changes since I've discovered that the chunks are getting constantly overwritten. That's why I was getting too small table results during the QA.

Jesus89 commented 9 months ago

Let's release 0.5.0 after merging this PR