search

markddrake / YADAMU---Yet-Another-DAta-Migration-Utility

A utility for Migrating Data between Oracle, Postgres, MySQL MariaDB, Snowflake. Stage Data from supported database to Amazon S3 and Azure Blob Storage in JSON and CSV Formats
MIT License
17 stars 10 forks source link

Snowflake operations fail with "Request to Snowflake failed" starting with snowflake-sdk 1.9.2 #244

Closed markddrake closed 9 months ago

markddrake commented 11 months ago

Starting with snowflake-sdk 1.9.2 there appears to be a be a limit on the size of the HTTP request associated with executing a snowflake SQL command. If a large amount of data is sent the snowflake-sdk returns a ""Request to Snowflake failed" error, with an underlying statuscode of 413. Yadamu correctly handles this by switching to BinaryInsert mode, but the failure generates extra logging.

markddrake commented 11 months ago

This scenario is now recognized as an 'expected and handled' error condition so the logging is supressed for this scenario. The Binary Insert will split the batch of records in half and retry the opeation until it succeeds, or it identifies a single record which cannot be inserted. At that point trace and data files will be generated for the errant record.