search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.19k stars 4.14k forks source link

Destination snowflake: load failed from stage, but sync status is succeded #27396

Open andrevus opened 1 year ago

andrevus commented 1 year ago

Connector Name

destination-snowflake

Connector Version

1.0.5

What step the error happened?

During the sync

Revelant information

Steps to reproduce: Some of records from source table are larger than 16mb. Making connection from this database to snowflake, airbyte will put this data in a stage and makes copy into table from stage, and on this query we will get LOAD_FAILED with information "Max LOB size (16777216) exceeded, actual size of parsed column is xxxx". Data will only exists on stage, query status is success, airbyte sync is success but destination table in snowflake will be empty or missing all data from affected file from stage.

I think that connector should use VALIDATION_MODE = 'RETURN_ERRORS' in COPY INTO MYTABLE statement to prevent this.

Relevant log output

No response

Contribute

kevin868 commented 1 year ago

Related issue: https://github.com/airbytehq/airbyte/issues/27124

miquelmartorell commented 6 months ago

Hi everyone, any fix solution to this? Desde Airbyte te dice que todo ha ido bien, sin embargo entras en Snowflake y ves que ha fallado en la etapa de COPY INTO y es deseperante. Alguna manera de solucionar esto? LOAD_FAILED,1,0,1,1,"Max LOB size (16777216) exceeded, actual size of parsed column is 25610863",1,28890242,

Edit: Stills present in 3.8.0 version Snowflake connector

octavia-squidington-iii commented 2 days ago

At Airbyte, we seek to be clear about the project priorities and roadmap. This issue has not had any activity for 180 days, suggesting that it's not as critical as others. It's possible it has already been fixed. It is being marked as stale and will be closed in 20 days if there is no activity. To keep it open, please comment to let us know why it is important to you and if it is still reproducible on recent versions of Airbyte.