[destination-GCS] Overwrite does not work properly using variables in GCS Bucket Path

[kev-datams]

Connector Name

destination-google-cloud-storage

Connector Version

0.4.4

What step the error happened?

During the sync

Revelant information

Hello,

The GCS destination does not properly apply the Overwrite strategy in certain circumstances. I mean the existing files are not removed from GCS path before the sync, leading to duplicates files after the sync.

In GCS destination configuration settings, while filling GCS Bucket Path value:

• with a simple path like my_folder/my_subfolder : Overwrite strategy works as expected
• with a dynamic path like my_folder/${STREAM_NAME}: Overwrite strategy does not work as expected as explained above.

Reproducibility is really simple:

1. set up a GCS destination with simple path in GCS Bucket Path value
2. create a connection using this GCS destination (source does not matter) using Overwrite strategy
3. sync the connection => a file will be generated as expected
4. re-sync the connection => the file will be dropped first, then re-generated as expected
5. update the GCS destination GCS Bucket Path value with a dynamic path
6. re-sync the connection => the file will NOT be dropped first, then re-generated, leading to duplicates

Thank you for your help 👍

Relevant log output

No response

Contribute

• [ ] Yes, I want to contribute

[kev-datams]

Can someone add the label connectors/destination/gcs to this issue please ?

[kev-datams]

Hi, looks like this issue is still open

[octavia-squidington-iii]

At Airbyte, we seek to be clear about the project priorities and roadmap. This issue has not had any activity for 180 days, suggesting that it's not as critical as others. It's possible it has already been fixed. It is being marked as stale and will be closed in 20 days if there is no activity. To keep it open, please comment to let us know why it is important to you and if it is still reproducible on recent versions of Airbyte.

[kev-datams]

At Airbyte, we seek to be clear about the project priorities and roadmap. This issue has not had any activity for 180 days, suggesting that it's not as critical as others. It's possible it has already been fixed. It is being marked as stale and will be closed in 20 days if there is no activity. To keep it open, please comment to let us know why it is important to you and if it is still reproducible on recent versions of Airbyte.

Hi, it is still an open issue to be fixed, thanks

[kev-datams]

Could you remove the Stale label ?