search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.68k stars 4.02k forks source link

[destination-azure-blob-storage] File spill size not being used when writing CSV format #34714

Open ttilberg opened 8 months ago

ttilberg commented 8 months ago

Connector Name

destination-azure-blob-storage

Connector Version

0.2.1

What step the error happened?

During the sync

Relevant information

Azure Blob Storage file spill size does not seem to be used when writing CSV files to Azure. In the current version, 0.2.1, if you specify CSV format with this option, the file does not get split. It appears that this PR implemented "Azure Blob Storage file spill size", but may only affect JL output format.

If it is not reasonable to spill to new files for CSV data, documentation in the configuration UI should specify the "Azure Blob Storage file spill size" is only available for JL format.

Relevant log output

No response

Contribute

marcosmarxm commented 6 months ago

Hello @ttilberg ! Destination Azure Blob Storage is a vital part of our community, but it's not currently on our roadmap for updates. This means it might take some time before it gets prioritized by the Airbyte Team. However, we encourage community involvement to improve it, and your contributions are welcome! If you're interested, please reach out to me on Slack so we can discuss how you can help.

octavia-squidington-iii commented 1 week ago

At Airbyte, we seek to be clear about the project priorities and roadmap. This issue has not had any activity for 180 days, suggesting that it's not as critical as others. It's possible it has already been fixed. It is being marked as stale and will be closed in 20 days if there is no activity. To keep it open, please comment to let us know why it is important to you and if it is still reproducible on recent versions of Airbyte.