search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.04k stars 4.11k forks source link

[Source-BigQuery] Cannot query over table #37025

Open edwardrpool opened 6 months ago

edwardrpool commented 6 months ago

Connector Name

source-bigquery

Connector Version

0.4.2

What step the error happened?

During the sync

Relevant information

If you are using a partitioned table with "Partition filter: Required" enabled, you do not have the ability to set a filter or increment based on the filter.

With larger data sets, it's normal to partition the data as it comes in for query efficiency, and reduce costs.

I have over 300m records, that are all partitioned by event_date, and it would make sense to be able to both be able to choose a start date, and to set an increment set range for the date that Airbyte can use to loop through all the data.

Relevant log output

Cannot query over table 'dataset.tablename' without a filter over column(s) 'column_name' that can be used for partition elimination

Contribute

marcosmarxm commented 6 months ago

Thanks for reporting the issue @edwardrpool Source Bigquery is a community connector and it isn’t in the current roadmap for improvements. If you want to contribute fixing the issue please reach me out in Slack so I can provide you instructions to make the contribution

fortunaphantom commented 6 months ago

@edwardrpool Interesting. How can I reproduce it?

edwardrpool commented 6 months ago

@fortunaphantom Fairly easily. If you have or create a table in BigQuery with a partition, and set the partition filtering to enforced, you will see the error. This is because we do not have the ability to set the queries WHERE statement.

octavia-squidington-iii commented 2 weeks ago

At Airbyte, we seek to be clear about the project priorities and roadmap. This issue has not had any activity for 180 days, suggesting that it's not as critical as others. It's possible it has already been fixed. It is being marked as stale and will be closed in 20 days if there is no activity. To keep it open, please comment to let us know why it is important to you and if it is still reproducible on recent versions of Airbyte.