search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.16k stars 4.13k forks source link

[airbyte-ci] Skip building connector image if its source code is not changed #48408

Open theyueli opened 6 days ago

theyueli commented 6 days ago

Topic

airbyte-ci

Relevant information

I'm currently modifying our acceptance tests which requires me to repeat running the tests using the following airbyte-ci command, for instance

airbyte-ci connectors --name=source-mysql test -k acceptance

My expectation is that if I do not modify our connector source code, the command would only execute the acceptance test code and spec that I'm modifying.

In reality, the command always spend minutes rebuilding the connector image, which forces to waste compute resource and developer's time. I would

postamar commented 1 day ago

Your expectation is entirely reasonable and in fact is exactly what dagger is supposed to do.

It's possible that the container layer containing the connector image is getting GCed by buildkit. This is something that @alafanechere and myself spent many hours on about a year ago. Evidently this is worth looking into again. At the time, we even ended up looping in the dagger people.