search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.4k stars 3.97k forks source link

new dagger run being attached to old dagger cloud run #28179

Open c-p-b opened 1 year ago

c-p-b commented 1 year ago

We ran into an issue with dagger cloud today that seems kind of sus that may have some significant implications. We are noticing some extremely strange behavior with some of our dagger runs. Specifically:

This PR: https://github.com/airbytehq/airbyte/pull/27845 We invoke a publish command manually: https://github.com/airbytehq/airbyte/actions/runs/5526271207/jobs/10080820839

This spawns two dagger jobs, as expected. The first you can see here:

https://dagger.cloud/runs/88cf5710-bec3-45e3-9f62-579aab0c6cf1

The second we had to dig into the logs to get, which is where things get REALLY weird

https://alpha.dagger.cloud/runs/721f55db-a603-42ba-9fde-f16299eb7579

This is pointing at a totally separate pull request from the one we launched things off on, with massive negative timestamps for things. The only pipelines we should see here should be related to destination-bigquery. We are also noticing that the published image produced has some discrepancies from the pull request, but we haven't established whether or not that could be related to this. Under what circumstances might something like this happen? I'm a bit worried that there may not be proper isolation of runs if something like this is happening, and it would be nice to get a confirmation that it's at least only related to how things are uploaded to cloud and only a visualization issue

c-p-b commented 1 year ago

For reference, @edgao ran the same pipeline locally and this is what it should look like: https://alpha.dagger.cloud/runs/849a07e2-abbd-4560-a563-1bbe50fcf732

c-p-b commented 1 year ago

We were able to reproduce the issue on this branch at this commit: https://github.com/airbytehq/airbyte/pull/28250 https://github.com/airbytehq/airbyte/pull/28250/commits/57422c1c6e145179afcbafeb55a929b92bc32e81 Publish job: https://github.com/airbytehq/airbyte/actions/runs/5536466899 Cloud url: https://dagger.cloud/runs/b36a6b86-c541-40c0-a544-268ce4f6ef31

jpadams commented 1 year ago

@cpdeethree are we good to close this issue now?

c-p-b commented 1 year ago

Can we confirm that a change was pushed on the dagger end that addressed the root cause of this issue and if so, point at that change?