The SyncStats in the UI should summarize what happened across all attempts.

[gosusnp]

Context

The UI is showing stats from the last attempt. This was correct but since we improved our checkpointing, those stats no longer reflect what has been moved for that run. For example, if we moved 80 records in the first attempt, then 20 in the second one, we'll be showing 20 when we in fact moved 100 records.

Proposal

For a given set of attempts:

• records/bytes emitted should be the sum of records/bytes emitted across all attempts
• records/bytes committed should be the sum of
  • the sum all records/bytes committed for incremental streams
  • the records/bytes committed of the last attempt that processed the stream for full refresh streams

[josephkmh]

Here's an example of a partially successful job that ran 10 attempts: https://cloud.airbyte.com/workspaces/bed3b473-1518-4461-a37f-730ea3d3a848/connections/2beb25a4-4dea-415f-8c08-f0dbc88f04c3/job-history#4222645::10

[josephkmh]

Refining notes:

• Syncs can have a mix of incremental and full refresh, which makes it hard to calculate a total amount of bytes moved. We need the catalog to determine which streams are which.

[davinchia]

@josephkmh should this actually be an update to the backend endpoint? Want to learn more about how Compose is thinking about this.

[josephkmh]

@davinchia I had a short chat with @gosusnp in SF last week and IMO we should add this to the backend endpoint rather than calculating on the frontend. It would require patching together data from multiple endpoints on the FE which feels brittle. I'll bring it up in our refinement session and see if there are any other ideas.

[davinchia]

Great! That was my intuition too. Will Compose handle this then? If so, I'll remove this from Move.

Move is happy to work on this - just asking to prevent too many cooks in the kitchen.

[pmossman]

@davinchia I think Move can take it - I don't see Compose having bandwidth for this any time soon.

[nataliekwong]

Adding some feedback from our lovely Support team, where adding up the time across all attempts helps with debugging and assisting customers. I've also run into this issue when trying to help customers to try to understand when jobs were actually kicked off and how long it really took to succeed.

For multi-attempt syncs, the sync summary only shows data from the last attempt. It’s tricky to find out actually how long a sync took without going into the logs and adding up all 10 attempts.

cc @amaliaroye

[nataliekwong]

I was thinking about this some more and have reservations around summing all records/bytes emitted. We could take the example we were looking at earlier this week, where records were emitted in all attempts, but not committed: https://cloud.airbyte.com/workspaces/8435ec6c-de4e-4de6-a128-36b8353f3333/connections/084cb796-1fa6-43b5-a64d-5a754c109aa2/job-history#5455302::0

I felt the solution deserved more detail than a GitHub comment, so I wrote up a short doc with tradeoffs of each option here: https://docs.google.com/document/d/1KMywQrD94UBOMCA3469wdjhUUXB73UmGgXVsZ-N9Igg/edit. Not holding extremely strong opinions here, as there could be context I am missing.

cc @alex-gron @AnilEMaharjan in case you have any opinions on how we sum up records across attempts.

And lastly, we should add to this issue the summarization of time elapsed. I think that one is an easy solve of summing across all attempts.

[nataliekwong]

Once we summarize these correctly, I'm also thinking we would no longer need to show the "Last attempt:" here:

Someone can always look into the logs and see the attempts shown across the top of the viewer.

[nataliekwong]

Related issue: We don't show any stats where there is no sync summary for the latest attempt. It will simply say "Sync Partially Succeeded" with no stats. I'm hoping that summarizing the attempts will resolve this issue.

https://cloud.airbyte.com/workspaces/d4ace984-bdbf-4664-a507-73a9c3978be3/connections/e85fc3ca-d425-47c4-a21b-86d2ece41f8e/job-history

[nataliekwong]

@davinchia Updated the doc to reflect the comments here and the feedback in the doc.

[nataliekwong]

Related community issue: https://github.com/airbytehq/airbyte/issues/33887 The sync summary will non-deterministically report stats from one of the attempts.

[davinchia]

@jpefaur is tackling this work right now.

[davinchia]

@josephkmh I'm assigning this issue to you since the backend work is done. Lmk if that is mistaken!

[nataliekwong]

Actually - I think we can close this. I created a separate issue for Frontend here that is specific to the attempts/Frontend improvements! https://github.com/orgs/airbytehq/projects/42/views/7?pane=issue&itemId=51950634

[davinchia]

ok!

[nataliekwong]

Linking relevant PRs in case they come in handy: https://github.com/airbytehq/airbyte-platform-internal/pull/10862