search

dart-lang / dart_ci

Tools used by Dart's continuous integration (CI) testing that aren't needed by Dart SDK contributors. Mirrored from dart.googlesource.com/dart_ci. Do not land pull requests on Github.
BSD 3-Clause "New" or "Revised" License
18 stars 5 forks source link

Make cloud functions faster by storing changes in parallel #69

Closed athomas closed 4 years ago

athomas commented 4 years ago

This should allow us to increase the number of results processed/chunk and therefore reduce the number of chunks making it less likely that we'll hit lock contention while updating the processed_chunks.

whesse commented 4 years ago

We are still hitting lock contention on the update of processed_chunks. Proceeding with the solution that will write a separate document for each chunk received, with its message ID, and that will check to see if we have received all chunks by querying for all these documents for a build. First CL to start implementation will send list of all message IDs with the final chunk: https://dart-review.googlesource.com/c/sdk/+/134524

athomas commented 4 years ago

I would suggest we postpone further work on the lock contention issue. This has become rather infrequent now. The example you mentioned was from a single CL. The execution time graphs for the cloud functions suggest that we can increase the size of the requests per message further, and it will be even less likely.

I'm closing this issue since the parallel requests are landed and working in production.