googlebot
commented
7 years ago Thanks for your pull request. It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).
:memo: Please visit https://cla.developers.google.com/ to sign.
Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.
- If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
- If you signed the CLA as a corporation, please let us know the company's name.
KotaiVictor
dhalperi
We ran into an issue in production recently where one of our workers was stalling on a particular hotkey.
Looking further into the issue we figured out that the given our row sizes we were inserting somewhere between 5 and 20 elements per call.
This coupled with our windowing size of around 5000 and the fixed number of 100 threads the BigQueryTableInserter uses to do inserts made it slow down.
We thought it would be useful to be able to configure this.
Ideally, we'd be able to set the thread count for the Executor too. Give that it's static, you'll be able to do at most 100 inserts in parallel from a worker.