Add chunking to avoid long GET request to bugzilla

ksy36 commented 3 months ago

We pass bugzilla ids as a parameter to the GET request as a a string. There are more than 1k bugs and the length of this string started exceeding 8k characters last week. The server stopped returning anything for this one request and the import didn't proceed further (I think 8k is the limit). This PR chunks the request in 400 ids in each chunk.

Checklist for reviewer:

[ ] Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title)
[ ] Scan the PR and verify that no changes (particularly to .circleci/config.yml) will cause environment variables (particularly credentials) to be exposed in test logs
[ ] Ensure the container image will be using permissions granted to telemetry-airflow responsibly.

ksy36 commented 3 months ago

r? @jgraham

ksy36 commented 3 months ago

A good longer-term refactor might be to move the bugzilla logic into a helper class, but I think this is enough to fix the bug for now.

Thanks! Yeah, that's a good idea.

ksy36 commented 3 months ago

@scholtzan could you merge this, please?

ksy36 commented 3 months ago

thank you :)

mozilla / docker-etl

Add chunking to avoid long GET request to bugzilla #260