mozilla / docker-etl

Collection of dockerized ETL jobs managed by data engineering.
Mozilla Public License 2.0
16 stars 14 forks source link

Add chunking to avoid long GET request to bugzilla #260

Closed ksy36 closed 1 month ago

ksy36 commented 1 month ago

We pass bugzilla ids as a parameter to the GET request as a a string. There are more than 1k bugs and the length of this string started exceeding 8k characters last week. The server stopped returning anything for this one request and the import didn't proceed further (I think 8k is the limit). This PR chunks the request in 400 ids in each chunk.

Checklist for reviewer:

ksy36 commented 1 month ago

r? @jgraham

ksy36 commented 1 month ago

A good longer-term refactor might be to move the bugzilla logic into a helper class, but I think this is enough to fix the bug for now.

Thanks! Yeah, that's a good idea.

ksy36 commented 1 month ago

@scholtzan could you merge this, please?

ksy36 commented 1 month ago

thank you :)