search

cambialens / lens-api-doc

10 stars 6 forks source link

DOI request limit #7

Closed zilch42 closed 5 years ago

zilch42 commented 5 years ago

Hi,

Just wondering if there is a limit to the number of DOIs I can query at a time using the API? I'm not expecting to get all of the queried publications back as I'm using has_patent_citations = true.

Sending all 704k of them returned a 500 error, but sending 5000 was ok, so guess it's somewhere in the middle. I'm hoping to minimise the total number of calls to the API so I don't hit my monthly limit.

Cheers

denizk commented 5 years ago

Hi @zilch42

Depending on how you're formulating your query, you might hit the default limit of 65,536 terms in a terms query.

zilch42 commented 5 years ago

Thanks @denizk. That's great.

It's easy to break my query up so its good to know what the limit is.

Thanks.

zilch42 commented 5 years ago

Just to give some feedback, I tried using a set size of 60,000 and got a error code 500. I reduced it to 30,000 and it was ok. I managed to successfully iterate through almost 900,000 DOIs using chunks that size.

So there may be something else limiting the size of the request, before the 65,536 term limit is reached.

Thanks

denizk commented 5 years ago

Thanks @zilch42 for the follow up. We had an additional payload limit in place which is now removed. Feel free to give it another go.