Auto batch and error handling query pubtator

JohnGiorgi / seq2rel-ds

This is a companion repository to seq2rel (https://github.com/JohnGiorgi/seq2rel) which aims to make it easy to generate training data.

5 stars 1 forks source link

Closed JohnGiorgi closed 3 years ago

JohnGiorgi commented 3 years ago

This PR updates the query_pubtator function so that it:

automatically batches its post requests (into chunks of size pmids_per_request). This dramatically speeds things up.
handles connection errors by recursively halving the size of the request. This prevents us from dropping PMIDs from the output*.

*I don't actually know why this trick prevents error, but I tested that it does.