search

pr-omethe-us / PyKED

Python interface to the ChemKED database format
https://pr-omethe-us.github.io/PyKED/
BSD 3-Clause "New" or "Revised" License
14 stars 15 forks source link

Why is the test suite so slow? #86

Closed bryanwweber closed 6 years ago

bryanwweber commented 6 years ago

On the recent commits (e.g., https://travis-ci.org/pr-omethe-us/PyKED/builds/285888781) the Travis tests take almost 10 minutes, and I'm seeing a similar problem on my local machine. This shouldn't be the case (or at least, I don't think it used to be the case), so what's going on?

bryanwweber commented 6 years ago

We're also starting to get 500 errors from the DOI lookup... I wonder if there's either a problem with Habanero, or if its rate-limited.

kyleniemeyer commented 6 years ago

Agreed, I've also noticed these are slow... not sure why.

bryanwweber commented 6 years ago

Based on the jobs here: https://travis-ci.org/pr-omethe-us/PyKED/builds/287938661 it seems like a significant fraction of the time is spent in the SSL library, presumably trying to connect to the CrossRef servers. Those jobs only took ~3 minutes, not quite as long as some other recent jobs. Might just be an intermittent connection issue with CrossRef.

bryanwweber commented 6 years ago

We might be able to create a single Request object from Habanero and reuse it. This might help with some of the rate-limiting or the repeated handshakes that are required with the API website.

Should probably also consider the CrossRef API etiquette: https://github.com/CrossRef/rest-api-doc#etiquette

And, they made a recent change to their TOS for "polite" users: https://github.com/CrossRef/rest-api-doc#good-manners--more-reliable-service

Basically, we need to add a mailto in the HTTP headers we send to Crossref. This should be possible to do from within Habanero, so let's look into that.