search

lukasschwab / arxiv.py

Python wrapper for the arXiv API
MIT License
1.11k stars 123 forks source link

Debug flaky CI integration tests #127

Closed lukasschwab closed 1 year ago

lukasschwab commented 1 year ago

Description

CI tests are failing for latest commit on master. I'm unable to reproduce locally. Debugging.

Breaking changes

List any changes that break the API usage supported on master.

Relevant issues

List GitHub issues relevant to this change.

Unexpected test failures:

Checklist

lukasschwab commented 1 year ago

Current theory: flakiness is caused by rate limiting.

lukasschwab commented 1 year ago

Even with all these changes (slowing things waaay down) the build is flaky.

I'm curious: can I up the parallelism again?

lukasschwab commented 1 year ago

This conclusion sucks, but I think I'm trying to adapt the tests to accommodate temporarily flaky API behavior. As I've poked around, the behavior has been unstable (e.g. earlier only HTTPS worked; now only HTTP works).

It's possible this is driven by arXiv is tightening their rate limiter behavior... but that only explains this if they enforce the rate limit through a variety of unclear responses (TCP hangup vs. empty result set vs...).

I might consider simplifying this PR to two fixes:

  1. Hard-pinning the feedparser version.
  2. Consolidating to HTTPS.
lukasschwab commented 1 year ago

3e55127 might've just been lucky.

lukasschwab commented 1 year ago

Closing; PRs in the v2.0.0 release handle the best changes explored here.