search

numberscope / backscope

Numberscope's back end: responsible for getting sequences and other data from the On-Line Encyclopedia of Integer Sequences, pre-processing it (factoring etc), and storing it.
MIT License
1 stars 9 forks source link

when crawling too fast, backend gets stuck on disallowed sequence even when crawling is allowed again #69

Open katestange opened 1 year ago

katestange commented 1 year ago

To reproduce: crawl a lot of sequences (using the API directly from URL bar is fine), until the OEIS complains about you crawling too fast. When this happens, trying the API on the most recent sequence will result in an error in the browser: "Error: Value fetching for {oeis_id} in progress.". (You can verify this by going to oeis.org and asking for something there, and OEIS will say you are crawling too fast, come back later.) Even after the OEIS allows crawling again (as verified by visiting the site after a few minutes pause), the backend seems to be stuck in this state and will not return good data on this sequence. It will keep returning this error indefinitely (even though it continues to work fine on other new sequences, which verifies that the OEIS is talking to it).

gwhitney commented 1 year ago

The underlying problem is in flaskr/nscope/views.py

    if seq.values_requested:
        return LookupError("Value fetching for {oeis_id} in progress.")
    seq.values_requested = True
    db.session.commit()
    # Now try to get it from the OEIS:
    r = requests.get(f"{domain}{oeis_id}/b{oeis_id[1:]}.txt")
    if r.status_code == 404:
        return LookupError(f"B-file for ID '{oeis_id}' not found in OEIS.")
    # Parse the b-file:

because see if it returns that LookupError the database is stuck in the state that it thinks fetching is in progress.