lukasschwab / arxiv.py

Python wrapper for the arXiv API
MIT License
1.07k stars 120 forks source link

Empty Response after few requests #158

Closed mayukh4 closed 6 months ago

mayukh4 commented 6 months ago

`import arxiv

def fetch_astrophysics_papers(category, search_terms, num_papers=5, summarize=False): keywords = [term.strip() for term in search_terms.split(',') if term.strip() != ''][:3] # Limit to 3 terms query_parts = [f'(title:"{keyword}" OR abs:"{keyword}")' for keyword in keywords] search_query = f'cat:astro-ph.{category} AND ({" OR ".join(query_parts)})'

# Fetch the papers from arXiv
search = arxiv.Search(
    query=search_query,
    max_results=min(num_papers, 50),  # Ensure num_papers does not exceed 50
    sort_by=arxiv.SortCriterion.SubmittedDate,
)'

So this is a part of a bigger code, essentially fetching arxiv papers. Recently, I have been facing an issue, where after running this code for 5-6 times I am getting empty responses. NOTE : at a time I am only fetching a maximum of 5 papers. The issue gets resolved after some time, or If I change my IP. It looks like I am reaching some kind of limit with just these limited requests. I don't know what I am doing wrong.

You should be able to reproduce this if you run it 5-6 times (let me know if you can).

lukasschwab commented 6 months ago

See #157, #129 — seems there's ongoing instability in the underlying API. This wrapper can't solve it.

mayukh4 commented 6 months ago

Yes, I did see some of the existing issues, but thought to ask out anyway! Thanks.