Get _all_ articles from query

[pournaki]

Thanks for writing this nice piece of software!

I wondering how to use it to get all possible articles given a query. For instance, if I use the example from the documentation

search_df = scopus.search("KEY(interdisciplinary collaboration)", count=20, view='STANDARD')

and leave out count, I only get 100 articles.

If I raise count to something very large, such as 1e10, I get the following error:

KeyError                                  Traceback (most recent call last)
<timed exec> in <module>

/usr/local/lib/python3.8/site-packages/pyscopus/scopus.py in search(self, query, count, type_, view)
     67         while True:
     68             index = 25*i
---> 69             result_df = result_df.append(_search_scopus(self.apikey, query, type_, view=view, index=index),
     70                                          ignore_index=True)
     71             if result_df.shape[0] >= count:

/usr/local/lib/python3.8/site-packages/pyscopus/utils.py in _search_scopus(key, query, type_, view, index)
    386     js = r.json()
    387     #print(r.url)
--> 388     total_count = int(js['search-results']['opensearch:totalResults'])
    389     entries = js['search-results']['entry']
    390 

KeyError: 'search-results'

Is there a way to find out the maximum number of results and use this as count?

[skilkis]

I get this exact KeyError due to the request failing with the following:

'statusCode':'AUTHORIZATION_ERROR'
'statusText':'The requestor is not authorized to access the requested view or fields of the resource'

I am not sure how to fix it and don't have the time to look into it. Would be great if you have time to look into this @zhiyzuo

[skilkis]

@pournaki I am also getting a similar error using pybiometrics so I think our access to Scopus is to blame.

[pournaki]

hi @skilkis, thanks for the clarification. I will close this issue then.

do you happen to know if there is a straightforward way to check the remaining number of requests for a given scopus API key?

[skilkis]

@pournaki I don't right now. I'll be working with the Scopus API in the next few weeks, I'll update you if I find out 😄