joalla / discogs_client

Continuation of the "Official Python Client for the Discogs API"
https://python3-discogs-client.readthedocs.io
Other
309 stars 50 forks source link

You are making requests too quickly. #62

Closed feacluster closed 3 years ago

feacluster commented 3 years ago

I seem to randomnly get this error when searching a seller's inventory of more than a thousand or so albums. Here is the code snippet you can try ( just edit the line with the user_token )

import discogs_client
import sys, time, re, time, json

debug = 0

buyer = 'faraz12inch45rpm'
seller = 'rockdonkey'
format = 'LP'
condition = 'Mint|NM|VG+'
budget = 10000

condition = re.sub ( '\+', '\\+', condition )

#######################

def check_conditions():

    global album, budget

    if ( re.search ( format , album.data['release']['format'],  re.IGNORECASE ) ):
        if ( re.search ( condition , album.data['condition'] ) ):
            if ( int ( album.data['original_price']['value'] ) > budget ):
                return 0
            else:
                return 1
    return 0

########
# MAIN #
########

d = discogs_client.Client('my_app/0.1', user_token="xxxxxx")

print('<table class=default_table CELLSPACING=0 CELLPADDING=10>')
print("<tr><td> Buyer: </td><td>", buyer, "</td></tr>")
print("<tr><td> Seller: </td><td>", seller, "</td></tr>")
print("<tr><td> Format: </td><td>", format, "</td></tr>")
print("<tr><td> Condition: </td><td>", condition, "</td></tr>")
print("<tr><td> Budget: </td><td>", budget, "</td></tr></table>")

me = d.user(buyer)

collection = me.collection_folders
all = collection[0].releases

if ( all.count < 2 ):
    print("<p>No artists found in your collection. Try adding at least a few dozen albums from artists you like. The script will search those artists from the sellers inventory. ")
    sys.exit()

print("<p>Total artists in my collection: ", all.count)

my_artists = []
for album in all:
     artist =  album.data['basic_information']['artists'][0]['name']
     if ( re.search ( 'Various', artist ) ):
         continue
     if artist not in my_artists:
         my_artists.append(artist)

print("<p>Unique artists in my collection: ", len( my_artists ))

with open('my_artists.json', 'w') as f:
    f.write(json.dumps( my_artists ))

if debug:
    for artist in my_artists:
        print(artist)

########################

options = [
    ('listed', 'asc'),
    ('listed', 'desc'),
    ('price', 'asc'),
    ('price', 'desc'),
    ('item', 'asc'),
    ('item', 'desc'),
    ('artist', 'asc'),
    ('artist', 'desc'),
    ('label', 'asc'),
    ('label', 'desc'),
    ('catno', 'asc'),
    ('catno', 'desc'),
    ('audio', 'asc'),
    ('audio', 'desc'),
]

seller = d.user( seller )
inventory = seller.inventory
inventory._per_page=100
pages = inventory.pages

# Unique albums
found = []

count = 0
total = len(inventory)

print("<p>", total, " items found in seller's inventory <p>")

if ( total > 1000000 ):
    print("<h3>Seller's inventory is too large: </h3>", total)
    sys.exit()
if ( total > 10000 ):
    print("<h3>Seller's inventory is too large. Job may crash before searching the first 10,000 </h3>", total)

print('<table class=default_table CELLSPACING=0 CELLPADDING=10>')
print('<tr> <th>Artist</th><th>Title</th> <th>Price</th> <th>Format</th> <th>Condition</th> <th> Label </th> <th> Year </th>  </tr>')

for option in options:

    # All albums found, exit
    if len(found) == total:
        exit()

    # Change the sorting
    inventory.sort(*option)

    # Get First 100 pages (10,000 albums)
    for page_num in range(1, min(100, pages) + 1):
        for album in inventory.page(page_num):
            # Find unique albums from page
            if album.id not in found:
                found.append(album.id)
                artist = album.data['release']['artist']
                if artist in my_artists:
                    if ( check_conditions() ):
                        print('<tr><td>', artist.encode('utf-8').strip().decode('utf8') , '</td>')
                        print('<td><a href=', album.data['uri'] , ' target=new>', album.data['release']['title'].encode('utf-8').strip().decode('utf8') , '</a></td>')
                        print('<td>', album.data['original_price']['value'], '</td>')
                        print('<td>', album.data['release']['format'], '</td>')
                        print('<td>', album.data['condition'], '</td>')

                        id = album.data['release']['id']
                        print('<td>', d.release(id).labels[0].data['name'].encode('utf-8').strip().decode('utf8'), '</td>')
                        print('<td>', d.release(id).year, '</td></tr>')

#        print( '<tr><td colspan=5>' + f'{page_num}: {len(found)} / {total} found.' + '</td></tr>')
        if ( len(found) % 1000 == 0 ):
            print( '<tr><td colspan=5>' + f'{len(found)} / {total} found.' + '</td></tr>')

print('</tr>')
AnssiAhola commented 3 years ago

Hey, @feacluster ! That code looks familiar 😄

What client version are you running? What about python version?

I tried with latest (2.3.12) and python 3.9. Also changed seller to "dingoacid" with around 5500 listings as the default one had no listings visible for me. And had no such errors.

Theres was a exponential backoff feature introduced in version 2.3.10 which should prevent these kinds of problems, try upgrading to that or latest if you havent allready.

feacluster commented 3 years ago

Thanks for the quick reply! Sorry, I am not sure why the seller rockdonkey is showing no listings now.. Anyways I checked and seems I am using version 2.2.2:

Python 3.6.8 (default, Nov 16 2020, 16:55:22)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import discogs_client
>>> print ( discogs_client.__version__)
2.2.2

I next tried:

 pip3 install --upgrade python3-discogs-client --user

But the version still showed 2.2.2:

Python 3.6.8 (default, Nov 16 2020, 16:55:22)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import discogs_client
>>> print ( discogs_client.__version__)
2.2.2
AnssiAhola commented 3 years ago

Seems like the discogs_client.__version__ hasnt been updated with the releases. https://github.com/joalla/discogs_client/blob/master/discogs_client/__init__.py Try command pip show python3-discogs-client

feacluster commented 3 years ago

Cool, yes it seems to show latest version using either pip show or pip3 show:

[feacluster@instance-5 feacluster]$ pip show python3-discogs-client
---
Metadata-Version: 1.1
Name: python3-discogs-client
Version: 2.3.5
Summary: Python API client for Discogs
Home-page: https://github.com/joalla/discogs_client
Author: joalla
Author-email: jt@peek-a-boo.at
License: UNKNOWN
Location: /home/feacluster/.local/lib/python2.7/site-packages
Requires: requests, oauthlib
Classifiers:
  Development Status :: 5 - Production/Stable
  Environment :: Console
  License :: OSI Approved :: BSD License
  Natural Language :: English
  Operating System :: OS Independent
  Programming Language :: Python
  Topic :: Communications
  Topic :: Utilities
You are using pip version 8.1.2, however version 21.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[feacluster@instance-5 feacluster]$ pip3 show python3-discogs-client
Name: python3-discogs-client
Version: 2.3.12
Summary: Python API client for Discogs
Home-page: https://github.com/joalla/discogs_client
Author: joalla
Author-email: jt@peek-a-boo.at
License: UNKNOWN
Location: /home/feacluster/.local/lib/python3.6/site-packages
Requires: requests, python-dateutil, oauthlib
AnssiAhola commented 3 years ago

And it still produces that error? pip show seems to show version 2.3.5 while pip3 show is the latest, which command are you using when running the script (pip vs. pip3)?

feacluster commented 3 years ago

Just tested again and it seems to be working without any errors now. I am running it with python3 . I guess the python2.7 client is some older installation that I don't use.