search

EskoSalaka / mtgtools

collection of tools for easy handling of Magic: The Gathering data on your computer
Other
65 stars 13 forks source link

MtgDB.full_update_from_scryfall fails #3

Closed gus3000 closed 5 years ago

gus3000 commented 5 years ago

The methods full_update_from_scryfall() and full_update_from_mtgio() fail (traceback at the end, a kind of NoneType error) I'm currently investigating. Things to note :

I'll try and correct this and do a proper pull request.

Traceback :

Traceback (most recent call last):
  File "PROJECT_PATH/deckGenerator/deckPreview.py", line 42, in <module>
    main()
  File "PROJECT_PATH/deckGenerator/deckPreview.py", line 30, in main
    db.full_update_from_scryfall()
  File "PROJECT_PATH/venv/lib/python3.6/site-packages/mtgtools/MtgDB.py", line 536, in full_update_from_scryfall
    loop.run_until_complete(process_scryfall_cards())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 473, in run_until_complete
    return future.result()
  File "PROJECT_PATH/venv/lib/python3.6/site-packages/mtgtools/MtgDB.py", line 519, in process_scryfall_cards
    cards = await future
  File "/usr/lib/python3.6/asyncio/tasks.py", line 458, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "PROJECT_PATH/venv/lib/python3.6/site-packages/mtgtools/MtgDB.py", line 173, in _process_card_page_response
    return [PCard(card_response) for card_response in self._get_response_json(card_page_uri)[list_identifier]]
TypeError: 'NoneType' object is not subscriptable
EskoSalaka commented 5 years ago

Good call, im on it too. I get some other type of errors too.

This could be some new type of card, something changed in the API or some info the API provides doesnt match whats actually in the Scryfall database

EskoSalaka commented 5 years ago

Okay, so what I am getting on my computer with Python 3.7 is an error coming from accessing elements of an empty list of cards

Scryfall seems to have some errors with documenting how many cards certain sets have. For example, their API says Odyssey has a card_count of 351 but actually has 350 cards. You can see this here: https://api.scryfall.com/sets. Odyssey is the number 450.

Did this change recently?

When starting the update, this card_count attribute of the sets is used to determine what requests to send to fetch the card objects of each set. One request resulting in an empty list of returned cards is https://api.scryfall.com/cards/search?include_extras=true&order=set&**page=3**&q=e%3Aody&unique=prints. This, the third page, should not be requested at all. Scryfall serves 175 cards of a set on one page. With 350 cards we would need 2 requests of 175 cards per page, but what I think is happening is that the extra card results in a false extra request of an empty third page. This request is handled and returns an empty list which eventually causes the program to interrupt on my computer.

I wonder why I havent gotten this error before? Is this maybe related to yours?

I can ignore the error I get by changing the line 520 in MtgDB.py: From: pset = current_sets.where_exactly(code=cards[0].set)[0]

To: if cards: pset = current_sets.where_exactly(code=cards[0].set)[0]

EskoSalaka commented 5 years ago

As for mtg.io, I am getting a similar type of error resulting from trying to add a card to a set that doesnt seem to exists in the database:

The set code is OARC and the name is "Archenemy Schemes". The cards seem to be there but the set not.

EskoSalaka commented 5 years ago 
* I did not see this before, but the requests encounter a 404 ( `Something went wrong with requesting url https://api.scryfall.com/cards/search?include_extras=true&order=set&page=1&q=e%3Apgp1&unique=prints : HTTP Error 404: Not Found` ) during the sending.

Here, the part q=e%3Apgp1 results in the 404. Its trying to find a set code pgp1, which doesnt exists. It should probably be pgpX instead. Maybe it used to be pgp1.

EskoSalaka commented 5 years ago

Ok, my latest update is that your error is probably caused by Scryfall changing names/codes of a few certain sets. The error i get is because of false numbers of cards in the set as given by the API. I also get an error by mtgio because they are missing a set but the cards are there.

Because of this, I will have to implement more robust way of updating the database to accomodate these kind of errors! A good learning experience!

EskoSalaka commented 5 years ago

Whew, ok I think i fixed most of the errors. I deprecated a few methods and changed a few names also, let's hope i didn't break anything!