KeyError 'best' in numplayers_to_result

[tapanther]

Hello,

I'm getting a KeyError on the 'best' key for result in line 120 of bgg_client.py. Stack trace is at the bottom.

SHA of your commit: aad8aee1e5ed357daae9ab319570462e68c827d2

I didn't inspect the rest of the code, but I suspect some games with 0 votes for best at # just don't generate that entry. I resolved it locally for myself by adding a try-except KeyError around the is_recommended, but that's probably not the best formal fix.

Additionally it seems that the existence of 'best' and 'recommended' can't be guaranteed (tried the except block without 'best' but still got a key error on 'recommended')

Traceback (most recent call last):
  File "download_and_index.py", line 75, in <module>
    main(args)
  File "download_and_index.py", line 17, in main
    extra_params=SETTINGS["boardgamegeek"]["extra_params"],
  File "/home/jsierra/mybgg/mybgg/downloader.py", line 37, in collection
    [game_in_collection["id"] for game_in_collection in collection_data]
  File "/home/jsierra/mybgg/mybgg/bgg_client.py", line 43, in game_list
    games += self._games_list_to_games(data)
  File "/home/jsierra/mybgg/mybgg/bgg_client.py", line 211, in _games_list_to_games
    games = xml.parse_from_string(game_processor, data)
  File "/tools/python/versions/3.7.0/lib/python3.7/site-packages/declxml.py", line 203, in parse_from_string
    return root_processor.parse_at_root(root, state)
  File "/tools/python/versions/3.7.0/lib/python3.7/site-packages/declxml.py", line 681, in parse_at_root
    parsed_dict = self.parse_at_element(dict_element, state)
  File "/tools/python/versions/3.7.0/lib/python3.7/site-packages/declxml.py", line 666, in parse_at_element
    parsed_dict[child.alias] = child.parse_from_parent(element, state)
  File "/tools/python/versions/3.7.0/lib/python3.7/site-packages/declxml.py", line 584, in parse_from_parent
    return self._parse(item_iter, state)
  File "/tools/python/versions/3.7.0/lib/python3.7/site-packages/declxml.py", line 625, in _parse
    parsed_array.append(self._item_processor.parse_at_element(item, state))
  File "/tools/python/versions/3.7.0/lib/python3.7/site-packages/declxml.py", line 666, in parse_at_element
    parsed_dict[child.alias] = child.parse_from_parent(element, state)
  File "/tools/python/versions/3.7.0/lib/python3.7/site-packages/declxml.py", line 746, in parse_from_parent
    xml_value = self._processor.parse_from_parent(parent, state)
  File "/tools/python/versions/3.7.0/lib/python3.7/site-packages/declxml.py", line 584, in parse_from_parent
    return self._parse(item_iter, state)
  File "/tools/python/versions/3.7.0/lib/python3.7/site-packages/declxml.py", line 625, in _parse
    parsed_array.append(self._item_processor.parse_at_element(item, state))
  File "/tools/python/versions/3.7.0/lib/python3.7/site-packages/declxml.py", line 666, in parse_at_element
    parsed_dict[child.alias] = child.parse_from_parent(element, state)
  File "/tools/python/versions/3.7.0/lib/python3.7/site-packages/declxml.py", line 747, in parse_from_parent
    return _hooks_apply_after_parse(self._hooks, state, xml_value)
  File "/tools/python/versions/3.7.0/lib/python3.7/site-packages/declxml.py", line 1033, in _hooks_apply_after_parse
    return hooks.after_parse(ProcessorStateView(state), value)
  File "/home/jsierra/mybgg/mybgg/bgg_client.py", line 120, in numplayers_to_result
    is_recommended = result['best'] + result['recommended'] > result['not_recommended']
KeyError: 'best'

[EmilStenstrom]

Thank you for reporting this. I'll see if I can get some time this weekend to have a look and see it I can reproduce. Do you know of a game that has no votes that I can test with?

[tapanther]

Sure thing.

And yes, check out Arranged!. It has zero votes for player counts, and it is super unlikely to get any.

[EmilStenstrom]

Hi again,

I've tried reproducing this with no luck. I started adding "Arranged!" to my "own" list.

I then ran python download_and_index.py --apikey MY_API_KEY. I didn't get any error, and starting the web server on localhost I can see and filter on Arranged! and see the game.

Just to make sure, I verified the XML that is returned. For Arranged!, the XML looks like this:

...
<poll name="suggested_numplayers" title="User Suggested Number of Players" totalvotes="0">
    <results numplayers="1">
        <result numvotes="0" value="Best"/>
        <result numvotes="0" value="Recommended"/>
        <result numvotes="0" value="Not Recommended"/>
    </results>
    <results numplayers="2">
        <result numvotes="0" value="Best"/>
        <result numvotes="0" value="Recommended"/>
        <result numvotes="0" value="Not Recommended"/>
    </results>
    <results numplayers="3">
        <result numvotes="0" value="Best"/>
        <result numvotes="0" value="Recommended"/>
        <result numvotes="0" value="Not Recommended"/>
    </results>
    <results numplayers="4">
        <result numvotes="0" value="Best"/>
        <result numvotes="0" value="Recommended"/>
        <result numvotes="0" value="Not Recommended"/>
    </results>
    <results numplayers="4+">
        <result numvotes="0" value="Best"/>
        <result numvotes="0" value="Recommended"/>
        <result numvotes="0" value="Not Recommended"/>
    </results>
</poll>
...

Inside numplayers_to_result, this gets transformed to {'best': 0, 'recommended': 0, 'not_recommended': 0}, where the keys originate from the statement: result["value"].lower().replace(" ", "_")

The only reason I can see that the "best"-key would be missing, is if the XML is somehow broken.

So, to debug this, would you mind updating the the latest version of mybgg (I removed a limit on the --debug flag to only print 25 lines). And run download_and_index.py with --debug set.

Please double-check that the XML returned has the same structure that I get above.

[tapanther]

Hi, sorry it took me a bit to get back to you.

It looks like the issue wasn't Arranged!, but either NMBR9 ("Want to Buy") or Nuclear Escalation ("Owned") - the two games after Nile Deluxor in my list. You can check tapanther on bgg. Attached is the full output of the debug run using your latest commit.

keyerror_best.log

[EmilStenstrom]

Hi again,

I finally managed to reproduce this, using your configuration. It was one of the expansions for Fleet Commander that had no votes, and made things crash. I've added better debug logging to troubleshoot this, and fixed the bug. I can now import all the games using your configuration.

[tapanther]

Cool, thank you for fixing this.