search

PennyDreadfulMTG / Penny-Dreadful-Tools

A suite of tools for the Penny Dreadful MTGO community
https://pennydreadfulmagic.com
MIT License
40 stars 28 forks source link

Gatherling scraper crashes complaining that it doesn't allow draws #5986

Closed bakert closed 5 years ago

bakert commented 5 years ago 
discord@katelyngigante:~> cd ~/decksite && python3 run.py scraper gatherling
Fetching https://api.scryfall.com/bulk-data (cache ok)
Fetching https://gatherling.com/eventreport.php?format=Penny+Dreadful&series=&season=&mode=Filter+Events (cache ok)
Fetching https://gatherling.com/eventreport.php?event=Penny Dreadful Saturdays 11.01 (cache ok)
Before BEGIN ([])
After BEGIN (['tournament'])
Fetching https://gatherling.com/deck.php?mode=view&id=65388 (cache ok)
[2019-01-27 10:37:37,210] WARNING in logger: Skipping 65388 because it played no matches.
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'insert_matches_without_dupes'])
Before COMMIT (['tournament', 'insert_matches_without_dupes'])
After COMMIT (['tournament'])
Before COMMIT (['tournament'])
After COMMIT ([])
Fetching https://gatherling.com/eventreport.php?event=Penny Dreadful Thursdays 10.16 (cache ok)
Before BEGIN ([])
After BEGIN (['tournament'])
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'insert_matches_without_dupes'])
Before COMMIT (['tournament', 'insert_matches_without_dupes'])
After COMMIT (['tournament'])
Before COMMIT (['tournament'])
After COMMIT ([])
Fetching https://gatherling.com/eventreport.php?event=Penny Dreadful Mondays 10.16 (cache ok)
Before BEGIN ([])
After BEGIN (['tournament'])
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'insert_matches_without_dupes'])
Before COMMIT (['tournament', 'insert_matches_without_dupes'])
After COMMIT (['tournament'])
Before COMMIT (['tournament'])
After COMMIT ([])
Fetching https://gatherling.com/eventreport.php?event=Penny Dreadful Sundays 10.16 (cache ok)
Before BEGIN ([])
After BEGIN (['tournament'])
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'insert_matches_without_dupes'])
Before COMMIT (['tournament', 'insert_matches_without_dupes'])
After COMMIT (['tournament'])
Before COMMIT (['tournament'])
After COMMIT ([])
Fetching https://gatherling.com/eventreport.php?event=APAC Penny Dreadful Sundays 10.16 (cache ok)
Before BEGIN ([])
After BEGIN (['tournament'])
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'insert_matches_without_dupes'])
Before COMMIT (['tournament', 'insert_matches_without_dupes'])
After COMMIT (['tournament'])
Before COMMIT (['tournament'])
After COMMIT ([])
Fetching https://gatherling.com/eventreport.php?event=Penny Dreadful Saturdays 10.16 (cache ok)
Before BEGIN ([])
After BEGIN (['tournament'])
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'insert_matches_without_dupes'])
Before COMMIT (['tournament', 'insert_matches_without_dupes'])
After COMMIT (['tournament'])
Before COMMIT (['tournament'])
After COMMIT ([])
Fetching https://gatherling.com/eventreport.php?event=Penny Dreadful Thursdays 10.15 (cache ok)
Before BEGIN ([])
After BEGIN (['tournament'])
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'insert_matches_without_dupes'])
Before COMMIT (['tournament', 'insert_matches_without_dupes'])
After COMMIT (['tournament'])
Before COMMIT (['tournament'])
After COMMIT ([])
Fetching https://gatherling.com/eventreport.php?event=Penny Dreadful Mondays 10.15 (cache ok)
Before BEGIN ([])
After BEGIN (['tournament'])
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'insert_matches_without_dupes'])
Before COMMIT (['tournament', 'insert_matches_without_dupes'])
After COMMIT (['tournament'])
Before COMMIT (['tournament'])
After COMMIT ([])
Fetching https://gatherling.com/eventreport.php?event=Penny Dreadful Sundays 10.15 (cache ok)
Before BEGIN ([])
After BEGIN (['tournament'])
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'insert_matches_without_dupes'])
Before COMMIT (['tournament', 'insert_matches_without_dupes'])
After COMMIT (['tournament'])
Before COMMIT (['tournament'])
After COMMIT ([])
Fetching https://gatherling.com/eventreport.php?event=APAC Penny Dreadful Sundays 10.15 (cache ok)
Before BEGIN ([])
After BEGIN (['tournament'])
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'insert_matches_without_dupes'])
Before COMMIT (['tournament', 'insert_matches_without_dupes'])
After COMMIT (['tournament'])
Before COMMIT (['tournament'])
After COMMIT ([])
Fetching https://gatherling.com/eventreport.php?event=Penny Dreadful Saturdays 10.15 (cache ok)
Before BEGIN ([])
After BEGIN (['tournament'])
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'insert_matches_without_dupes'])
Before COMMIT (['tournament', 'insert_matches_without_dupes'])
After COMMIT (['tournament'])
Before COMMIT (['tournament'])
After COMMIT ([])
Fetching https://gatherling.com/eventreport.php?event=Penny Dreadful Thursdays 10.14 (cache ok)
Before BEGIN ([])
After BEGIN (['tournament'])
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'insert_competition'])
Before COMMIT (['tournament', 'insert_competition'])
After COMMIT (['tournament'])
POSTing to https://gatherling.com/deckdl.php with {'id': '65037'} / None
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'add_deck'])
Before BEGIN (['tournament', 'add_deck'])
After BEGIN (['tournament', 'add_deck', 'add_cards'])
Before COMMIT (['tournament', 'add_deck', 'add_cards'])
After COMMIT (['tournament', 'add_deck'])
Updating Legalities…
Before COMMIT (['tournament', 'add_deck'])
After COMMIT (['tournament'])
Fetching https://gatherling.com/deck.php?mode=view&id=65037 (cache ok)
POSTing to https://gatherling.com/deckdl.php with {'id': '65039'} / None
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'add_deck'])
Before BEGIN (['tournament', 'add_deck'])
After BEGIN (['tournament', 'add_deck', 'add_cards'])
Before COMMIT (['tournament', 'add_deck', 'add_cards'])
After COMMIT (['tournament', 'add_deck'])
Before COMMIT (['tournament', 'add_deck'])
After COMMIT (['tournament'])
Fetching https://gatherling.com/deck.php?mode=view&id=65039 (cache ok)
POSTing to https://gatherling.com/deckdl.php with {'id': '65035'} / None
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'add_deck'])
Before BEGIN (['tournament', 'add_deck'])
After BEGIN (['tournament', 'add_deck', 'add_cards'])
Before COMMIT (['tournament', 'add_deck', 'add_cards'])
After COMMIT (['tournament', 'add_deck'])
Before COMMIT (['tournament', 'add_deck'])
After COMMIT (['tournament'])
Fetching https://gatherling.com/deck.php?mode=view&id=65035 (cache ok)
POSTing to https://gatherling.com/deckdl.php with {'id': '65043'} / None
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'add_deck'])
Before BEGIN (['tournament', 'add_deck'])
After BEGIN (['tournament', 'add_deck', 'add_cards'])
Before COMMIT (['tournament', 'add_deck', 'add_cards'])
After COMMIT (['tournament', 'add_deck'])
Before COMMIT (['tournament', 'add_deck'])
After COMMIT (['tournament'])
Fetching https://gatherling.com/deck.php?mode=view&id=65043 (cache ok)
POSTing to https://gatherling.com/deckdl.php with {'id': '65041'} / None
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'add_deck'])
Before BEGIN (['tournament', 'add_deck'])
After BEGIN (['tournament', 'add_deck', 'add_cards'])
Before COMMIT (['tournament', 'add_deck', 'add_cards'])
After COMMIT (['tournament', 'add_deck'])
Before COMMIT (['tournament', 'add_deck'])
After COMMIT (['tournament'])
Fetching https://gatherling.com/deck.php?mode=view&id=65041 (cache ok)
POSTing to https://gatherling.com/deckdl.php with {'id': '65040'} / None
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'add_deck'])
Before BEGIN (['tournament', 'add_deck'])
After BEGIN (['tournament', 'add_deck', 'add_cards'])
Before COMMIT (['tournament', 'add_deck', 'add_cards'])
After COMMIT (['tournament', 'add_deck'])
Before COMMIT (['tournament', 'add_deck'])
After COMMIT (['tournament'])
Fetching https://gatherling.com/deck.php?mode=view&id=65040 (cache ok)
POSTing to https://gatherling.com/deckdl.php with {'id': '65036'} / None
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'add_deck'])
Before BEGIN (['tournament', 'add_deck'])
After BEGIN (['tournament', 'add_deck', 'add_cards'])
Before COMMIT (['tournament', 'add_deck', 'add_cards'])
After COMMIT (['tournament', 'add_deck'])
Before COMMIT (['tournament', 'add_deck'])
After COMMIT (['tournament'])
Fetching https://gatherling.com/deck.php?mode=view&id=65036 (cache ok)
POSTing to https://gatherling.com/deckdl.php with {'id': '65038'} / None
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'add_deck'])
Before BEGIN (['tournament', 'add_deck'])
After BEGIN (['tournament', 'add_deck', 'add_cards'])
Before COMMIT (['tournament', 'add_deck', 'add_cards'])
After COMMIT (['tournament', 'add_deck'])
Before COMMIT (['tournament', 'add_deck'])
After COMMIT (['tournament'])
Fetching https://gatherling.com/deck.php?mode=view&id=65038 (cache ok)
POSTing to https://gatherling.com/deckdl.php with {'id': '65018'} / None
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'add_deck'])
Before BEGIN (['tournament', 'add_deck'])
After BEGIN (['tournament', 'add_deck', 'add_cards'])
Before COMMIT (['tournament', 'add_deck', 'add_cards'])
After COMMIT (['tournament', 'add_deck'])
Before COMMIT (['tournament', 'add_deck'])
After COMMIT (['tournament'])
Fetching https://gatherling.com/deck.php?mode=view&id=65018 (cache ok)
POSTing to https://gatherling.com/deckdl.php with {'id': '65042'} / None
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'add_deck'])
Before BEGIN (['tournament', 'add_deck'])
After BEGIN (['tournament', 'add_deck', 'add_cards'])
Before COMMIT (['tournament', 'add_deck', 'add_cards'])
After COMMIT (['tournament', 'add_deck'])
Before COMMIT (['tournament', 'add_deck'])
After COMMIT (['tournament'])
Fetching https://gatherling.com/deck.php?mode=view&id=65042 (cache ok)
POSTing to https://gatherling.com/deckdl.php with {'id': '65004'} / None
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'add_deck'])
Before BEGIN (['tournament', 'add_deck'])
After BEGIN (['tournament', 'add_deck', 'add_cards'])
Before COMMIT (['tournament', 'add_deck', 'add_cards'])
After COMMIT (['tournament', 'add_deck'])
Before COMMIT (['tournament', 'add_deck'])
After COMMIT (['tournament'])
Fetching https://gatherling.com/deck.php?mode=view&id=65004 (cache ok)
Before BEGIN (['tournament'])
After BEGIN (['tournament', 'insert_matches_without_dupes'])
Before BEGIN (['tournament', 'insert_matches_without_dupes'])
After BEGIN (['tournament', 'insert_matches_without_dupes', 'insert_match'])
Before COMMIT (['tournament', 'insert_matches_without_dupes', 'insert_match'])
After COMMIT (['tournament', 'insert_matches_without_dupes'])
Before BEGIN (['tournament', 'insert_matches_without_dupes'])
After BEGIN (['tournament', 'insert_matches_without_dupes', 'insert_match'])
Before COMMIT (['tournament', 'insert_matches_without_dupes', 'insert_match'])
After COMMIT (['tournament', 'insert_matches_without_dupes'])
Before BEGIN (['tournament', 'insert_matches_without_dupes'])
After BEGIN (['tournament', 'insert_matches_without_dupes', 'insert_match'])
Before COMMIT (['tournament', 'insert_matches_without_dupes', 'insert_match'])
After COMMIT (['tournament', 'insert_matches_without_dupes'])
Before BEGIN (['tournament', 'insert_matches_without_dupes'])
After BEGIN (['tournament', 'insert_matches_without_dupes', 'insert_match'])
Before COMMIT (['tournament', 'insert_matches_without_dupes', 'insert_match'])
After COMMIT (['tournament', 'insert_matches_without_dupes'])
Before BEGIN (['tournament', 'insert_matches_without_dupes'])
After BEGIN (['tournament', 'insert_matches_without_dupes', 'insert_match'])
Before COMMIT (['tournament', 'insert_matches_without_dupes', 'insert_match'])
After COMMIT (['tournament', 'insert_matches_without_dupes'])
Before BEGIN (['tournament', 'insert_matches_without_dupes'])
After BEGIN (['tournament', 'insert_matches_without_dupes', 'insert_match'])
Before COMMIT (['tournament', 'insert_matches_without_dupes', 'insert_match'])
After COMMIT (['tournament', 'insert_matches_without_dupes'])
Before BEGIN (['tournament', 'insert_matches_without_dupes'])
After BEGIN (['tournament', 'insert_matches_without_dupes', 'insert_match'])
Before COMMIT (['tournament', 'insert_matches_without_dupes', 'insert_match'])
After COMMIT (['tournament', 'insert_matches_without_dupes'])
Before BEGIN (['tournament', 'insert_matches_without_dupes'])
After BEGIN (['tournament', 'insert_matches_without_dupes', 'insert_match'])
Before COMMIT (['tournament', 'insert_matches_without_dupes', 'insert_match'])
After COMMIT (['tournament', 'insert_matches_without_dupes'])
Before BEGIN (['tournament', 'insert_matches_without_dupes'])
After BEGIN (['tournament', 'insert_matches_without_dupes', 'insert_match'])
Before COMMIT (['tournament', 'insert_matches_without_dupes', 'insert_match'])
After COMMIT (['tournament', 'insert_matches_without_dupes'])
Before BEGIN (['tournament', 'insert_matches_without_dupes'])
After BEGIN (['tournament', 'insert_matches_without_dupes', 'insert_match'])
Before COMMIT (['tournament', 'insert_matches_without_dupes', 'insert_match'])
After COMMIT (['tournament', 'insert_matches_without_dupes'])
Before BEGIN (['tournament', 'insert_matches_without_dupes'])
After BEGIN (['tournament', 'insert_matches_without_dupes', 'insert_match'])
Before COMMIT (['tournament', 'insert_matches_without_dupes', 'insert_match'])
After COMMIT (['tournament', 'insert_matches_without_dupes'])
Traceback (most recent call last):
  File "run.py", line 120, in <module>
    run()
  File "run.py", line 34, in run
    task(sys.argv)
  File "run.py", line 80, in task
    s.scrape() # type: ignore
  File "/home/discord/decksite/decksite/scrapers/gatherling.py", line 28, in scrape
    i = tournament(url, name)
  File "/home/discord/decksite/decksite/scrapers/gatherling.py", line 56, in tournament
    n = add_decks(dt, competition_id, final, s)
  File "/home/discord/decksite/decksite/scrapers/gatherling.py", line 100, in add_decks
    insert_matches_without_dupes(dt, matches)
  File "/home/discord/decksite/decksite/scrapers/gatherling.py", line 245, in insert_matches_without_dupes
    match.insert_match(dt, m['left_id'], m['left_games'], m['right_id'], m['right_games'], m['round'], m['elimination'])
  File "/home/discord/decksite/decksite/data/match.py", line 22, in insert_match
    raise InvalidDataException('`insert_match` does not support draws.')
shared.pd_exception.InvalidDataException: `insert_match` does not support draws.
bakert commented 5 years ago

https://gatherling.com/deck.php?mode=view&id=65038 https://gatherling.com/deck.php?mode=view&id=65018

Solution here is probably to delete these decks from Gatherling.

Waiting for an ok from @silasary

bakert commented 5 years ago

The draw is for a double no-show. I deleted the match in Gatherling. Not sure if that's perfect but it's expedient.