MagiCircles / SchoolIdolAPI

The ultimate resource for LLSIF players
http://schoolido.lu
Apache License 2.0
150 stars 28 forks source link

Can't fill database with cards/events - getting 404 error #161

Open dburr opened 9 years ago

dburr commented 9 years ago

Hi, I am experimenting with the schoolido.lu code in a VM so that I can understand how it works and maybe (eventually) help contribute. Anyway I am following the steps in the Contributor's Guide. Everything is going well until the "Fill database with cards" step at which point the script crashes. It looks like it's having trouble trying to add the event 獲物は…君だよ!

(env)vagrant@schoolidolu:~/SchoolIdolAPI$ python manage.py importcards
### Import card ids & stats from decaf wiki
Importing card # 1 ... Done
Importing card # 2 ... Done
Importing card # 3 ... Done
[...]
Importing card # 388 ... Done
Importing card # 389 ... Done
Importing card # 390 ... Done
### Import events from decaf wiki
Import event  Score Match Round 21 ... Done
  Import event image... Done
Import event  Loving you! ... Done
  Import event image... Done
Import event  Medley Festival Round 7 ... Done
  Import event image... Done
[...]
Import event  Score Match Round 2 ... Done
  Import event image... Done
Import event  獲物は…君だよ!... Done
  Import event image...
Traceback (most recent call last):
  File "manage.py", line 11, in <module>
    execute_from_command_line(sys.argv)
  File "/home/vagrant/SchoolIdolAPI/env/local/lib/python2.7/site-packages/django
/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/home/vagrant/SchoolIdolAPI/env/local/lib/python2.7/site-packages/django
/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.arg)
  File "/home/vagrant/SchoolIdolAPI/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/vagrant/SchoolIdolAPI/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
  File "/home/vagrant/SchoolIdolAPI/api/management/commands/importcards.py", line 30, in handle
    import_jp_events()
  File "/home/vagrant/SchoolIdolAPI/api/management/commands/import_jp_events.py", line 51, in import_jp_events
    f_event = urllib2.urlopen(url)
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found

manage.py supposedly has a -v VERBOSITY argument to increase verbosity but it does not seem to change the output at all (I tried 1, 2 and 3)

So I added my own debug print statement in import_jp_events() in api/management/commands/import_jp_events.py:

Import event  Score Match Round 2 ... Done
Import event  獲物は…君だよ! ... Done
  Import event image... URL IS http://decaf.kouhi.me/lovelive/index.php?title=%E
7%8D%B2%E7%89%A9%E3%81%AF%E2%80%A6%E5%90%9B%E3%81%A0%E3%82%88%21
Traceback (most recent call last):
  File "manage.py", line 11, in <module>
[...same error output as above...]

I tried going to that URL and it does in fact return a 404 page.

BTW this occurs whether or not I use noimages

Also why does it stop at card #390 when importing cards? Unless I'm mistaken, I'm pretty sure there are more cards in the game than that.

db0company commented 9 years ago

I think I introduced this bug recently when I imported the songs. Some songs required to replace the Japanese character "!" with a regular ASCII "!", so I added that in the cleaner function. Since this event doesn't need this replacement, I moved that in the import_songs script (see @4564254d86b33a6f89876136da7b1cdc6508587e).

Can you tell me if the script works now?

The reason you see the script stop at this number is probably because the cards are not imported in order of ID, but ordered by rarity/promo (not sure). Tell me if you see all of them in your local db or not though.

Thanks for reporting this.