Adding games is broken with Django 4.2

[rjmagley]

This is something we foolishly didn't bother to test before - but adding games seems to be very broken.

When trying to add a SA thread (the Mafia discussion thread), this happens:

[2023-09-03 19:40:36 -0400] [1447689] [DEBUG] POST /add_game
Internal Server Error: /add_game
Traceback (most recent call last):
  File "/srv/vf/votefinder/main/views.py", line 77, in add_game
    game = Game.objects.get(thread_id=threadid, home_forum=home_forum)
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/db/models/query.py", line 637, in get
    raise self.model.DoesNotExist(
votefinder.main.models.Game.DoesNotExist: Game matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/db/models/query.py", line 916, in get_or_create
    return self.get(**kwargs), False
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/db/models/query.py", line 637, in get
    raise self.model.DoesNotExist(
votefinder.main.models.Game.DoesNotExist: Game matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/contrib/auth/decorators.py", line 23, in _wrapper_view
    return view_func(request, *args, **kwargs)
  File "/srv/vf/votefinder/main/views.py", line 89, in add_game
    game = page_parser.add_game(threadid, state)
  File "/srv/vf/votefinder/main/SAPageParser.py", line 27, in add_game
    return self.download_and_update(f'https://forums.somethingawful.com/showthread.php?threadid={threadid}', threadid)
  File "/srv/vf/votefinder/main/SAPageParser.py", line 34, in download_and_update
    game = self.parse_page(page_html, threadid)
  File "/srv/vf/votefinder/main/SAPageParser.py", line 77, in parse_page
    game, game_created = Game.objects.get_or_create(thread_id=threadid,
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/db/models/query.py", line 923, in get_or_create
    return self.create(**params), True
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/db/models/query.py", line 658, in create
    obj.save(force_insert=True, using=self.db)
  File "/srv/vf/votefinder/main/models.py", line 161, in save
    self.update_counts()
  File "/srv/vf/votefinder/main/models.py", line 123, in update_counts
    self.players_count = self.count_players()
  File "/srv/vf/votefinder/main/models.py", line 168, in count_players
    return self.players.filter(spectator=False, moderator=False).count()
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/db/models/fields/related_descriptors.py", line 718, in get_queryset
    raise ValueError(
ValueError: 'Game' instance needs to have a primary key value before this relationship can be used.

Attempting to add a game on Bread and Roses causes a different issue:

Traceback (most recent call last):
  File "/srv/vf/votefinder/main/views.py", line 77, in add_game
    game = Game.objects.get(thread_id=threadid, home_forum=home_forum)
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/db/models/query.py", line 637, in get
    raise self.model.DoesNotExist(
votefinder.main.models.Game.DoesNotExist: Game matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/db/models/query.py", line 916, in get_or_create
    return self.get(**kwargs), False
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/db/models/query.py", line 637, in get
    raise self.model.DoesNotExist(
votefinder.main.models.Game.DoesNotExist: Game matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/contrib/auth/decorators.py", line 23, in _wrapper_view
    return view_func(request, *args, **kwargs)
  File "/srv/vf/votefinder/main/views.py", line 89, in add_game
    game = page_parser.add_game(threadid, state)
  File "/srv/vf/votefinder/main/SAPageParser.py", line 27, in add_game
    return self.download_and_update(f'https://forums.somethingawful.com/showthread.php?threadid={threadid}', threadid)
  File "/srv/vf/votefinder/main/SAPageParser.py", line 34, in download_and_update
    game = self.parse_page(page_html, threadid)
  File "/srv/vf/votefinder/main/SAPageParser.py", line 77, in parse_page
    game, game_created = Game.objects.get_or_create(thread_id=threadid,
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/db/models/query.py", line 923, in get_or_create
    return self.create(**params), True
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/db/models/query.py", line 658, in create
    obj.save(force_insert=True, using=self.db)
  File "/srv/vf/votefinder/main/models.py", line 161, in save
    self.update_counts()
  File "/srv/vf/votefinder/main/models.py", line 123, in update_counts
    self.players_count = self.count_players()
  File "/srv/vf/votefinder/main/models.py", line 168, in count_players
    return self.players.filter(spectator=False, moderator=False).count()
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/db/models/fields/related_descriptors.py", line 718, in get_queryset
    raise ValueError(
ValueError: 'Game' instance needs to have a primary key value before this relationship can be used.
[2023-09-03 19:54:49 -0400] [1447690] [DEBUG] POST /add_game
Internal Server Error: /add_game
Traceback (most recent call last):
  File "/srv/vf/votefinder/main/views.py", line 77, in add_game
    game = Game.objects.get(thread_id=threadid, home_forum=home_forum)
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/db/models/query.py", line 637, in get
    raise self.model.DoesNotExist(
votefinder.main.models.Game.DoesNotExist: Game matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/srv/vf/vfvenv/lib/python3.10/site-packages/django/contrib/auth/decorators.py", line 23, in _wrapper_view
    return view_func(request, *args, **kwargs)
  File "/srv/vf/votefinder/main/views.py", line 89, in add_game
    game = page_parser.add_game(threadid, state)
  File "/srv/vf/votefinder/main/BNRPageParser.py", line 27, in add_game
    return self.download_and_update(threadid)
  File "/srv/vf/votefinder/main/BNRPageParser.py", line 32, in download_and_update
    game = self.parse_page(thread, threadid)
  File "/srv/vf/votefinder/main/BNRPageParser.py", line 47, in parse_page
    self.pageNumber = thread['pagination']['current_page']
KeyError: 'pagination'

In either case, adding the game fails. I suspect that at some point, get_or_create had some changes moving from Django 3 to Django 4 that are causing this problem, but I can't know that for sure until I do some more work.

[rjmagley]

Appears to have been fixed - will certainly get some more robust testing as time goes by.