Evidentally, sometimes no `default_to_screenshot_view`?

[rebeccacremona]

Internal Server Error: /api/v1/archives/

IntegrityError at /api/v1/archives/
null value in column "default_to_screenshot_view" violates not-null constraint
DETAIL:  Failing row contains (f, null, Q97S-WXR6, <redacted>, 2022-11-18 16:27:27.421379+00, <redacted>, , f, null, f, 2022-11-19 16:27:27.421379+00, null, 74978, null, null, not_started, null, null, <redacted>, null, f, null).

Request Method: POST
Request URL: https://perma.cc/api/v1/archives/
Django Version: 2.2.28
Python Executable: /usr/bin/uwsgi-core
Python Version: 3.9.2
Server time: Fri, 18 Nov 2022 16:27:27 +0000

Traceback:  

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/django/db/backends/utils.py" in _execute
  84.                 return self.cursor.execute(sql, params)

      The above exception (null value in column "default_to_screenshot_view" violates not-null constraint
DETAIL:  Failing row contains (f, null, Q97S-WXR6, <redacted>, 2022-11-18 16:27:27.421379+00, <redacted>, , f, null, f, 2022-11-19 16:27:27.421379+00, null, 74978, null, null, not_started, null, null, <redacted>, null, f, null).
) was the direct cause of the following exception:

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/django/core/handlers/exception.py" in inner
  34.             response = get_response(request)

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/django/core/handlers/base.py" in _get_response
  115.                 response = self.process_exception_by_middleware(e, request)

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/django/core/handlers/base.py" in _get_response
  113.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/sentry_sdk/integrations/django/views.py" in sentry_wrapped_callback
  67.             return callback(request, *args, **kwargs)

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/django/views/decorators/csrf.py" in wrapped_view
  54.         return view_func(*args, **kwargs)

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/django/views/generic/base.py" in view
  71.             return self.dispatch(request, *args, **kwargs)

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/rest_framework/views.py" in dispatch
  509.             response = self.handle_exception(exc)

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/rest_framework/views.py" in handle_exception
  469.             self.raise_uncaught_exception(exc)

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/rest_framework/views.py" in raise_uncaught_exception
  480.         raise exc

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/rest_framework/views.py" in dispatch
  506.             response = handler(request, *args, **kwargs)

File "/usr/local/share/perma/perma_web/api/utils.py" in func_wrapper
  111.         return func(self, request, *args, **kwargs)

File "/usr/local/share/perma/perma_web/api/views.py" in post
  486.                 link = serializer.save(created_by=request.user, bonus_link=bonus_link)

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/rest_framework/serializers.py" in save
  212.             self.instance = self.create(validated_data)

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/rest_framework/serializers.py" in create
  962.             instance = ModelClass._default_manager.create(**validated_data)

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/django/db/models/manager.py" in manager_method
  82.                 return getattr(self.get_queryset(), name)(*args, **kwargs)

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/django/db/models/query.py" in create
  422.         obj.save(force_insert=True, using=self.db)

File "./perma/models.py" in save

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/django/db/models/base.py" in save
  743.         self.save_base(using=using, force_insert=force_insert,

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/model_utils/tracker.py" in inner
  375.                 return original(instance, *args, **kwargs)

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/django/db/models/base.py" in save_base
  780.             updated = self._save_table(

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/django/db/models/base.py" in _save_table
  873.             result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/django/db/models/base.py" in _do_insert
  910.         return manager._insert([self], fields=fields, return_id=update_pk,

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/django/db/models/manager.py" in manager_method
  82.                 return getattr(self.get_queryset(), name)(*args, **kwargs)

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/django/db/models/query.py" in _insert
  1186.         return query.get_compiler(using=using).execute_sql(return_id)

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/django/db/models/sql/compiler.py" in execute_sql
  1377.                 cursor.execute(sql, params)

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/sentry_sdk/integrations/django/__init__.py" in execute
  544.             return real_execute(self, sql, params)

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/django/db/backends/utils.py" in execute
  67.         return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/django/db/backends/utils.py" in _execute_with_wrappers
  76.         return executor(sql, params, many, context)

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/django/db/backends/utils.py" in _execute
  84.                 return self.cursor.execute(sql, params)

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/django/db/utils.py" in __exit__
  89.                 raise dj_exc_value.with_traceback(traceback) from exc_value

File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/django/db/backends/utils.py" in _execute
  84.                 return self.cursor.execute(sql, params)

Exception Type: IntegrityError at /api/v1/archives/
Exception Value: null value in column "default_to_screenshot_view" violates not-null constraint
DETAIL:  Failing row contains (f, null, Q97S-WXR6, <redacted>, 2022-11-18 16:27:27.421379+00, <redacted>, , f, null, f, 2022-11-19 16:27:27.421379+00, null, 74978, null, null, not_started, null, null, <redacted>, null, f, null).

[rebeccacremona]

A little unexpected that the default didn't kick in: to be investigated.

[rebeccacremona]

Update: I believed from my initial review of the emails about this error that a problem was happening for API users. That's incorrect.

We saw fewer than a dozen integrity errors like the above, a few for each kind of user (GUI users at perma.cc/api/v1, API users at api.perma.cc authing via headers, API users at api.perma.cc authing via the querystring).... all at 11:27AM on November the 18th. That was right at the very tail end of a deployment where the default-to-screenshot feature was introduced:

So here's what I'm pretty sure happened.

Somehow or other, the database got migrated and the new non-nullable column was added.... but a few lingering requests somehow were processed using the old application code in the meantime (Django supplies default values via the python, not by designating a default value for the column at the db level).

We enter maintenance mode as the first step of a deployment, preventing additional requests, but what happens to requests that are already in-flight is a bit of a mystery to me. This suggests that uwsgi continued with the already in-flight requests normally, and then, after it got bounced, picked up new requests with the new code as expected... and in this case, the DB migration was fast enough that the in-flight requests talked to the migrated DB.

How about that.

It hasn't happened again since, and I can't reproduce.

Bringing to @bensteinberg's attention, in case we want to think about implications for deployment strategies, but also closing as #wontfix.