Upgrade django and python

[sagepe]

48 includes an update to Debian Buster and therefore Python 3.7. This is currently passing tests.

Django updates cleanly to 3.1.3, but I encountered some errors running scripts - I'll do a full test and provide details.

[sagepe]

Errors when running script/test having updated Django:

vagrant@buster:/vagrant/caps$ script/test
Creating test database for alias 'default'...
Destroying test database for alias 'default'...
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/django/template/utils.py", line 66, in __getitem__
    return self._engines[alias]
KeyError: 'django'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 121, in get_package_libraries
    module = import_module(entry[1])
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.7/dist-packages/pipeline/templatetags/pipeline.py", line 9, in <module>
    from django.template.base import Context, VariableDoesNotExist
ImportError: cannot import name 'Context' from 'django.template.base' (/usr/local/lib/python3.7/dist-packages/django/template/base.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "script/../manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.7/dist-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.7/dist-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.7/dist-packages/django/core/management/commands/test.py", line 23, in run_from_argv
    super().run_from_argv(argv)
  File "/usr/local/lib/python3.7/dist-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.7/dist-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.7/dist-packages/django/core/management/commands/test.py", line 53, in handle
    failures = test_runner.run_tests(test_labels)
  File "/usr/local/lib/python3.7/dist-packages/django/test/runner.py", line 698, in run_tests
    self.run_checks(databases)
  File "/usr/local/lib/python3.7/dist-packages/django/test/runner.py", line 636, in run_checks
    call_command('check', verbosity=self.verbosity, databases=databases)
  File "/usr/local/lib/python3.7/dist-packages/django/core/management/__init__.py", line 168, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python3.7/dist-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.7/dist-packages/django/core/management/commands/check.py", line 69, in handle
    databases=options['databases'],
  File "/usr/local/lib/python3.7/dist-packages/django/core/management/base.py", line 396, in check
    databases=databases,
  File "/usr/local/lib/python3.7/dist-packages/django/core/checks/registry.py", line 70, in run_checks
    new_errors = check(app_configs=app_configs, databases=databases)
  File "/usr/local/lib/python3.7/dist-packages/django/contrib/admin/checks.py", line 78, in check_dependencies
    for engine in engines.all():
  File "/usr/local/lib/python3.7/dist-packages/django/template/utils.py", line 90, in all
    return [self[alias] for alias in self]
  File "/usr/local/lib/python3.7/dist-packages/django/template/utils.py", line 90, in <listcomp>
    return [self[alias] for alias in self]
  File "/usr/local/lib/python3.7/dist-packages/django/template/utils.py", line 81, in __getitem__
    engine = engine_cls(params)
  File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 25, in __init__
    options['libraries'] = self.get_templatetag_libraries(libraries)
  File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 43, in get_templatetag_libraries
    libraries = get_installed_libraries()
  File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 108, in get_installed_libraries
    for name in get_package_libraries(pkg):
  File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 125, in get_package_libraries
    "trying to load '%s': %s" % (entry[1], e)
django.template.library.InvalidTemplateLibrary: Invalid template library specified. ImportError raised when trying to load 'pipeline.templatetags.pipeline': cannot import name 'Context' from 'django.template.base' (/usr/local/lib/python3.7/dist-packages/django/template/base.py)

[crowbot]

Does https://github.com/mysociety/caps/pull/38/commits/5a324f22a48f9669581bc70508ed44a67af1e983 help at all?

[sagepe]

Not by itself, but it was useful. Cherry-picking this commit on top of updating django-pipeline from 1.6.14 to 2.0.5 gets the tests passing.

Without this commit, but with an updated django-pipeline, the tests pass with the following warning:

?: (admin.W411) 'django.template.context_processors.request' must be enabled in DjangoTemplates (TEMPLATES) in order to use the admin navigation sidebar.

Which the commit resolves.

Thanks!

[sagepe]

OK, spoke to soon. Tests pass, but then the following error when trying to visit the site.

I'm wondering about django-pipeline-csscompressor?

Internal Server Error: /
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/base.py", line 202, in _get_response
    response = response.render()
  File "/usr/local/lib/python3.7/dist-packages/django/template/response.py", line 105, in render
    self.content = self.rendered_content
  File "/usr/local/lib/python3.7/dist-packages/django/template/response.py", line 83, in rendered_content
    return template.render(context, self._request)
  File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 61, in render
    return self.template.render(context)
  File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 170, in render
    return self._render(context)
  File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
    return self.nodelist.render(context)
  File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python3.7/dist-packages/django/template/loader_tags.py", line 150, in render
    return compiled_parent._render(context)
  File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
    return self.nodelist.render(context)
  File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python3.7/dist-packages/pipeline/templatetags/pipeline.py", line 136, in render
    return self.render_compressed(package, package_name, 'css')
  File "/usr/local/lib/python3.7/dist-packages/pipeline/templatetags/pipeline.py", line 70, in render_compressed
    package_type)
  File "/usr/local/lib/python3.7/dist-packages/pipeline/templatetags/pipeline.py", line 102, in render_compressed_sources
    paths = packager.compile(package.paths)
  File "/usr/local/lib/python3.7/dist-packages/pipeline/packager.py", line 104, in compile
    force=force,
  File "/usr/local/lib/python3.7/dist-packages/pipeline/compilers/__init__.py", line 53, in compile
    return list(executor.map(_compile, paths))
  File "/usr/lib/python3.7/concurrent/futures/_base.py", line 586, in result_iterator
    yield fs.pop().result()
  File "/usr/lib/python3.7/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/dist-packages/pipeline/compilers/__init__.py", line 40, in _compile
    **compiler_options)
  File "/usr/local/lib/python3.7/dist-packages/pipeline/compilers/sass.py", line 20, in compile_file
    return self.execute_command(command, cwd=dirname(infile))
  File "/usr/local/lib/python3.7/dist-packages/pipeline/compilers/__init__.py", line 133, in execute_command
    error_output=stderr)
pipeline.exceptions.CompilerError: ['sassc', '/vagrant/caps/caps/static/scss/main.scss', '/vagrant/caps/static/scss/main.css'] exit code 1
b'sassc: error: Error: File to import not found or unreadable: ../bootstrap/functions.\n       Parent style sheet: /vagrant/caps/caps/static/scss/main.scss\n        on line 18 of main.scss\n>> @import "../bootstrap/functions";\n   ^\n\n'
[05/Nov/2020 12:08:27] "GET / HTTP/1.1" 500 169994

[sagepe]

We should update to the latest Python 3.9 and switch Vagrant to Bullseye; as part of this we should use an exact minor version tag in the Dockerfile for more consistent builds, plus we can reliably tie the version of Python in the image to a given git commit that way.

[struan]

I've fixed the include thing which was pipeline changing where it was getting the compiled files from.

I'm not sure what you mean by an exact minor version in the Dockerfile. Is that for python?

[sagepe]

I'm not sure what you mean by an exact minor version in the Dockerfile. Is that for python?

Yes, basically what you've done in d66b23d0