update dockerfile base

[vsoch]

This will update Django / other deps in the container.

[vsoch]

It doesn't look like you built a container from the new code - I removed those deprecated calls - but let me double check if I did for plugins too.

[SethosII]

You replaced it in most cases but not in the plugins. See for example here: https://github.com/singularityhub/sregistry/blob/update/dockerfile-base/shub/plugins/ldap_auth/urls.py#L1.

[vsoch]

Yep seeing that now! Pushing a fix soon (just getting the version of black right!)

[SethosII]

Now I'm getting this error during startup:

uwsgi_1      | Traceback (most recent call last):
uwsgi_1      |   File "/code/manage.py", line 9, in <module>
uwsgi_1      |     execute_from_command_line(sys.argv)
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
uwsgi_1      |     utility.execute()
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
uwsgi_1      |     self.fetch_command(subcommand).run_from_argv(self.argv)
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
uwsgi_1      |     self.execute(*args, **cmd_options)
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 453, in execute
uwsgi_1      |     self.check()
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 485, in check
uwsgi_1      |     all_issues = checks.run_checks(
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/django/core/checks/registry.py", line 88, in run_checks
uwsgi_1      |     new_errors = check(app_configs=app_configs, databases=databases)
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/django/core/checks/urls.py", line 14, in check_url_config
uwsgi_1      |     return check_resolver(resolver)
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/django/core/checks/urls.py", line 24, in check_resolver
uwsgi_1      |     return check_method()
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/django/urls/resolvers.py", line 494, in check
uwsgi_1      |     for pattern in self.url_patterns:
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
uwsgi_1      |     res = instance.__dict__[self.name] = self.func(instance)
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/django/urls/resolvers.py", line 715, in url_patterns
uwsgi_1      |     patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
uwsgi_1      |     res = instance.__dict__[self.name] = self.func(instance)
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/django/urls/resolvers.py", line 708, in urlconf_module
uwsgi_1      |     return import_module(self.urlconf_name)
uwsgi_1      |   File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
uwsgi_1      |     return _bootstrap._gcd_import(name[level:], package, level)
uwsgi_1      |   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
uwsgi_1      |   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
uwsgi_1      |   File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
uwsgi_1      |   File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
uwsgi_1      |   File "<frozen importlib._bootstrap_external>", line 850, in exec_module
uwsgi_1      |   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
uwsgi_1      |   File "/code/shub/urls.py", line 17, in <module>
uwsgi_1      |     from rest_framework.documentation import include_docs_urls
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/rest_framework/documentation.py", line 3, in <module>
uwsgi_1      |     from rest_framework.renderers import (
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/rest_framework/renderers.py", line 17, in <module>
uwsgi_1      |     from django.http.multipartparser import parse_header
uwsgi_1      | ImportError: cannot import name 'parse_header' from 'django.http.multipartparser' (/usr/local/lib/python3.9/site-packages/django/http/multipart
parser.py)

[vsoch]

Did you completely stop, remove, rebuild, and start again?

[SethosII]

Yes, I removed everything like that:

sudo docker system prune --volumes -f
sudo docker rmi -f $(sudo docker images -aq)
sudo rm -rf sregistry/
git clone https://github.com/singularityhub/sregistry
git checkout update/dockerfile-base

[vsoch]

But did you build the base container again? The dependencies have changed so you would need to rebuild the quay.io main image with the new Dockerfile.

[SethosII]

Yes, I rebuild the image like so: sudo docker build --build-arg ENABLE_LDAP=true -t quay.io/vanessa/sregistry ..

During the build with LDAP enabled there is still a reference that Django 2.2.28 gets replaced by 4.2:

Step 18/40 : RUN if $ENABLE_LDAP; then pip install django-auth-ldap ; fi;                                                                                     
 ---> Running in 7c7c9596c3a6                                                                                                                                 
Collecting django-auth-ldap                                                                                                                                   
  Downloading django_auth_ldap-4.2.0-py3-none-any.whl (20 kB)                                                                                                 
Collecting python-ldap>=3.1                                                                                                                                   
  Downloading python-ldap-3.4.3.tar.gz (377 kB)                                                                                                               
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 377.4/377.4 kB 14.5 MB/s eta 0:00:00                                                                              
  Installing build dependencies: started                                                                                                                      
  Installing build dependencies: finished with status 'done'                                                                                                  
  Getting requirements to build wheel: started                                                                                                                
  Getting requirements to build wheel: finished with status 'done'                                                                                            
  Preparing metadata (pyproject.toml): started                                                                                                                
  Preparing metadata (pyproject.toml): finished with status 'done'                                                                                            
Collecting Django>=3.2                                                                                                                                        
  Downloading Django-4.2-py3-none-any.whl (8.0 MB)                                                                                                            
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.0/8.0 MB 74.9 MB/s eta 0:00:00                                                                                
Collecting asgiref<4,>=3.6.0                                                                                                                                  
  Downloading asgiref-3.6.0-py3-none-any.whl (23 kB)                                                                                                          
Requirement already satisfied: sqlparse>=0.3.1 in /usr/local/lib/python3.9/site-packages (from Django>=3.2->django-auth-ldap) (0.4.3)                         
Requirement already satisfied: pyasn1-modules>=0.1.5 in /usr/local/lib/python3.9/site-packages (from python-ldap>=3.1->django-auth-ldap) (0.2.8)              
Requirement already satisfied: pyasn1>=0.3.7 in /usr/local/lib/python3.9/site-packages (from python-ldap>=3.1->django-auth-ldap) (0.4.8)
Building wheels for collected packages: python-ldap                            
  Building wheel for python-ldap (pyproject.toml): started                     
  Building wheel for python-ldap (pyproject.toml): finished with status 'done' 
  Created wheel for python-ldap: filename=python_ldap-3.4.3-cp39-cp39-linux_x86_64.whl size=314174 sha256=ef6075a80bfce35b12ab63ecfd7e038497b67cb0bb3cba7819eb
7e49a4af26dc                 
  Stored in directory: /root/.cache/pip/wheels/79/43/5e/1a029878450f5949466f351db7ff4c5459c0b1937999f39000
Successfully built python-ldap
Installing collected packages: asgiref, python-ldap, Django, django-auth-ldap
  Attempting uninstall: Django
    Found existing installation: Django 2.2.28
    Uninstalling Django-2.2.28:
      Successfully uninstalled Django-2.2.28
Successfully installed Django-4.2 asgiref-3.6.0 django-auth-ldap-4.2.0 python-ldap-3.4.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a
 virtual environment instead: https://pip.pypa.io/warnings/venv
Removing intermediate container 7c7c9596c3a6
 ---> 39fb31cdee35

[vsoch]

I intended to remove the Django pin from the requirements.txt - feel free to do that for now, and the update to 4.x is what we want.

It's late here, but I can test this again tomorrow to see if I can reproduce your build.

[SethosII]

I removed the Version from Django in requirements.txt the step now looks like this:

Step 18/40 : RUN if $ENABLE_LDAP; then pip install django-auth-ldap ; fi;
 ---> Running in 474b98311594                                                  
Collecting django-auth-ldap  
  Downloading django_auth_ldap-4.2.0-py3-none-any.whl (20 kB)
Requirement already satisfied: Django>=3.2 in /usr/local/lib/python3.9/site-packages (from django-auth-ldap) (4.2)
Collecting python-ldap>=3.1                                                                                                                                   
  Downloading python-ldap-3.4.3.tar.gz (377 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 377.4/377.4 kB 19.1 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'      
Requirement already satisfied: sqlparse>=0.3.1 in /usr/local/lib/python3.9/site-packages (from Django>=3.2->django-auth-ldap) (0.4.3)
Requirement already satisfied: asgiref<4,>=3.6.0 in /usr/local/lib/python3.9/site-packages (from Django>=3.2->django-auth-ldap) (3.6.0)
Requirement already satisfied: pyasn1-modules>=0.1.5 in /usr/local/lib/python3.9/site-packages (from python-ldap>=3.1->django-auth-ldap) (0.2.8)
Requirement already satisfied: pyasn1>=0.3.7 in /usr/local/lib/python3.9/site-packages (from python-ldap>=3.1->django-auth-ldap) (0.4.8)
Building wheels for collected packages: python-ldap
  Building wheel for python-ldap (pyproject.toml): started
  Building wheel for python-ldap (pyproject.toml): finished with status 'done'
  Created wheel for python-ldap: filename=python_ldap-3.4.3-cp39-cp39-linux_x86_64.whl size=314178 sha256=f732dc00bbdc937cfcaf74e1ca046c5382379ff7fb5231e37fde
34661b7af7a9                 
  Stored in directory: /root/.cache/pip/wheels/79/43/5e/1a029878450f5949466f351db7ff4c5459c0b1937999f39000
Successfully built python-ldap
Installing collected packages: python-ldap, django-auth-ldap
Successfully installed django-auth-ldap-4.2.0 python-ldap-3.4.3                                                                                               
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a
 virtual environment instead: https://pip.pypa.io/warnings/venv
Removing intermediate container 474b98311594
 ---> 9fe64a6aa4d1

However, when opening the website I get a very long error message on the server (first thing is KeyError: 'staticfiles'). I attached it as a text file.

[vsoch]

okay I reproduced and pushed a bunch of changes:

• staticfiles -> static
• added default auto field for models
• pinned django to new 4.2

[SethosII]

That fixes the errors, thanks!

[vsoch]

@SethosII give me the A-ok when all is good here to merge and release. Thanks!

[karcaw]

If i update to this commit from a4b0bf0 I get the following error:

uwsgi_1      | Traceback (most recent call last):
uwsgi_1      |   File "shub/wsgi.py", line 7, in <module>
uwsgi_1      |     application = get_wsgi_application()
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
uwsgi_1      |     django.setup(set_prefix=False)
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
uwsgi_1      |     apps.populate(settings.INSTALLED_APPS)
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/django/apps/registry.py", line 114, in populate
uwsgi_1      |     app_config.import_models()
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/django/apps/config.py", line 211, in import_models
uwsgi_1      |     self.models_module = import_module(models_module_name)
uwsgi_1      |   File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
uwsgi_1      |     return _bootstrap._gcd_import(name[level:], package, level)
uwsgi_1      |   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
uwsgi_1      |   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
uwsgi_1      |   File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
uwsgi_1      |   File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
uwsgi_1      |   File "<frozen importlib._bootstrap_external>", line 850, in exec_module
uwsgi_1      |   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
uwsgi_1      |   File "/code/./shub/apps/logs/models.py", line 51, in <module>
uwsgi_1      |     class APIRequestCount(models.Model):
uwsgi_1      |   File "/code/./shub/apps/logs/models.py", line 56, in APIRequestCount
uwsgi_1      |     from shub.apps.main.models import Collection
uwsgi_1      |   File "/code/./shub/apps/main/models/__init__.py", line 1, in <module>
uwsgi_1      |     from .containers import Container
uwsgi_1      |   File "/code/./shub/apps/main/models/containers.py", line 40, in <module>
uwsgi_1      |     class Container(models.Model):
uwsgi_1      |   File "/code/./shub/apps/main/models/containers.py", line 58, in Container
uwsgi_1      |     metadata = models.JSONField(default=dict, blank=True)
uwsgi_1      | AttributeError: module 'django.db.models' has no attribute 'JSONField'

Is there something i am missing in the update?

[vsoch]

That looks like you are using an old version of Django - the new install should be using Django 4.x that has JSONField added. Can you double check the version you have in this container?

[karcaw]

looks like it has Django 2.2.28

do i need to request a specific version of the sregistry docker image?

Also, I am just using the public images with the docker-compose.yml file currently. I'll try to build my own image if that would be better.

[vsoch]

If you are pulling the updated code from the repository here, you also need to use the matching image, so yes I'd pull the newer image. But you want to be very careful about this - I can't guarantee that you won't run into issues migrating. I would make a backup of everything first, or just stick to the older code. One of the challenges of maintaining these kinds of apps!

[karcaw]

just testing at this point, so no worries about data in the image. the older code seemed to have issues with the larger images i need to upload, so I was trying to move to the newer code that seems to handle it better..

[karcaw]

The containers I build myself from the Dockerfile work fine. just not the 'latest' tags out there on quay.io

[vsoch]

You're right! The last update was 6 months ago, and I'm not sure about why https://quay.io/repository/vanessa/sregistry?tab=tags. I think our best bet is to probably build to ghcr.io alongside the repository - I'll get this tweak in soon. Thanks for noticing I had not!