LDAP login: Python package jose missing, Python syntax error during login

[SethosII]

Describe the bug

I try to setup LDAP login with Active Directory. There is the Beep Boop, Error! page after I enter my credentials and click on login. The logs show this:

uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/social_core/backends/open_id_connect.py", line 5, in <module>
uwsgi_1      |     from jose import jwk, jwt
uwsgi_1      | ModuleNotFoundError: No module named 'jose'

Therefore I added jose to the requirements.txt and rebuilt the image. Now I still get the Beep Boop, Error! page after I enter me credentials and click on login, but the logs show this:

uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/social_core/backends/open_id_connect.py", line 5, in <module>
uwsgi_1      |     from jose import jwk, jwt
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/jose.py", line 546
uwsgi_1      |     print decrypt(deserialize_compact(jwt), {'k':key},
uwsgi_1      |           ^
uwsgi_1      | SyntaxError: invalid syntax**

So this seeams to be a Python error not a configuration error. Also, jose seems to be outdated (last release on pypi 2015) so probably it doesn't work with Python 3. Do you have any ideas how to fix this?

To Reproduce

Setup sregsitry for LDAP login and try to login.

I followed these steps:

sudo apt install docker.io docker-compose
git clone https://github.com/singularityhub/sregistry
cd sregistry/
cp shub/dummy-settings.yaml settings.yaml
vi settings.yaml (I modified SECRET_KEY, DOMAIN_NAME, DOMAIN_NAME_HTTP, REGISTRY_NAME, REGISTRY_URI, AUTH_LDAP_SERVER_URI, AUTH_LDAP_BIND_DN, AUTH_LDAP_BIND_PASSWORD, AUTH_LDAP_USER_SEARCH, AUTH_LDAP_GROUP_SEARCH, AUTH_LDAP_SUPERUSER_GROUP_FLAGS
vi Dockerfile (I modified ARG ENABLE_LDAP)
vi shub/settings.py (I modified ldap_auth)
sudo docker build --build-arg ENABLE_LDAP=true -t quay.io/vanessa/sregistry .
sudo docker-compose up
# during LDAP login:
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/social_core/backends/open_id_connect.py", line 5, in <module>
uwsgi_1      |     from jose import jwk, jwt
uwsgi_1      | ModuleNotFoundError: No module named 'jose'
vi requirements.txt (I added jose)
# afterwards during LDAP login:
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/social_core/backends/open_id_connect.py", line 5, in <module>
uwsgi_1      |     from jose import jwk, jwt
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/jose.py", line 546
uwsgi_1      |     print decrypt(deserialize_compact(jwt), {'k':key},
uwsgi_1      |           ^
uwsgi_1      | SyntaxError: invalid syntax**

Expected behavior

Login should work.

If applicable, add versions and screenshots to help explain your problem.

I used a git clone of the master branch to setup sregistry on a fresh installation of Ubuntu 22.04.

[vsoch]

It looks like it’s social core that brings that dependency in. And you don’t want jose typically, you want python-jose and possibly python-jose[cryptography].

and yes we need to look into what dependencies are triggering these issues and likely update or pin.

[SethosII]

@vsoch Thank you for the hint. I added the package to requirements.txt and created a pull request for it: https://github.com/singularityhub/sregistry/pull/418.

Now there is a new error during login:

uwsgi_1      | Binding as CN=X.X,OU=Service-Accounts,DC=company,DC=com
uwsgi_1      | Caught LDAPError while authenticating XY.XY: SERVER_DOWN({'result': -1, 'desc': "Can't contact LDAP server", 'ctrls': [], 'info': '(unknown error code)'})
uwsgi_1      | [pid: 28|app: 0|req: 2/4] 10.XXX.XXX.XXX () {50 vars in 1024 bytes} [Mon Mar 27 01:54:40 2023] POST /ldap_auth/login/ => generated 5994 bytes in 344 msecs (HTTP/1.1 200) 7 headers in 395 bytes (1 switches on core 3)

I copied the values for AUTH_LDAP_SERVER_URI, AUTH_LDAP_BIND_DN and AUTH_LDAP_BIND_PASSWORD from a working LDAP setup, so these values should work. Also I'm able to ping the LDAP server. I set the base (DC=company,DC=com) for AUTH_LDAP_USER_SEARCH and AUTH_LDAP_GROUP_SEARCH, this should be no problem? Is there anything else to look out for?

[vsoch]

Perhaps start with the example https://singularityhub.github.io/sregistry/docs/plugins/ldap confirm it works, and change one thing at a time until you reproduce the error?

[SethosII]

I tried to follow these steps but there are some other errors. First thing is in the section 'Interact with it' line 87. bash isn't available in the container but sh is, so it should be docker exec -it openldap sh. Then in section 'Add a user and group to the directory' I get this:

ldapsearch -x -b 'dc=my-company,dc=com'                                                                                        
# extended LDIF                                                                                                                                               
#                                                                                                                                                             
# LDAPv3                                                                                                                                                      
# base <dc=my-company,dc=com> with scope subtree                                                                                                              
# filter: (objectclass=*)                                                                                                                                     
# requesting: ALL                                                                                                                                             
#                                                                                                                                                             

# search result
search: 2
result: 32 No such object

# numResponses: 1

32 No such object is also the return value for subsequent commands (adding test users and groups).

Also the image for the LDAP login page isn't displayed at the bottom of this page. Line 326 should be ![ldap.png](ldap/ldap.png).

I will create a pull request for the documentation corrections.

[SethosII]

I got a bit further but LDAP authentication still doesn't work. The current error while logging in is this:

uwsgi_1      | Binding as CN=X.X,OU=Service-Accounts,DC=company,DC=com
uwsgi_1      | Invoking search_s('dc=company,dc=com', 2, '(uid=Y.Y)')
uwsgi_1      | search_s('dc=company,dc=com', 2, '(uid=Y.Y)') raised OPERATIONS_ERROR({'msgtype': 115, 'msgid': 2, 'result': 1, 'desc': 'Operations error', 'ctrls': [], 'info': '000004DC: LdapErr: DSID-0C0907ED, comment: In order to perform this operation a successful bind must be completed on the connection., data 0, v2580'})
uwsgi_1      | search_s('dc=company,dc=com', 2, '(uid=%(user)s)') returned 0 objects: 
uwsgi_1      | Authentication failed for Y.Y: failed to map the username to a DN.
uwsgi_1      | [pid: 28|app: 0|req: 3/3] 10.XXX.XXX.XXX () {50 vars in 1024 bytes} [Wed Mar 29 08:32:28 2023] POST /ldap_auth/login/ => generated 5994 bytes in 599 msecs (HTTP/1.1 200) 7 headers in 395 bytes (1 switches on core 2)

So, I think the attribute mapping is still missing. I use the settings.yaml and there is no exmaple for it there. How would an example look like in the YAML syntax?

[vsoch]

You define it via: https://singularityhub.github.io/sregistry/docs/plugins/ldap#configure-sregistry

[SethosII]

I know this one, but I'm looking for a way to set this in the settings.yaml. Shouldn't these be functionally equivalent? I tried it like this:

AUTH_LDAP_USER_ATTR_MAP:
  first_name: "givenName"
  last_name: "sn"
  email: "mail"

However, currently I'm getting a ModuleNotFoundError for django.utils.six when starting the containers if LDAP is enabled in the sregistry image:

uwsgi_1      | Traceback (most recent call last):                                                                                                             
uwsgi_1      |   File "/code/manage.py", line 9, in <module>                                                                                                  
uwsgi_1      |     execute_from_command_line(sys.argv)                                                                                                        
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line                     
uwsgi_1      |     utility.execute()                                                                                                                          
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 420, in execute                                       
uwsgi_1      |     django.setup()                                                                                                                             
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
uwsgi_1      |     apps.populate(settings.INSTALLED_APPS)
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/django/apps/registry.py", line 116, in populate
uwsgi_1      |     app_config.import_models()
uwsgi_1      |   File "/usr/local/lib/python3.9/site-packages/django/apps/config.py", line 269, in import_models
uwsgi_1      |     self.models_module = import_module(models_module_name)
uwsgi_1      |   File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
uwsgi_1      |     return _bootstrap._gcd_import(name[level:], package, level)
uwsgi_1      |   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
uwsgi_1      |   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
uwsgi_1      |   File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
uwsgi_1      |   File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
uwsgi_1      |   File "<frozen importlib._bootstrap_external>", line 850, in exec_module
uwsgi_1      |   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
uwsgi_1      |   File "/code/shub/apps/logs/models.py", line 11, in <module>
uwsgi_1      |     from django.utils.six import python_2_unicode_compatible
uwsgi_1      | ModuleNotFoundError: No module named 'django.utils.six'

This module seems to be removed in Django 3 and while building the container there is this output:

Step 18/40 : RUN if $ENABLE_LDAP; then pip install django-auth-ldap ; fi;                                                                                     
 ---> Running in 8db5bb4d2cd6                                                                                                                                 
Collecting django-auth-ldap                                                                                                                                   
  Downloading django_auth_ldap-4.2.0-py3-none-any.whl (20 kB)                                                                                                 
Collecting Django>=3.2                                                                                                                                        
  Downloading Django-4.1.7-py3-none-any.whl (8.1 MB)                                                                                                          
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.1/8.1 MB 55.8 MB/s eta 0:00:00                                                                                
Collecting python-ldap>=3.1                                                                                                                                   
  Downloading python-ldap-3.4.3.tar.gz (377 kB)                                                                                                               
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 377.4/377.4 kB 39.6 MB/s eta 0:00:00                                                                              
  Installing build dependencies: started                                                                                                                      
  Installing build dependencies: finished with status 'done'                                                                                                  
  Getting requirements to build wheel: started                                                                                                                
  Getting requirements to build wheel: finished with status 'done'                                                                                            
  Preparing metadata (pyproject.toml): started                                                                                                                
  Preparing metadata (pyproject.toml): finished with status 'done'                                                                                            
Requirement already satisfied: sqlparse>=0.2.2 in /usr/local/lib/python3.9/site-packages (from Django>=3.2->django-auth-ldap) (0.4.3)                         
Collecting asgiref<4,>=3.5.2                                                                                                                                  
  Downloading asgiref-3.6.0-py3-none-any.whl (23 kB)                                                                                                          
Requirement already satisfied: pyasn1-modules>=0.1.5 in /usr/local/lib/python3.9/site-packages (from python-ldap>=3.1->django-auth-ldap) (0.2.8)              
Requirement already satisfied: pyasn1>=0.3.7 in /usr/local/lib/python3.9/site-packages (from python-ldap>=3.1->django-auth-ldap) (0.4.8)                      
Building wheels for collected packages: python-ldap                                                                                                           
  Building wheel for python-ldap (pyproject.toml): started                                                                                                    
  Building wheel for python-ldap (pyproject.toml): finished with status 'done'                                                                                
  Created wheel for python-ldap: filename=python_ldap-3.4.3-cp39-cp39-linux_x86_64.whl size=314161 sha256=b8a3898a5884b999f309f4189b032e4bd02fe00e631d98ef074a
3d545d67fc40                                                                                                                                                  
  Stored in directory: /root/.cache/pip/wheels/79/43/5e/1a029878450f5949466f351db7ff4c5459c0b1937999f39000                                                    
Successfully built python-ldap                                                                                                                                
Installing collected packages: asgiref, python-ldap, Django, django-auth-ldap                                                                                 
  Attempting uninstall: Django                                                                                                                                
    Found existing installation: Django 2.2.28                                                                                                                
    Uninstalling Django-2.2.28:                                                                                                                               
      Successfully uninstalled Django-2.2.28                                                                                                                  
Successfully installed Django-4.1.7 asgiref-3.6.0 django-auth-ldap-4.2.0 python-ldap-3.4.3                                                                    
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a
 virtual environment instead: https://pip.pypa.io/warnings/venv                                                                                               
Removing intermediate container 8db5bb4d2cd6                   
 ---> ff9aa5d5eb10

So, Django 2 seems to be replaced by Django 4 and therefore django.utils.six isn't available anymore?

[vsoch]

We probably should update the container base and Django versions. Share with me your Dockerfile to reproduce the error and I can make some time this weekend.

[SethosII]

The Dockerfile looks like this:

FROM python:3.9
ENV PYTHONUNBUFFERED 1
ENV DEBIAN_FRONTEND noninteractive
ENV MESSAGELEVEL QUIET

ARG ENABLE_LDAP=true
ARG ENABLE_PAM=false
ARG ENABLE_PGP=false
ARG ENABLE_GOOGLEBUILD=false
ARG ENABLE_GLOBUS=false
ARG ENABLE_SAML=false

################################################################################
# CORE
# Do not modify this section

RUN apt-get update && apt-get install -y \
    pkg-config \
    cmake \
    openssl \
    wget \
    git \
    vim

RUN apt-get update && apt-get install -y \
    anacron \
    autoconf \
    automake \
    libarchive-dev \
    libtool \
    libopenblas-dev \
    libglib2.0-dev \
    gfortran \
    libxml2-dev \
    libxmlsec1-dev \
    libhdf5-dev \
    libgeos-dev \
    libsasl2-dev \
    libldap2-dev \
    squashfs-tools \
    build-essential

# Install Python requirements out of /tmp so not triggered if other contents of /code change
COPY requirements.txt /tmp/requirements.txt
RUN pip install --upgrade pip
RUN pip install -r /tmp/requirements.txt

COPY . /code/

################################################################################
# PLUGINS
# You are free to uncomment the plugins that you want to use

# Install LDAP (uncomment if wanted)
RUN if $ENABLE_LDAP; then pip install python3-ldap ; fi;
RUN if $ENABLE_LDAP; then pip install django-auth-ldap ; fi;

# Install PAM Authentication (uncomment if wanted)
RUN if $ENABLE_PAM; then pip install django-pam ; fi;

# PGP keystore dependencies
RUN if $ENABLE_PGP; then pip install pgpdump>=1.4; fi;

# Ensure Google Build Installed
RUN if $ENABLE_GOOGLEBUILD; then pip install sregistry[google-build] ; fi;
ENV SREGISTRY_GOOGLE_STORAGE_PRIVATE=true

# Install Globus (uncomment if wanted)
RUN if $ENABLE_GLOBUS; then /bin/bash /code/scripts/globus/globus-install.sh ; fi;

# Install SAML (uncomment if wanted)
RUN if $ENABLE_SAML; then pip install python3-saml ; fi;
RUN if $ENABLE_SAML; then pip install social-auth-core[saml] ; fi;

################################################################################
# BASE

RUN mkdir -p /code && mkdir -p /code/images
RUN mkdir -p /var/www/images && chmod -R 0755 /code/images/

WORKDIR /code
RUN apt-get remove -y gfortran

RUN apt-get autoremove -y
RUN apt-get clean
RUN rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

# Install crontab to setup jobs
RUN echo "0 0 * * * /usr/local/bin/python /code/manage.py reset_container_limits > /var/log/reset_container_limits.log 2>&1 " >> /code/cronjob
RUN echo "0 1 * * * /bin/bash /code/scripts/backup_db.sh > /var/log/backup_db.log 2>&1 " >> /code/cronjob
RUN echo "0 2 * * * /usr/local/bin/python /code/manage.py cleanup_dummy > /var/log/cleanup_dummy.log 2>&1 " >> /code/cronjob
RUN crontab /code/cronjob
RUN rm /code/cronjob

# Create hashed temporary upload locations
RUN mkdir -p /var/www/images/_upload/{0..9} && chmod 777 -R /var/www/images/_upload

CMD /code/run_uwsgi.sh

EXPOSE 3031

The only difference to the current one from the repository is ARG ENABLE_LDAP=true.

[vsoch]

@SethosII here is a branch to test! https://github.com/singularityhub/sregistry/pull/421

[SethosII]

I also got LDAP authentication to work. The last error I had was this: Authentication failed for USERNAME: failed to map the username to a DN. This solved it: https://stackoverflow.com/a/37033098. So I had to replace uid with samaccountname here: https://github.com/singularityhub/sregistry/blob/master/shub/settings.py#L580. Maybe this should be added to the documentation or is there a way to automate it if LDAP is enabled?

[vsoch]

Great! I added that note to the PR in progress (sorry actually added this morning early before work but forgot to commit and push). Let me know if/when that PR is all good for you and we can merge and release.

[SethosII]

LDAP authentication now works for me. The only thing I haven't figured out is connecting via LDAPS, but that is not required for me at the moment. The error I get when using LDAPS is Caught LDAPError while authenticating USERNAME: SERVER_DOWN({'result': -1, 'desc': "Can't contact LDAP server", 'ctrls': [], 'info': '(unknown error code)'}).