Running with docker-compose give ModuleNotFoundError

[phillipjohnson]

When trying to run DSA with the docker-compose file, the girder and worker containers repeat this block in the logs over and over:

The user `root' is already a member of `root'.
id: ‘987’: no such user
net.ipv4.conf.eth0.route_localnet = 1
Traceback (most recent call last):
  File "/opt/digital_slide_archive/devops/dsa/provision.py", line 18, in <module>
    from girder_large_image.models.image_item import ImageItem
ModuleNotFoundError: No module named 'girder_large_image'

When running this, it looks like the module is present docker run -it --rm dsarchive/dsa_common pip list, so perhaps that error is a red herring. I am not sure about the user 987, I'm running this as root.

RHEL 7.9 Docker 20.10.6 Image is dsarchive/dsa_common:latest

[manthey]

I'm surprised by the ModuleNotFoundError: No module named 'girder_large_image' error. The id: ‘987’: no such user error I could reproduce on a fresh CentOS machine, and have a fix for it in PR #150 (the number is the docker group id, which as stated, isn't a user). Can you try on that branch and see if it will start for you?

One difference is that I haven't tested running as root (it usually isn't recommended).

[phillipjohnson]

Thanks for the quick reply @manthey. Your PR looks to have fixed the user 987 error, I'm just getting this output now:

Adding user `root' to group `root' ...
Adding user root to group root
Done.
Adding group `docker' (GID 987) ...
Done.
Adding user `root' to group `docker' ...
Adding user root to group docker
Done.
Traceback (most recent call last):
  File "/opt/digital_slide_archive/devops/dsa/provision.py", line 18, in <module>
    from girder_large_image.models.image_item import ImageItem
ModuleNotFoundError: No module named 'girder_large_image'
The user `root' is already a member of `root'.
The user `root' is already a member of `docker'.

What's really strange to me is that this command works:

docker run -it --rm dsarchive/dsa_common su -c "PATH=\"/opt/digital_slide_archive/devops/dsa/utils:/opt/venv/bin:/.pyenv/bin:/.pyenv/shims:$PATH\"; python /opt/digital_slide_archive/devops/dsa/provision.py"

Running in mode: development
Connecting to MongoDB: mongodb://mongodb:27017/girder?socketTimeoutMS=3600000

I agree about not using root user, I was just trying to see about getting something set up for a demo. Definitely would create a special user if we choose to stand this up in a production capacity. I can also give that a shot if you think it's causing the module not found problem.

[manthey]

I just tried running as root. The only issue I see is that I get an error on the worker since celery won't run as root with the C_FORCE_ROOT environment variable set.

Are you using a custom docker-compose.yml file? Did you build the docker image locally (if so, could you try to pull the built image)? Could you share the entire docker-compose logs girder output?

[manthey]

I'll merge PR #150, as it is a strict improvement.

[phillipjohnson]

I'm using a slightly customized docker file (mount the image storage, port number, and C_FORCE_ROOT):

docker-compose.yml

```yaml version: '3' services: girder: image: dsarchive/dsa_common build: ../.. # Instead of privileged mode, fuse can use: # devices: # - /dev/fuse:/dev/fuse # security_opt: # - apparmor:unconfined # cap_add: # - SYS_ADMIN # but these may be somewhat host specific, so we default to privileged. If # the docker daemon is being run with --no-new-privileges, fuse may not # work. privileged: true # Set DSA_USER to a user id that is part of the docker group (e.g., # `DSA_USER=$(id -u):$(id -g)`). This makes files in assetstores and logs # owned by that user and provides permissions to manage docker environment: DSA_USER: ${DSA_USER:-} C_FORCE_ROOT: 1 restart: unless-stopped # Set DSA_PORT to expose the interface on another port (default 8080). ports: - 8080:8080 volumes: # Needed to use slicer_cli_web to run docker containers - /usr/bin/docker:/usr/bin/docker - /var/run/docker.sock:/var/run/docker.sock # Default assetstore - ./assetstore:/assetstore # Location of girder.cfg - ./girder.cfg:/etc/girder.cfg # Location of provision.py - ./provision.py:/opt/digital_slide_archive/devops/dsa/provision.py # Location to store logs - ./logs:/logs # For local development, uncomment the set of mounts associated with the # local source files. Adding the editable egg directories first allows # allow mounting source files from the host without breaking the internal # data. # - /opt/girder/girder.egg-info # - /opt/girder/clients/python/girder_client.egg-info # - ../../../girder:/opt/girder # - /opt/girder_worker/girder_worker.egg-info # - ../../../../girder_worker:/opt/girder_worker # - /opt/girder_worker_utils/girder_worker_utils.egg-info # - ../../../../girder_worker_utils:/opt/girder_worker_utils # - /opt/HistomicsUI/histomicsui.egg-info # - ../../../HistomicsUI:/opt/HistomicsUI # - /opt/slicer_cli_web/girder_slicer_cli_web.egg-info # - ../../../slicer_cli_web:/opt/slicer_cli_web # - /opt/large_image/girder_annotation/girder_large_image_annotation.egg-info # - /opt/large_image/girder/girder_large_image.egg-info # - /opt/large_image/sources/bioformats/large_image_source_bioformats.egg-info # - /opt/large_image/sources/openslide/large_image_source_openslide.egg-info # - /opt/large_image/sources/ometiff/large_image_source_ometiff.egg-info # - /opt/large_image/sources/pil/large_image_source_pil.egg-info # - /opt/large_image/sources/test/large_image_source_test.egg-info # - /opt/large_image/sources/dummy/large_image_source_dummy.egg-info # - /opt/large_image/sources/tiff/large_image_source_tiff.egg-info # - /opt/large_image/sources/mapnik/large_image_source_mapnik.egg-info # - /opt/large_image/sources/openjpeg/large_image_source_openjpeg.egg-info # - /opt/large_image/sources/gdal/large_image_source_gdal.egg-info # - /opt/large_image/sources/nd2/large_image_source_nd2.egg-info # - /opt/large_image/large_image.egg-info # - /opt/large_image/utilities/converter/large_image_converter.egg-info # - /opt/large_image/utilities/tasks/large_image_tasks.egg-info # - ../../../large_image:/opt/large_image # Add additional mounts here to get access to existing files on your # system. Also add them to the worker container to reduce copying. - /opt/dsa-images:/opt/large_image depends_on: - mongodb - memcached - rabbitmq # The command does: # - Ensures that the main process runs as the DSA_USER and is part of both # that group and the docker group. This is done by: # - adding a user with the DSA_USER's id; this user is named ubuntu if it # doesn't exist. # - adds a group with the DSA_USER's group id. # - adds the user to the user group. # - adds a group with the docker group id. # - adds the user to the docker group. # - Use iptables to make some services appear as if they are on localhost # (as well as on the docker network). This is done to allow tox tests to # run. # - Run subsequent commands as the DSA_USER. This sets some paths based on # what is expected in the Docker so that the current python environment # and the devops/dsa/utils are available. # - Provision the Girder instance. This sets values in the database, such # as creating an admin user if there isn't one. See the provision.py # script for the details. # - If possible, set up a girder mount. This allows file-like access of # girder resources. It requires the host to have fuse installed and # the docker container to be run with enough permissions to use fuse. # - Start the main girder process. command: bash -c ' if [[ -z "$DSA_USER" ]]; then echo "Set the DSA_USER before starting (e.g, DSA_USER=\$$(id -u):\$$(id -g) "; exit 1; fi; adduser --uid $${DSA_USER%%:*} --disabled-password --gecos "" ubuntu 2>/dev/null; addgroup --gid $${DSA_USER#*:} $$(id -ng $${DSA_USER#*:}) 2>/dev/null; adduser $$(id -nu $${DSA_USER%%:*}) $$(getent group $${DSA_USER#*:} | cut "-d:" -f1) 2>/dev/null; addgroup --gid $$(stat -c "%g" /var/run/docker.sock) docker 2>/dev/null; adduser $$(id -nu $${DSA_USER%%:*}) $$(getent group $$(stat -c "%g" /var/run/docker.sock) | cut "-d:" -f1) 2>/dev/null; sysctl -w net.ipv4.conf.eth0.route_localnet=1; iptables -t nat -A OUTPUT -o lo -p tcp -m tcp --dport 27017 -j DNAT --to-destination `dig +short mongodb`:27017; iptables -t nat -A OUTPUT -o lo -p tcp -m tcp --dport 11211 -j DNAT --to-destination `dig +short memcached`:11211; iptables -t nat -A POSTROUTING -o eth0 -m addrtype --src-type LOCAL --dst-type UNICAST -j MASQUERADE; su $$(id -nu $${DSA_USER%%:*}) -c " PATH=\"/opt/digital_slide_archive/devops/dsa/utils:/opt/venv/bin:/.pyenv/bin:/.pyenv/shims:$PATH\"; python /opt/digital_slide_archive/devops/dsa/provision.py && (girder mount /fuse || true) && girder serve --dev "' mongodb: image: "mongo:latest" # Set DSA_USER to your user id (e.g., `DSA_USER=$(id -u):$(id -g)`) # so that database files are owned by yourself. user: ${DSA_USER:-PLEASE SET DSA_USER} restart: unless-stopped # Using --nojournal means that changes can be lost between the last # checkpoint and an unexpected shutdown, but can substantially reduce # writes. command: --nojournal volumes: # Location to store database files - ./db:/data/db # Uncomment to allow access to the database from outside of the docker # network. # ports: # - "27017" logging: options: max-size: "10M" max-file: "5" memcached: image: memcached command: -m 4096 --max-item-size 8M restart: unless-stopped # Uncomment to allow access to memcached from outside of the docker network # ports: # - "11211" logging: options: max-size: "10M" max-file: "5" rabbitmq: image: "rabbitmq:latest" restart: unless-stopped # Uncomment to allow access to rabbitmq from outside of the docker network # ports: # - "5672" logging: options: max-size: "10M" max-file: "5" worker: image: dsarchive/dsa_common build: ../.. # Set DSA_USER to a user id that is part of the docker group (e.g., # `DSA_USER=$(id -u):$(id -g)`). This provides permissions to manage # docker environment: DSA_USER: ${DSA_USER:-} DSA_WORKER_CONCURRENCY: ${DSA_WORKER_CONCURRENCY:-2} C_FORCE_ROOT: 1 TMPDIR: restart: unless-stopped volumes: # Needed to use slicer_cli_web to run docker containers - /usr/bin/docker:/usr/bin/docker - /var/run/docker.sock:/var/run/docker.sock # Needed to allow transferring data to slicer_cli_web docker containers - ${TMPDIR:-/tmp}:${TMPDIR:-/tmp} # Add additional mounts here to get access to existing files on your # system if they have the same path as on the girder container. - /opt/dsa-images:/opt/large_image depends_on: - rabbitmq # See the girder container for an explanation of most of this. # The main command is to run girder_worker command: bash -c ' if [[ -z "$DSA_USER" ]]; then echo "Set the DSA_USER before starting (e.g, DSA_USER=\$$(id -u):\$$(id -g) "; exit 1; fi; adduser --uid $${DSA_USER%%:*} --disabled-password --gecos "" ubuntu 2>/dev/null; addgroup --gid $${DSA_USER#*:} $$(id -ng $${DSA_USER#*:}) 2>/dev/null; adduser $$(id -nu $${DSA_USER%%:*}) $$(getent group $${DSA_USER#*:} | cut "-d:" -f1) 2>/dev/null; addgroup --gid $$(stat -c "%g" /var/run/docker.sock) docker 2>/dev/null; adduser $$(id -nu $${DSA_USER%%:*}) $$(getent group $$(stat -c "%g" /var/run/docker.sock) | cut "-d:" -f1) 2>/dev/null; su $$(id -nu $${DSA_USER%%:*}) -c " PATH=\"/opt/digital_slide_archive/devops/dsa/utils:/opt/venv/bin:/.pyenv/bin:/.pyenv/shims:$PATH\"; DOCKER_CLIENT_TIMEOUT=86400 TMPDIR=${TMPDIR:-/tmp} GW_DIRECT_PATHS=true python -m girder_worker --concurrency=$${DSA_WORKER_CONCURRENCY:-2} -Ofair --prefetch-multiplier=1 "' logging: options: max-size: "10M" max-file: "5" ```

docker logs output

``` # docker-compose logs girder Attaching to dsa_girder_1 girder_1 | Adding user `root' to group `root' ... girder_1 | Adding user root to group root girder_1 | Done. girder_1 | Adding group `docker' (GID 987) ... girder_1 | Done. girder_1 | Adding user `root' to group `docker' ... girder_1 | Adding user root to group docker girder_1 | Done. girder_1 | net.ipv4.conf.eth0.route_localnet = 1 girder_1 | Traceback (most recent call last): girder_1 | File "/opt/digital_slide_archive/devops/dsa/provision.py", line 18, in girder_1 | from girder_large_image.models.image_item import ImageItem girder_1 | ModuleNotFoundError: No module named 'girder_large_image' girder_1 | The user `root' is already a member of `root'. girder_1 | The user `root' is already a member of `docker'. girder_1 | net.ipv4.conf.eth0.route_localnet = 1 girder_1 | Traceback (most recent call last): girder_1 | File "/opt/digital_slide_archive/devops/dsa/provision.py", line 18, in girder_1 | from girder_large_image.models.image_item import ImageItem girder_1 | ModuleNotFoundError: No module named 'girder_large_image' girder_1 | The user `root' is already a member of `root'. girder_1 | The user `root' is already a member of `docker'. girder_1 | net.ipv4.conf.eth0.route_localnet = 1 girder_1 | Traceback (most recent call last): girder_1 | File "/opt/digital_slide_archive/devops/dsa/provision.py", line 18, in girder_1 | from girder_large_image.models.image_item import ImageItem girder_1 | ModuleNotFoundError: No module named 'girder_large_image' ... (more of the same) ``` I pulled the images instead of building them using `docker-compose pull`.

[manthey]

It's you're extra mount: - /opt/dsa-images:/opt/large_image. /opt/large_image contains python modules including the girder_large_image module. By mounting at that location, those modules are no longer available. You'll either want to use a different mount point or make sure the appropriate files also exist in the local /opt/dsa-images directory.

[phillipjohnson]

Ah, that's my fault for reading the file incorrectly, I thought you were supposed to mount to that point. Didn't realize you'd be able to define the asset stores once in the app. I've got it up and running now. Thank you for your help!