Thumbor don't find local images when using file loader and bind mount

[IlCallo]

Hello there! Thank you for creating and maintaining this docker image. I'm working on a webpack loader which uses thumbor to solve the Art Direction problem in websites development.

The loader works fine on linux environments with a local thumbor installation, but I'm trying to add Windows and Mac support and docker seems our best shot to unify the DX. Right now, we are testing the Windows integration.

Unluckily, it seems that using the file loader to work with images into a folder mounted via --mount option isn't working. Right now I'm manually testing the system before automatizing the process and all I can get are 404 errors.

This is the command I use to launch the container docker run -p 8888:80 --name ril-thumbor --env-file .thumbor-env --mount type=bind,source="$(pwd)",target=/app/loader,readonly --rm minimalcompact/thumbor

.thumbor-env

LOADER=thumbor.loaders.file_loader
RESULT_STORAGE_EXPIRATION_SECONDS=86400
STORES_CRYPTO_KEY_FOR_EACH_IMAGE=True
LOG_LEVEL=debug
DETECTORS=['thumbor.detectors.face_detector','thumbor.detectors.profile_detector','thumbor.detectors.feature_detector']

I use docker exec -it ril-thumbor bash to enter the container and check what's going on inside.

What works

• The container starts correctly and thumbor starts listening and answering the requests when I hit localhost:8888
• Thumbor configuration, as found into /app/thumbor.conf, seems correct, all options provided via env file are correctly overriden
• The current directory is correctly mounted into /app/loader (which is the default FILE_LOADER_ROOT_PATH) and I can access and read the content of the mounted folder when connected via bash
• If I use a different loader, eg the default http loader, thumbor works as expected

What doesn't work

Any request result in a 404 error, which leads me to think something is wrong with the path (the root path, the one we provide, or both). We already successfully use thumbor in this way when it's installed locally, the only difference being FILE_LOADER_ROOT_PATH set to /home/ (absolute paths to images are provided except the initial /home/ part).

What URLs I tried, by accessing them via Chrome browser:

• http://localhost:8888/unsafe/500x150/smart/src/assets/images/my-image.jpg: this should be the correct one, AFAIK
• http://localhost:8888/unsafe/500x150/smart/my-image.jpg: we moved the image into /app/loader to check if the problem was about the path slashes, still 404, seems that's not the problem
• http://localhost:8888/unsafe/500x150/smart/app/loader/src/assets/images/my-image.jpg: random guess
• http://localhost:8888/unsafe/500x150/smart/loader/src/assets/images/my-image.jpg: random guess
• http://localhost:8888/unsafe/500x150/src/assets/images/my-image.jpg: random guess
• http://localhost:8888/unsafe/500x150//src/assets/images/my-image.jpg: strange random guess

Thumbor isn't providing any useful info even with debug log level, especially I cannot find a way to make thumbor tell me where it's actually searching the image, so I'm just going on blindly.

Thumbor log with DEBUG level

2020-08-04 07:02:45 thumbor:DEBUG METRICS: inc: response.count:1
2020-08-04 07:02:45 thumbor:DEBUG METRICS: inc: storage.miss:1
2020-08-04 07:02:45 tornado.access:WARNING 404 GET /unsafe/500x150/smart/src/assets/images/my-image.jpg (172.17.0.1) 3.21ms
2020-08-04 07:02:45 thumbor:DEBUG METRICS: timing: response.time:2
2020-08-04 07:02:45 thumbor:DEBUG METRICS: timing: response.time.404:2
2020-08-04 07:02:45 thumbor:DEBUG METRICS: inc: response.status.404:1

Other possible causes:

• there is some obscure option of thumbor configuration which is set by default on the docker image which isn't set into a normal thumbor installation
• the bind mount from a Windows host cause some esoteric path conflicts which I'm not aware of (even if I thought Docker main job was to avoid these problems)

Does this ring any bell to you? Have you run into something like this in the past?

[malmi]

I used volumes instead of a mount and it worked fine for me. Maybe this would be something to try? I never worked with mounts.

[IlCallo]

Could be a good thing to try, even if docs say --mount is the preferred way and I thought type=bind was more fitting in this case, due to the transitory nature of the container

[gingerlime]

Thanks for the detailed report @IlCallo. I'm also using volumes usually rather than mount, but I'm not sure if it makes a difference to be honest.

One thing worth checking is FILE_LOADER_ROOT_PATH which you should probably set explicitly?

Here's a simple docker-compose example that works for me mapping a local ./images folder

version: '3'
services:
  thumbor:
    image: minimalcompact/thumbor
    environment:
      # VIRTUAL_HOST is picked up by nginx-proxy. Here it's set for localhost
      # but you usually need to point it to your domain, e.g. thumbor.example.com
      - VIRTUAL_HOST=localhost
      # THUMBOR_NUM_PROCESSES control how many processes run inside the container
      # Normally this is set in connection with the number of CPU cores
      # Note however that you can also use the docker-compose scale option to dynamically
      # scale your thumbor instances
      - THUMBOR_NUM_PROCESSES=4
      # this would allow CORS from any origin (you can restrict to specific origins if you want)
      - CORS_ALLOW_ORIGIN=*
      # returns a webp image if browser Accept headers match
      - AUTO_WEBP=True
      # nginx-proxy does caching automatically, so no need to store the result storage cache
      # (this greatly speeds up and saves on CPU)
      - RESULT_STORAGE=thumbor.result_storages.no_storage
      - RESULT_STORAGE_STORES_UNSAFE=True
      - LOADER=thumbor.loaders.file_loader
      - FILE_LOADER_ROOT_PATH=/var/images
    restart: always
    networks:
      - app
    volumes:
      - ./images:/var/images
    ports:
      - "8888:80"
networks:
  app:
    driver: bridge

If it still doesn't work, can you try to create a reproducible docker-compose.yml file ?

[gingerlime]

@IlCallo see edited note above

[IlCallo]

I tried to make it work with volumes, but it seems like there are a couple problems.

Volumes are not what I'm searching for, as they are persisted and slower than bindings (as stated by the docs), and their management is much more complicated than bindings, for the lightweight and transitory usage we need.

Volumes shared between windows host and linux containers doesn't seem to work well together, at least when using Hyper-V mode. I updated Windows to get WSL2 and will try again next week.

I specified all env vars, as suggested, but to no avail.

Are you aware of any way to make thumbor log the path where it's searching for the images? Unluckily I never used python and reading the codebase isn't so easy for me.

Also, I'm a newbie in Docker world, so I'm not sure what is Docker compose, even if I get the grasp of it by the configuration you provided. Will read more about it and check if I can produce a configuration like you asked @gingerlime

Could be because I set the binding on /app/loader? Will try to create it in a random root level folder to check if it's the default folder which is causing problems.

[gingerlime]

Hi @IlCallo if you'd like to submit a docker-compose.yml file that reproduces this, I'd be happy to take a look. Otherwise, the example I posted above works for me. I don't have any issues using volumes.

[gingerlime]

I'll close this for now, but please re-open if you have any further info :)

[IlCallo]

The current directory is correctly mounted into /app/loader (which is the default FILE_LOADER_ROOT_PATH)

That's the mistake. Default is /data/loader.

The right command then is docker run -p 8888:80 --name ril-thumbor --env-file .thumbor-env --mount type=bind,source="$(pwd)",target=/data/loader,readonly --rm minimalcompact/thumbor.

The wonders that taking a long break can make...

I could not find a way to tell Thumbor tell me where he was picking up the file, so I

• installed strace
• run ps aux to get python process PID
• run strace -p <PID> to attach to the process and listen to its syscall
• check output searching for open calls

Hope this can be useful for other newbies like me (or myself in some months) which will need to debug what Thumbor is actually doing.

[gingerlime]

Thanks for the details, @IlCallo and glad to hear you figured it out! 👍