FUSE and permission related issues with pict-rs when running in an unprivileged LXC container (not Docker)

[gribodyr]

Question

Hi guys,

I am running lemmy with embedded pictrs in an unprivileged LXC container. Apparently, pictrs uses fuse under the hood because I am getting the following errors when I try to upload an image.

lemmy lemmy_server[16899]: fusermount: mount failed: Operation not permitted
lemmy lemmy_server[16896]: open dir error: No such file or directory

The pictrs repo is https://git.asonix.dog/asonix/pict-rs and the registration is closed there, so it's not possible to report anything there.

So here are the questions :) Can someone point me in the right direction regarding this issue? Is the pictrs author active here?

Thank you!

[asonix]

pict-rs shouldn't use fuse unless you have configured it to point to a fuse volume. pict-rs will simply use the filesystem you give it

[asonix]

Also, it seems like those log lines originate from lemmy_server, not from pict-rs

Edit: ah, I see you used embedded pict-rs. Never mind then

[asonix]

It looks like Lemmy hardcodes some paths when using embedded pict-rs. This might be the source of your problem:

https://github.com/LemmyNet/lemmy/blob/08739e2925762eb7152f48032e2b170c061b5da0/src/main.rs#L21

[gribodyr]

@asonix thanks for chiming in!

I had solved the issues with the paths before I posted this. Without solving those embedded pict-rs shows different errors.

So are you suggesting the code in lemmy needs to be changed? I.e. to be able to read not the hardcoded config, but let's say some values from lemmy.hjson.

Please point me in the right direction and I will work on a PR to help those lemmy users who don't use the containerized installation like me.

[asonix]

If you had already fixed these paths, then I wonder if the issue is pict-rs' use of /tmp

Or maybe the "no such file or directory" is from a missing binary like ffmpeg, imagemagick, or exiftool. It's hard to tell exactly what's wrong with so few log lines

[asonix]

Can I also ask which version of pict-rs is being bundled?

[gribodyr]

@asonix

pict-rs = { version = "0.4.0-rc.12", optional = true }

Here's a debug log:

2023-10-27T13:19:03.471353Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=REDACTED http.target=/pictrs/image otel.kind="server" request_id=126310cf-bfc6-42b0-94b8-04fd15bbbbfe}:HTTP request{http.method=POST http.scheme=http http.host=localhost net.host.port=8080 otel.kind="client" otel.name=POST}: reqwest::connect: starting new connection: http://localhost:8080/
2023-10-27T13:19:03.471465Z DEBUG hyper::client::connect::dns: resolving host="localhost"
2023-10-27T13:19:03.471666Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=REDACTED http.target=/pictrs/image otel.kind="server" request_id=126310cf-bfc6-42b0-94b8-04fd15bbbbfe}:HTTP request{http.method=POST http.scheme=http http.host=localhost net.host.port=8080 otel.kind="client" otel.name=POST}: hyper::client::connect::http: connecting to [::1]:8080
2023-10-27T13:19:03.471755Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=REDACTED http.target=/pictrs/image otel.kind="server" request_id=126310cf-bfc6-42b0-94b8-04fd15bbbbfe}:HTTP request{http.method=POST http.scheme=http http.host=localhost net.host.port=8080 otel.kind="client" otel.name=POST}: hyper::client::connect::http: connecting to 127.0.0.1:8080
2023-10-27T13:19:03.471835Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=REDACTED http.target=/pictrs/image otel.kind="server" request_id=126310cf-bfc6-42b0-94b8-04fd15bbbbfe}:HTTP request{http.method=POST http.scheme=http http.host=localhost net.host.port=8080 otel.kind="client" otel.name=POST}: hyper::client::connect::http: connected to 127.0.0.1:8080
2023-10-27T13:19:03.471971Z DEBUG hyper::proto::h1::io: flushed 22108 bytes
2023-10-27T13:19:03.518011Z DEBUG HTTP request{http.method=POST http.route=/image http.flavor=1.1 http.scheme=https http.host=REDACTED http.client_ip=REDACTED http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 http.target=/image otel.name=HTTP POST /image otel.kind="server" request_id=1f21db66-e6f7-4e8d-b445-35314c076c11 trace_id=00000000000000000000000000000000}:file-upload{filename="jenk.png"}:ingest{declared_alias=None should_validate=true}:validate_bytes:details_file: pict_rs::ffmpeg: OUTPUT: 228
311
1
png_pipe
fusermount: mount failed: Operation not permitted
open dir error: No such file or directory
2023-10-27T13:19:03.521713Z DEBUG hyper::proto::h1::io: parsed 4 headers
2023-10-27T13:19:03.521736Z DEBUG hyper::proto::h1::conn: incoming body is content-length (34 bytes)
2023-10-27T13:19:03.521754Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-10-27T13:19:03.521963Z  INFO actix_web::middleware::logger: REDACTED 'POST /pictrs/image HTTP/1.1' 400 47 'https://REDACTED/post/2?scrollToComments=true' 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36' 0.050793

Not much additional info here for me :/

[asonix]

Interesting, so ffmpeg is failing for some reason

Can I ask what version of ffmpeg you have installed?

[asonix]

Actually it might not be ffmpeg. Maybe imagemagick (which is invoked after ffmpeg)

Can I ask what version of that you have installed also?

[asonix]

And this probably won't help, but I'd suggest upgrading to 0.4.5, the latest stable release. Although rc.12 should be mostly fine :p

[gribodyr]

@asonix

Okay, I updated the pict-rs version to 0.4.5, the output is slightly different:

2023-10-27T19:54:48.928426Z DEBUG hyper::proto::h1::io: flushed 22108 bytes
2023-10-27T19:54:48.928605Z DEBUG actix_form_data::upload: Finished consuming field
2023-10-27T19:54:48.971105Z DEBUG HTTP request{http.method=POST http.route=/image http.flavor=1.1 http.scheme=https http.host=REDACTED http.client_ip=REDACTED http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 http.target=/image otel.name=HTTP POST /image otel.kind="server" request_id=3ce4d423-cd0a-4e01-8cfd-913e5aa27c00 trace_id=00000000000000000000000000000000}:file-upload{filename="jenk.png"}:ingest{declared_alias=None external_validation=None external_validation_timeout=30 should_validate=true timeout=30}:validate_bytes:details_file{timeout=30}: pict_rs::ffmpeg: OUTPUT: DetailsOutput { streams: [Stream { width: 228, height: 311, nb_read_frames: Some("1") }], format: Format { format_name: "png_pipe" } }
fusermount: mount failed: Operation not permitted
open dir error: No such file or directory
2023-10-27T19:54:48.974391Z DEBUG HTTP request{http.method=POST http.route=/image http.flavor=1.1 http.scheme=https http.host=REDACTED http.client_ip=REDACTED http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 http.target=/image otel.name=HTTP POST /image otel.kind="server" request_id=3ce4d423-cd0a-4e01-8cfd-913e5aa27c00 trace_id=00000000000000000000000000000000}: actix_form_data::upload: Finished consuming multipart
2023-10-27T19:54:48.974512Z DEBUG hyper::proto::h1::io: parsed 4 headers
2023-10-27T19:54:48.974523Z DEBUG hyper::proto::h1::conn: incoming body is content-length (45 bytes)
2023-10-27T19:54:48.974537Z DEBUG hyper::proto::h1::conn: incoming body completed

ImageMagick:

lemmy@lemmy:~/lemmy$ /usr/bin/magick --version
Version: ImageMagick 7.1.1-19 Q16-HDRI x86_64 ff509f4c0:20230930 https://imagemagick.org
Copyright: (C) 1999 ImageMagick Studio LLC
License: https://imagemagick.org/script/license.php
Features: Cipher DPC HDRI OpenMP(4.5) 
Delegates (built-in): bzlib djvu fontconfig freetype heic jbig jng jp2 jpeg lcms lqr lzma openexr png raqm tiff webp x xml zlib
Compiler: gcc (7.5)

ffmpeg:

lemmy@lemmy:~/lemmy$ ffmpeg
ffmpeg version 4.2.7-0ubuntu0.1 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample 
--disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig 
--enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg 
--enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh 
--enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack 
--enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 
--enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100

[asonix]

I'm still confused about the fusermount failure. I don't know how lxc works at all. How do you have your mounts set up?

[gribodyr]

Well, I must confess this area is sorta confusing to me, but ...

1. I had figured embedded pict-rs was looking into the working directory

   [Service]
   User=lemmy
   ExecStart=/opt/lemmy/server/lemmy_server
   Environment=LEMMY_CONFIG_LOCATION=/opt/lemmy/server/lemmy.hjson
   # remove these two lines if you don't need pict-rs
   Environment=PICTRS_PATH=/opt/lemmy/pictrs
   Environment=PICTRS_ADDR=127.0.0.1:8080
   Environment=RUST_LOG="debug,lemmy_server=debug,lemmy_api=debug,lemmy_api_common=debug,lemmy_api_crud=debug,lemmy_apub=debug,lemmy_db_schema=debug,lemmy_db_views=debug,lemmy_db_views_actor=debug,lemmy_db_views_moderator=debug,lemmy_routes=debug,lemmy_utils=debug,lemmy_websocket=debug"
   Restart=on-failure
   WorkingDirectory=/opt/lemmy

2. My deployment is in /opt/lemmy; so I just chmod 777 -R the pict-rs folder for testing

   lemmy@lemmy:/opt/lemmy$ ls -la
   total 20
   drwxr-xr-x 5 lemmy lemmy 4096 Oct 24 02:41 .
   drwxr-xr-x 3 root  root  4096 Oct 24 02:04 ..
   drwxrwxr-x 9 lemmy lemmy 4096 Oct 24 02:42 lemmy-ui
   drwxrwxrwx 5 lemmy lemmy 4096 Oct 24 02:09 pictrs
   drwxr-xr-x 2 lemmy lemmy 4096 Oct 24 03:18 server
   lemmy@lemmy:/opt/lemmy$ cd pictrs
   lemmy@lemmy:/opt/lemmy/pictrs$ ls -la
   total 20
   drwxrwxrwx 5 lemmy lemmy 4096 Oct 24 02:09 .
   drwxr-xr-x 5 lemmy lemmy 4096 Oct 24 02:41 ..
   drwxrwxrwx 2 lemmy lemmy 4096 Oct 24 02:09 files
   drwxrwxrwx 2 lemmy lemmy 4096 Oct 24 02:09 old
   drwxrwxrwx 3 lemmy lemmy 4096 Oct 24 02:23 sled-repo

Is there anything else I could share with you to help troubleshoot this?

[asonix]

Could I convince you to run standalone pict-rs instead of embedded pict-rs

[gribodyr]

@asonix man oh man.

standalone works like a breeze when run from the terminal. thank you!

as for embedded, I wonder if anyone is actually using it.

[gribodyr]

Permission issues start to creep in when I launch it as a daemon. Basically, for now the only way that works for me is to run it as root.

[asonix]

I wonder if the issue is access to /tmp when run as a daemon. When I containerize it with docker, it runs as 991:991. Root shouldn't be required

[gribodyr]

It doesn't look like a /tmp issue. I think the permissions are correct.

root@lemmy:/etc/systemd/system# ls -la /tmp
total 52
drwxrwxrwt 13 root  root  4096 Oct 28 19:26 .

[gribodyr]

Here's the log when I run the daemon as user "lemmy" (same user used to build everything)

Oct 28 19:39:57 lemmy pict-rs[10720]: 2023-10-28T19:39:57.333231Z  INFO HTTP request{http.method=POST http.route=/image http.flavor=1.1 http.scheme=https http.host=REDACTED http.client_ip=REDACTED http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 http.target=/image otel.name=HTTP POST /image otel.kind="server" request_id=a3a3f81c-e06f-41ab-bf27-f0bbcf854df5}: tracing_actix_web::root_span_builder: new
Oct 28 19:39:57 lemmy pict-rs[10759]: fusermount: mount failed: Operation not permitted
Oct 28 19:39:57 lemmy pict-rs[10756]: open dir error: No such file or directory
Oct 28 19:39:57 lemmy pict-rs[10720]: 2023-10-28T19:39:57.385998Z  INFO HTTP request{http.method=POST http.route=/image http.flavor=1.1 http.scheme=https http.host=REDACTED http.client_ip=REDACTED http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 http.target=/image otel.name=HTTP POST /image otel.kind="server" request_id=a3a3f81c-e06f-41ab-bf27-f0bbcf854df5 trace_id=00000000000000000000000000000000 exception.message=Error in imagemagick exception.details=
Oct 28 19:39:57 lemmy pict-rs[10720]:    0: Error in imagemagick
Oct 28 19:39:57 lemmy pict-rs[10720]:    1: Error reading bytes
Oct 28 19:39:57 lemmy pict-rs[10720]:    2: Failed with status exit status: 127
Oct 28 19:39:57 lemmy pict-rs[10720]: Location:
Oct 28 19:39:57 lemmy pict-rs[10720]:    /home/lemmy/pict-rs/src/error.rs:42
Oct 28 19:39:57 lemmy pict-rs[10720]:   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Oct 28 19:39:57 lemmy pict-rs[10720]:    0: pict_rs::validate::validate_bytes
Oct 28 19:39:57 lemmy pict-rs[10720]:       at src/validate.rs:38
Oct 28 19:39:57 lemmy pict-rs[10720]:    1: pict_rs::ingest::ingest with declared_alias=None external_validation=None external_validation_timeout=30 should_validate=true timeout=30
Oct 28 19:39:57 lemmy pict-rs[10720]:       at src/ingest.rs:50
Oct 28 19:39:57 lemmy pict-rs[10720]:    2: pict_rs::file-upload with filename="jenk.png"
Oct 28 19:39:57 lemmy pict-rs[10720]:       at src/lib.rs:173
Oct 28 19:39:57 lemmy pict-rs[10720]:    3: tokio::task::runtime.spawn with kind=local task.name= task.id=66 loc.file="/home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-form-data-0.7.0-beta.4/src/upload.rs" loc.line=342 loc.col=33
Oct 28 19:39:57 lemmy pict-rs[10720]:       at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/util/trace.rs:17
Oct 28 19:39:57 lemmy pict-rs[10720]:    4: tracing_actix_web::root_span_builder::HTTP request with http.method=POST http.route=/image http.flavor=1.1 http.scheme=https http.host=REDACTED http.client_ip=REDACTED http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 http.target=/image otel.name=HTTP POST /image otel.kind="server" request_id=a3a3f81c-e06f-41ab-bf27-f0bbcf854df5 trace_id=00000000000000000000000000000000
Oct 28 19:39:57 lemmy pict-rs[10720]:       at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-actix-web-0.7.7/src/root_span_builder.rs:41
Oct 28 19:39:57 lemmy pict-rs[10720]:    5: tokio::task::runtime.spawn with kind=local task.name= task.id=65 loc.file="/home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/service.rs" loc.line=74 loc.col=17
Oct 28 19:39:57 lemmy pict-rs[10720]:       at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/util/trace.rs:17
Oct 28 19:39:57 lemmy pict-rs[10720]:    6: tokio::task::runtime.spawn with kind=local task.name= task.id=17 loc.file="/home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs" loc.line=461 loc.col=25
Oct 28 19:39:57 lemmy pict-rs[10720]:       at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/util/trace.rs:17
Oct 28 19:39:57 lemmy pict-rs[10720]:    7: tokio::task::runtime.spawn with kind=local task.name= task.id=13 loc.file="/home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs" loc.line=435 loc.col=21
Oct 28 19:39:57 lemmy pict-rs[10720]:       at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/util/trace.rs:17
Oct 28 19:39:57 lemmy pict-rs[10720]:    8: tokio::task::runtime.spawn with kind=local task.name= task.id=12 loc.file="/home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-rt-2.9.0/src/arbiter.rs" loc.line=311 loc.col=25
Oct 28 19:39:57 lemmy pict-rs[10720]:       at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/util/trace.rs:17
Oct 28 19:39:57 lemmy pict-rs[10720]:    9: tokio::task::runtime.spawn with kind=block_on task.name= task.id=11 loc.file="/home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-rt-2.9.0/src/arbiter.rs" loc.line=143 loc.col=24
Oct 28 19:39:57 lemmy pict-rs[10720]:       at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/util/trace.rs:17
Oct 28 19:39:57 lemmy pict-rs[10720]: Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Oct 28 19:39:57 lemmy pict-rs[10720]: Run with RUST_BACKTRACE=full to include source snippets. http.status_code=500 otel.status_code="ERROR"}: tracing_actix_web::root_span_builder: close time.busy=167µs time.idle=52.6ms

[asonix]

Can I ask how you installed imagemagick? Are you by chance using the appimage?

[gribodyr]

I think so, yes. It was like this:

wget https://imagemagick.org/archive/binaries/magick
sudo cp magick /usr/bin/magick

[asonix]

Appimage requires fuse, and fuse requires you add your user to the fuse group: https://github.com/AppImage/AppImageKit/wiki/FUSE

[gribodyr]

@asonix we are slowly getting there with your help. I really appreciate it!

I reinstalled imagemagick using the imei.sh install method so that it doesn't use AppImage (in my case using fuse the way it is described is not an option). And now I have new entries in the log! :/

Oct 29 14:36:16 lemmy pict-rs[29536]:    0: Error in store
Oct 29 14:36:16 lemmy pict-rs[29536]:    1: Requested file is not found
Oct 29 14:36:16 lemmy pict-rs[29536]:    2: No such file or directory (os error 2)
Oct 29 14:36:16 lemmy pict-rs[29536]: Location:
Oct 29 14:36:16 lemmy pict-rs[29536]:    /home/lemmy/pict-rs/src/error.rs:42

Running as root does not show this issue.

[asonix]

The files are likely saved on your drive as root:root if it works when running as root

Make sure that pict-rs can read the files directory and its contents

[gribodyr]

@asonix THANK YOU! Now this is working. Do you think there's any sense in summarizing all this and making some sort of a guide?

[asonix]

That's probably a good idea. I think with what we've learned the embedded pict-rs should also work now :)

[gribodyr]

Created a pull request in lemmy-docs: https://github.com/LemmyNet/lemmy-docs/pull/279 Closing this ticket.