search

LemmyNet / lemmy

🐀 A link aggregator and forum for the fediverse
https://join-lemmy.org
GNU Affero General Public License v3.0
13.29k stars 884 forks source link

FUSE and permission related issues with pict-rs when running in an unprivileged LXC container (not Docker) #4112

Closed gribodyr closed 1 year ago

gribodyr commented 1 year ago

Question

Hi guys,

I am running lemmy with embedded pictrs in an unprivileged LXC container. Apparently, pictrs uses fuse under the hood because I am getting the following errors when I try to upload an image. 

lemmy lemmy_server[16899]: fusermount: mount failed: Operation not permitted
lemmy lemmy_server[16896]: open dir error: No such file or directory

The pictrs repo is https://git.asonix.dog/asonix/pict-rs and the registration is closed there, so it's not possible to report anything there. image

So here are the questions :) Can someone point me in the right direction regarding this issue? Is the pictrs author active here?

Thank you!

asonix commented 1 year ago

pict-rs shouldn't use fuse unless you have configured it to point to a fuse volume. pict-rs will simply use the filesystem you give it

asonix commented 1 year ago

Also, it seems like those log lines originate from lemmy_server, not from pict-rs

Edit: ah, I see you used embedded pict-rs. Never mind then

asonix commented 1 year ago

It looks like Lemmy hardcodes some paths when using embedded pict-rs. This might be the source of your problem:

https://github.com/LemmyNet/lemmy/blob/08739e2925762eb7152f48032e2b170c061b5da0/src/main.rs#L21

gribodyr commented 1 year ago

@asonix thanks for chiming in!

I had solved the issues with the paths before I posted this. Without solving those embedded pict-rs shows different errors.

So are you suggesting the code in lemmy needs to be changed? I.e. to be able to read not the hardcoded config, but let's say some values from lemmy.hjson.

Please point me in the right direction and I will work on a PR to help those lemmy users who don't use the containerized installation like me.

asonix commented 1 year ago

If you had already fixed these paths, then I wonder if the issue is pict-rs' use of /tmp

Or maybe the "no such file or directory" is from a missing binary like ffmpeg, imagemagick, or exiftool. It's hard to tell exactly what's wrong with so few log lines

asonix commented 1 year ago

Can I also ask which version of pict-rs is being bundled?

gribodyr commented 1 year ago

@asonix

pict-rs = { version = "0.4.0-rc.12", optional = true }

Here's a debug log:

2023-10-27T13:19:03.471353Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=REDACTED http.target=/pictrs/image otel.kind="server" request_id=126310cf-bfc6-42b0-94b8-04fd15bbbbfe}:HTTP request{http.method=POST http.scheme=http http.host=localhost net.host.port=8080 otel.kind="client" otel.name=POST}: reqwest::connect: starting new connection: http://localhost:8080/
2023-10-27T13:19:03.471465Z DEBUG hyper::client::connect::dns: resolving host="localhost"
2023-10-27T13:19:03.471666Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=REDACTED http.target=/pictrs/image otel.kind="server" request_id=126310cf-bfc6-42b0-94b8-04fd15bbbbfe}:HTTP request{http.method=POST http.scheme=http http.host=localhost net.host.port=8080 otel.kind="client" otel.name=POST}: hyper::client::connect::http: connecting to [::1]:8080
2023-10-27T13:19:03.471755Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=REDACTED http.target=/pictrs/image otel.kind="server" request_id=126310cf-bfc6-42b0-94b8-04fd15bbbbfe}:HTTP request{http.method=POST http.scheme=http http.host=localhost net.host.port=8080 otel.kind="client" otel.name=POST}: hyper::client::connect::http: connecting to 127.0.0.1:8080
2023-10-27T13:19:03.471835Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=REDACTED http.target=/pictrs/image otel.kind="server" request_id=126310cf-bfc6-42b0-94b8-04fd15bbbbfe}:HTTP request{http.method=POST http.scheme=http http.host=localhost net.host.port=8080 otel.kind="client" otel.name=POST}: hyper::client::connect::http: connected to 127.0.0.1:8080
2023-10-27T13:19:03.471971Z DEBUG hyper::proto::h1::io: flushed 22108 bytes
2023-10-27T13:19:03.518011Z DEBUG HTTP request{http.method=POST http.route=/image http.flavor=1.1 http.scheme=https http.host=REDACTED http.client_ip=REDACTED http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 http.target=/image otel.name=HTTP POST /image otel.kind="server" request_id=1f21db66-e6f7-4e8d-b445-35314c076c11 trace_id=00000000000000000000000000000000}:file-upload{filename="jenk.png"}:ingest{declared_alias=None should_validate=true}:validate_bytes:details_file: pict_rs::ffmpeg: OUTPUT: 228
311
1
png_pipe
fusermount: mount failed: Operation not permitted
open dir error: No such file or directory
2023-10-27T13:19:03.521713Z DEBUG hyper::proto::h1::io: parsed 4 headers
2023-10-27T13:19:03.521736Z DEBUG hyper::proto::h1::conn: incoming body is content-length (34 bytes)
2023-10-27T13:19:03.521754Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-10-27T13:19:03.521963Z  INFO actix_web::middleware::logger: REDACTED 'POST /pictrs/image HTTP/1.1' 400 47 'https://REDACTED/post/2?scrollToComments=true' 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36' 0.050793

Not much additional info here for me :/

asonix commented 1 year ago

Interesting, so ffmpeg is failing for some reason

Can I ask what version of ffmpeg you have installed?

asonix commented 1 year ago

Actually it might not be ffmpeg. Maybe imagemagick (which is invoked after ffmpeg)

Can I ask what version of that you have installed also?

asonix commented 1 year ago

And this probably won't help, but I'd suggest upgrading to 0.4.5, the latest stable release. Although rc.12 should be mostly fine :p

gribodyr commented 1 year ago

@asonix

Okay, I updated the pict-rs version to 0.4.5, the output is slightly different:

2023-10-27T19:54:48.928426Z DEBUG hyper::proto::h1::io: flushed 22108 bytes
2023-10-27T19:54:48.928605Z DEBUG actix_form_data::upload: Finished consuming field
2023-10-27T19:54:48.971105Z DEBUG HTTP request{http.method=POST http.route=/image http.flavor=1.1 http.scheme=https http.host=REDACTED http.client_ip=REDACTED http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 http.target=/image otel.name=HTTP POST /image otel.kind="server" request_id=3ce4d423-cd0a-4e01-8cfd-913e5aa27c00 trace_id=00000000000000000000000000000000}:file-upload{filename="jenk.png"}:ingest{declared_alias=None external_validation=None external_validation_timeout=30 should_validate=true timeout=30}:validate_bytes:details_file{timeout=30}: pict_rs::ffmpeg: OUTPUT: DetailsOutput { streams: [Stream { width: 228, height: 311, nb_read_frames: Some("1") }], format: Format { format_name: "png_pipe" } }
fusermount: mount failed: Operation not permitted
open dir error: No such file or directory
2023-10-27T19:54:48.974391Z DEBUG HTTP request{http.method=POST http.route=/image http.flavor=1.1 http.scheme=https http.host=REDACTED http.client_ip=REDACTED http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 http.target=/image otel.name=HTTP POST /image otel.kind="server" request_id=3ce4d423-cd0a-4e01-8cfd-913e5aa27c00 trace_id=00000000000000000000000000000000}: actix_form_data::upload: Finished consuming multipart
2023-10-27T19:54:48.974512Z DEBUG hyper::proto::h1::io: parsed 4 headers
2023-10-27T19:54:48.974523Z DEBUG hyper::proto::h1::conn: incoming body is content-length (45 bytes)
2023-10-27T19:54:48.974537Z DEBUG hyper::proto::h1::conn: incoming body completed

ImageMagick:

lemmy@lemmy:~/lemmy$ /usr/bin/magick --version
Version: ImageMagick 7.1.1-19 Q16-HDRI x86_64 ff509f4c0:20230930 https://imagemagick.org
Copyright: (C) 1999 ImageMagick Studio LLC
License: https://imagemagick.org/script/license.php
Features: Cipher DPC HDRI OpenMP(4.5) 
Delegates (built-in): bzlib djvu fontconfig freetype heic jbig jng jp2 jpeg lcms lqr lzma openexr png raqm tiff webp x xml zlib
Compiler: gcc (7.5)

ffmpeg:

lemmy@lemmy:~/lemmy$ ffmpeg
ffmpeg version 4.2.7-0ubuntu0.1 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample 
--disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig 
--enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg 
--enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh 
--enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack 
--enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 
--enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
asonix commented 1 year ago

I'm still confused about the fusermount failure. I don't know how lxc works at all. How do you have your mounts set up?

gribodyr commented 1 year ago

Well, I must confess this area is sorta confusing to me, but ...

  1. I had figured embedded pict-rs was looking into the working directory 

    [Service]
User=lemmy
ExecStart=/opt/lemmy/server/lemmy_server
Environment=LEMMY_CONFIG_LOCATION=/opt/lemmy/server/lemmy.hjson
# remove these two lines if you don't need pict-rs
Environment=PICTRS_PATH=/opt/lemmy/pictrs
Environment=PICTRS_ADDR=127.0.0.1:8080
Environment=RUST_LOG="debug,lemmy_server=debug,lemmy_api=debug,lemmy_api_common=debug,lemmy_api_crud=debug,lemmy_apub=debug,lemmy_db_schema=debug,lemmy_db_views=debug,lemmy_db_views_actor=debug,lemmy_db_views_moderator=debug,lemmy_routes=debug,lemmy_utils=debug,lemmy_websocket=debug"
Restart=on-failure
WorkingDirectory=/opt/lemmy

  2. My deployment is in /opt/lemmy; so I just chmod 777 -R the pict-rs folder for testing

    lemmy@lemmy:/opt/lemmy$ ls -la
total 20
drwxr-xr-x 5 lemmy lemmy 4096 Oct 24 02:41 .
drwxr-xr-x 3 root  root  4096 Oct 24 02:04 ..
drwxrwxr-x 9 lemmy lemmy 4096 Oct 24 02:42 lemmy-ui
drwxrwxrwx 5 lemmy lemmy 4096 Oct 24 02:09 pictrs
drwxr-xr-x 2 lemmy lemmy 4096 Oct 24 03:18 server
lemmy@lemmy:/opt/lemmy$ cd pictrs
lemmy@lemmy:/opt/lemmy/pictrs$ ls -la
total 20
drwxrwxrwx 5 lemmy lemmy 4096 Oct 24 02:09 .
drwxr-xr-x 5 lemmy lemmy 4096 Oct 24 02:41 ..
drwxrwxrwx 2 lemmy lemmy 4096 Oct 24 02:09 files
drwxrwxrwx 2 lemmy lemmy 4096 Oct 24 02:09 old
drwxrwxrwx 3 lemmy lemmy 4096 Oct 24 02:23 sled-repo

Is there anything else I could share with you to help troubleshoot this?

asonix commented 1 year ago

Could I convince you to run standalone pict-rs instead of embedded pict-rs

gribodyr commented 1 year ago

@asonix man oh man.

standalone works like a breeze when run from the terminal. thank you!

as for embedded, I wonder if anyone is actually using it.

gribodyr commented 1 year ago

Permission issues start to creep in when I launch it as a daemon. Basically, for now the only way that works for me is to run it as root.

asonix commented 1 year ago

I wonder if the issue is access to /tmp when run as a daemon. When I containerize it with docker, it runs as 991:991. Root shouldn't be required

gribodyr commented 1 year ago

It doesn't look like a /tmp issue. I think the permissions are correct. 

root@lemmy:/etc/systemd/system# ls -la /tmp
total 52
drwxrwxrwt 13 root  root  4096 Oct 28 19:26 .
gribodyr commented 1 year ago

Here's the log when I run the daemon as user "lemmy" (same user used to build everything)

Oct 28 19:39:57 lemmy pict-rs[10720]: 2023-10-28T19:39:57.333231Z  INFO HTTP request{http.method=POST http.route=/image http.flavor=1.1 http.scheme=https http.host=REDACTED http.client_ip=REDACTED http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 http.target=/image otel.name=HTTP POST /image otel.kind="server" request_id=a3a3f81c-e06f-41ab-bf27-f0bbcf854df5}: tracing_actix_web::root_span_builder: new
Oct 28 19:39:57 lemmy pict-rs[10759]: fusermount: mount failed: Operation not permitted
Oct 28 19:39:57 lemmy pict-rs[10756]: open dir error: No such file or directory
Oct 28 19:39:57 lemmy pict-rs[10720]: 2023-10-28T19:39:57.385998Z  INFO HTTP request{http.method=POST http.route=/image http.flavor=1.1 http.scheme=https http.host=REDACTED http.client_ip=REDACTED http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 http.target=/image otel.name=HTTP POST /image otel.kind="server" request_id=a3a3f81c-e06f-41ab-bf27-f0bbcf854df5 trace_id=00000000000000000000000000000000 exception.message=Error in imagemagick exception.details=
Oct 28 19:39:57 lemmy pict-rs[10720]:    0: Error in imagemagick
Oct 28 19:39:57 lemmy pict-rs[10720]:    1: Error reading bytes
Oct 28 19:39:57 lemmy pict-rs[10720]:    2: Failed with status exit status: 127
Oct 28 19:39:57 lemmy pict-rs[10720]: Location:
Oct 28 19:39:57 lemmy pict-rs[10720]:    /home/lemmy/pict-rs/src/error.rs:42
Oct 28 19:39:57 lemmy pict-rs[10720]:   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Oct 28 19:39:57 lemmy pict-rs[10720]:    0: pict_rs::validate::validate_bytes
Oct 28 19:39:57 lemmy pict-rs[10720]:       at src/validate.rs:38
Oct 28 19:39:57 lemmy pict-rs[10720]:    1: pict_rs::ingest::ingest with declared_alias=None external_validation=None external_validation_timeout=30 should_validate=true timeout=30
Oct 28 19:39:57 lemmy pict-rs[10720]:       at src/ingest.rs:50
Oct 28 19:39:57 lemmy pict-rs[10720]:    2: pict_rs::file-upload with filename="jenk.png"
Oct 28 19:39:57 lemmy pict-rs[10720]:       at src/lib.rs:173
Oct 28 19:39:57 lemmy pict-rs[10720]:    3: tokio::task::runtime.spawn with kind=local task.name= task.id=66 loc.file="/home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-form-data-0.7.0-beta.4/src/upload.rs" loc.line=342 loc.col=33
Oct 28 19:39:57 lemmy pict-rs[10720]:       at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/util/trace.rs:17
Oct 28 19:39:57 lemmy pict-rs[10720]:    4: tracing_actix_web::root_span_builder::HTTP request with http.method=POST http.route=/image http.flavor=1.1 http.scheme=https http.host=REDACTED http.client_ip=REDACTED http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 http.target=/image otel.name=HTTP POST /image otel.kind="server" request_id=a3a3f81c-e06f-41ab-bf27-f0bbcf854df5 trace_id=00000000000000000000000000000000
Oct 28 19:39:57 lemmy pict-rs[10720]:       at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-actix-web-0.7.7/src/root_span_builder.rs:41
Oct 28 19:39:57 lemmy pict-rs[10720]:    5: tokio::task::runtime.spawn with kind=local task.name= task.id=65 loc.file="/home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/service.rs" loc.line=74 loc.col=17
Oct 28 19:39:57 lemmy pict-rs[10720]:       at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/util/trace.rs:17
Oct 28 19:39:57 lemmy pict-rs[10720]:    6: tokio::task::runtime.spawn with kind=local task.name= task.id=17 loc.file="/home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs" loc.line=461 loc.col=25
Oct 28 19:39:57 lemmy pict-rs[10720]:       at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/util/trace.rs:17
Oct 28 19:39:57 lemmy pict-rs[10720]:    7: tokio::task::runtime.spawn with kind=local task.name= task.id=13 loc.file="/home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs" loc.line=435 loc.col=21
Oct 28 19:39:57 lemmy pict-rs[10720]:       at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/util/trace.rs:17
Oct 28 19:39:57 lemmy pict-rs[10720]:    8: tokio::task::runtime.spawn with kind=local task.name= task.id=12 loc.file="/home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-rt-2.9.0/src/arbiter.rs" loc.line=311 loc.col=25
Oct 28 19:39:57 lemmy pict-rs[10720]:       at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/util/trace.rs:17
Oct 28 19:39:57 lemmy pict-rs[10720]:    9: tokio::task::runtime.spawn with kind=block_on task.name= task.id=11 loc.file="/home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-rt-2.9.0/src/arbiter.rs" loc.line=143 loc.col=24
Oct 28 19:39:57 lemmy pict-rs[10720]:       at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/util/trace.rs:17
Oct 28 19:39:57 lemmy pict-rs[10720]: Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Oct 28 19:39:57 lemmy pict-rs[10720]: Run with RUST_BACKTRACE=full to include source snippets. http.status_code=500 otel.status_code="ERROR"}: tracing_actix_web::root_span_builder: close time.busy=167µs time.idle=52.6ms
asonix commented 1 year ago

Can I ask how you installed imagemagick? Are you by chance using the appimage?

gribodyr commented 1 year ago

I think so, yes. It was like this:


wget https://imagemagick.org/archive/binaries/magick
sudo cp magick /usr/bin/magick
asonix commented 1 year ago

Appimage requires fuse, and fuse requires you add your user to the fuse group: https://github.com/AppImage/AppImageKit/wiki/FUSE

gribodyr commented 1 year ago

@asonix we are slowly getting there with your help. I really appreciate it!

I reinstalled imagemagick using the imei.sh install method so that it doesn't use AppImage (in my case using fuse the way it is described is not an option). And now I have new entries in the log! :/

Oct 29 14:36:16 lemmy pict-rs[29536]:    0: Error in store
Oct 29 14:36:16 lemmy pict-rs[29536]:    1: Requested file is not found
Oct 29 14:36:16 lemmy pict-rs[29536]:    2: No such file or directory (os error 2)
Oct 29 14:36:16 lemmy pict-rs[29536]: Location:
Oct 29 14:36:16 lemmy pict-rs[29536]:    /home/lemmy/pict-rs/src/error.rs:42

Running as root does not show this issue.

asonix commented 1 year ago

The files are likely saved on your drive as root:root if it works when running as root

Make sure that pict-rs can read the files directory and its contents

gribodyr commented 1 year ago

@asonix THANK YOU! Now this is working. Do you think there's any sense in summarizing all this and making some sort of a guide?

asonix commented 1 year ago

That's probably a good idea. I think with what we've learned the embedded pict-rs should also work now :)

gribodyr commented 1 year ago

Created a pull request in lemmy-docs: https://github.com/LemmyNet/lemmy-docs/pull/279 Closing this ticket.