search

YunoHost-Apps / lemmy_ynh

A link aggregator for the fediverse.
https://join.lemmy.ml
GNU General Public License v3.0
20 stars 12 forks source link

Reimplement Pict-RS #43

Closed tituspijean closed 9 months ago

tituspijean commented 1 year ago

Closes #31

Reimplements Pict-RS, while avoiding build time by extracting builds from Docker images, as for Lemmy and Lemmy UI themselves.

PR Status

Automatic tests

Automatic tests can be triggered on https://ci-apps-dev.yunohost.org/ after creating the PR, by commenting "!testme", "!gogogadgetoci" or "By the power of systemd, I invoke The Great App CI to test this Pull Request!". (N.B. : for this to work you need to be a member of the Yunohost-Apps organization)

tituspijean commented 1 year ago

Status: file upload fails with error 500 and the following error:

(stacktrace enabled by adding Environment=RUST_BACKTRACE=full in /etc/systemd/system/lemmy-pict-rs.service)

# cat /var/log/lemmy/lemmy-pict-rs.log
2023-06-19T22:16:50.414812Z  INFO actix_server::builder: starting 4 workers
2023-06-19T22:16:50.414842Z  INFO actix_server::server: Actix runtime found; starting in Actix runtime
2023-06-19T22:16:53.532435Z  INFO HTTP request{http.method=POST http.route=/image http.flavor=1.1 http.scheme=http http.host=127.0.0.1:8081 http.client_ip=172.19.112.1 http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0 http.target=/image otel.name=HTTP POST /image otel.kind="server" request_id=169fc643-1d45-42ba-b5ac-bdf9850db875}: tracing_actix_web::root_span_builder: new
2023-06-19T22:16:53.630965Z  INFO HTTP request{http.method=POST http.route=/image http.flavor=1.1 http.scheme=http http.host=127.0.0.1:8081 http.client_ip=172.19.112.1 http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0 http.target=/image otel.name=HTTP POST /image otel.kind="server" request_id=169fc643-1d45-42ba-b5ac-bdf9850db875 trace_id=00000000000000000000000000000000 exception.message=Error interacting with filesystem exception.details=
   0: Error interacting with filesystem
   1: No such file or directory (os error 2)

Location:
   /drone/src/src/error.rs:38

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

   0: pict_rs::magick::details_bytes with hint=None
      at src/magick.rs:135
   1: pict_rs::validate::validate_bytes
      at src/validate.rs:38
   2: pict_rs::ingest::ingest with declared_alias=None should_validate=true
      at src/ingest.rs:44
   3: pict_rs::file-upload with filename="ynh_logo_white_300dpi.png"
      at src/lib.rs:154
   4: tracing_actix_web::root_span_builder::HTTP request with http.method=POST http.route=/image http.flavor=1.1 http.scheme=http http.host=127.0.0.1:8081 http.client_ip=172.19.112.1 http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0 http.target=/image otel.name=HTTP POST /image otel.kind="server" request_id=169fc643-1d45-42ba-b5ac-bdf9850db875 trace_id=00000000000000000000000000000000
      at /opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-actix-web-0.7.5/src/root_span_builder.rs:41
   5: tokio::task::runtime.spawn with kind=local task.name= task.id=46 loc.file="/opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.2.0/src/service.rs" loc.line=74 loc.col=17
      at /opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/util/trace.rs:16
   6: tokio::task::runtime.spawn with kind=local task.name= task.id=16 loc.file="/opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.2.0/src/worker.rs" loc.line=457 loc.col=25
      at /opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/util/trace.rs:16
   7: tokio::task::runtime.spawn with kind=local task.name= task.id=12 loc.file="/opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.2.0/src/worker.rs" loc.line=431 loc.col=21
      at /opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/util/trace.rs:16
   8: tokio::task::runtime.spawn with kind=local task.name= task.id=11 loc.file="/opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-rt-2.8.0/src/arbiter.rs" loc.line=316 loc.col=25
      at /opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/util/trace.rs:16
   9: tokio::task::runtime.spawn with kind=block_on task.name= task.id=10 loc.file="/opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-rt-2.8.0/src/arbiter.rs" loc.line=144 loc.col=24
      at /opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/util/trace.rs:16

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  <empty backtrace>

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering. http.status_code=500 otel.status_code="ERROR"}: tracing_actix_web::root_span_builder: close time.busy=3.23ms time.idle=95.3ms
DrMxrcy commented 1 year ago

Status: file upload fails with error 500 and the following error:

(stacktrace enabled by adding Environment=RUST_BACKTRACE=full in /etc/systemd/system/lemmy-pict-rs.service)

# cat /var/log/lemmy/lemmy-pict-rs.log
2023-06-19T22:16:50.414812Z  INFO actix_server::builder: starting 4 workers
2023-06-19T22:16:50.414842Z  INFO actix_server::server: Actix runtime found; starting in Actix runtime
2023-06-19T22:16:53.532435Z  INFO HTTP request{http.method=POST http.route=/image http.flavor=1.1 http.scheme=http http.host=127.0.0.1:8081 http.client_ip=172.19.112.1 http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0 http.target=/image otel.name=HTTP POST /image otel.kind="server" request_id=169fc643-1d45-42ba-b5ac-bdf9850db875}: tracing_actix_web::root_span_builder: new
2023-06-19T22:16:53.630965Z  INFO HTTP request{http.method=POST http.route=/image http.flavor=1.1 http.scheme=http http.host=127.0.0.1:8081 http.client_ip=172.19.112.1 http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0 http.target=/image otel.name=HTTP POST /image otel.kind="server" request_id=169fc643-1d45-42ba-b5ac-bdf9850db875 trace_id=00000000000000000000000000000000 exception.message=Error interacting with filesystem exception.details=
   0: Error interacting with filesystem
   1: No such file or directory (os error 2)

Location:
   /drone/src/src/error.rs:38

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

   0: pict_rs::magick::details_bytes with hint=None
      at src/magick.rs:135
   1: pict_rs::validate::validate_bytes
      at src/validate.rs:38
   2: pict_rs::ingest::ingest with declared_alias=None should_validate=true
      at src/ingest.rs:44
   3: pict_rs::file-upload with filename="ynh_logo_white_300dpi.png"
      at src/lib.rs:154
   4: tracing_actix_web::root_span_builder::HTTP request with http.method=POST http.route=/image http.flavor=1.1 http.scheme=http http.host=127.0.0.1:8081 http.client_ip=172.19.112.1 http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0 http.target=/image otel.name=HTTP POST /image otel.kind="server" request_id=169fc643-1d45-42ba-b5ac-bdf9850db875 trace_id=00000000000000000000000000000000
      at /opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-actix-web-0.7.5/src/root_span_builder.rs:41
   5: tokio::task::runtime.spawn with kind=local task.name= task.id=46 loc.file="/opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.2.0/src/service.rs" loc.line=74 loc.col=17
      at /opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/util/trace.rs:16
   6: tokio::task::runtime.spawn with kind=local task.name= task.id=16 loc.file="/opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.2.0/src/worker.rs" loc.line=457 loc.col=25
      at /opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/util/trace.rs:16
   7: tokio::task::runtime.spawn with kind=local task.name= task.id=12 loc.file="/opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.2.0/src/worker.rs" loc.line=431 loc.col=21
      at /opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/util/trace.rs:16
   8: tokio::task::runtime.spawn with kind=local task.name= task.id=11 loc.file="/opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-rt-2.8.0/src/arbiter.rs" loc.line=316 loc.col=25
      at /opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/util/trace.rs:16
   9: tokio::task::runtime.spawn with kind=block_on task.name= task.id=10 loc.file="/opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-rt-2.8.0/src/arbiter.rs" loc.line=144 loc.col=24
      at /opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/util/trace.rs:16

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  <empty backtrace>

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering. http.status_code=500 otel.status_code="ERROR"}: tracing_actix_web::root_span_builder: close time.busy=3.23ms time.idle=95.3ms

Getting the Same Error, Using local and S3 Paths.

ericgaspar commented 1 year ago

!testme

yunohost-bot commented 1 year ago

:rocket: Test Badge

csolisr commented 1 year ago

Looks to me like this is missing some configuration files to be properly set - namely, the location of the folder for the uploads. As per the default settings, this defaults to /mnt/files.

ericgaspar commented 1 year ago

!testme

yunohost-bot commented 1 year ago

May the CI gods be with you! Test Badge

csolisr commented 1 year ago

So! Is it working now with this fix?

DrMxrcy commented 1 year ago

So! Is it working now with this fix?

Looks like it was approved!

csolisr commented 1 year ago

And just in time for the 0.18.3 update I guess! Can somebody merge it please?

ericgaspar commented 1 year ago

before merging, did someone tested this?

tituspijean commented 1 year ago

I am still experiencing what's described above: https://github.com/YunoHost-Apps/lemmy_ynh/pull/43#issuecomment-1597834043.

Do not merge.

tituspijean commented 1 year ago

Status after 4ab3b02

I had to download ImageMagick manually, the file downloaded binary was empty. 🤔 When trying to upload a file:

$ tail -f /var/log/lemmy/lemmy-pict-rs.log -n 0
2023-07-29T20:20:03.752414Z  INFO HTTP request{http.method=POST http.route=/image http.flavor=1.1 http.scheme=http http.host=127.0.0.1:8081 http.client_ip=2a01:cb11:9f:3200:3dbb:f936:cf89:ab51 http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0 http.target=/image otel.name=HTTP POST /image otel.kind="server" request_id=3fe80ef6-0da2-43b8-b21a-29bc62bae915}: tracing_actix_web::root_span_builder: new
fuse: device not found, try 'modprobe fuse' first
open dir error: No such file or directory
2023-07-29T20:20:03.909883Z  INFO HTTP request{http.method=POST http.route=/image http.flavor=1.1 http.scheme=http http.host=127.0.0.1:8081 http.client_ip=2a01:cb11:9f:3200:3dbb:f936:cf89:ab51 http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0 http.target=/image otel.name=HTTP POST /image otel.kind="server" request_id=3fe80ef6-0da2-43b8-b21a-29bc62bae915 trace_id=00000000000000000000000000000000 exception.message=Error in imagemagick exception.details=
   0: Error in imagemagick
   1: Invalid output format
   2: expected value at line 2 column 1

Location:
   /drone/src/src/error.rs:38

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

   0: pict_rs::validate::validate_bytes
      at src/validate.rs:38
   1: pict_rs::ingest::ingest with declared_alias=None should_validate=true
      at src/ingest.rs:44
   2: pict_rs::file-upload with filename="ynh_logo_white_300dpi.png"
      at src/lib.rs:160
   3: tracing_actix_web::root_span_builder::HTTP request with http.method=POST http.route=/image http.flavor=1.1 http.scheme=http http.host=127.0.0.1:8081 http.client_ip=2a01:cb11:9f:3200:3dbb:f936:cf89:ab51 http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0 http.target=/image otel.name=HTTP POST /image otel.kind="server" request_id=3fe80ef6-0da2-43b8-b21a-29bc62bae915 trace_id=00000000000000000000000000000000
      at /opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-actix-web-0.7.5/src/root_span_builder.rs:41
   4: tokio::task::runtime.spawn with kind=local task.name= task.id=51 loc.file="/opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.2.0/src/service.rs" loc.line=74 loc.col=17
      at /opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/util/trace.rs:16
   5: tokio::task::runtime.spawn with kind=local task.name= task.id=17 loc.file="/opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.2.0/src/worker.rs" loc.line=457 loc.col=25
      at /opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/util/trace.rs:16
   6: tokio::task::runtime.spawn with kind=local task.name= task.id=13 loc.file="/opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.2.0/src/worker.rs" loc.line=431 loc.col=21
      at /opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/util/trace.rs:16
   7: tokio::task::runtime.spawn with kind=local task.name= task.id=12 loc.file="/opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-rt-2.8.0/src/arbiter.rs" loc.line=316 loc.col=25
      at /opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/util/trace.rs:16
   8: tokio::task::runtime.spawn with kind=block_on task.name= task.id=11 loc.file="/opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-rt-2.8.0/src/arbiter.rs" loc.line=144 loc.col=24
      at /opt/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/util/trace.rs:16

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  <empty backtrace>

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering. http.status_code=500 otel.status_code="ERROR"}: tracing_actix_web::root_span_builder: close time.busy=3.43ms time.idle=154ms

The fuse: device not found, try 'modprobe fuse' first open dir error: No such file or directory might not be the actual issue, since I could run the ./magick command manually as $app user. Might be a permissions issue.

tituspijean commented 1 year ago

Status as of 8757210

Works! 🎉 Though... some explanations are required here.

  1. Pict-RS requirese ImageMagick 7, which is not available on Debian
  2. ImageMagick offers a nice AppImage binary, which makes is compatible on multiple platforms.
    • The ImageMagick download page does not offer a checksum, nor a versioning system, so I ended up directly wgetting it.
  3. To avoid running an unchecked binary running as root, I installed it in $install_dir/pict-rs/magick, and chowned it to lemmy system user
  4. However, I discover the AppImages require some privileges, as they need FUSE to do whatever they need to do to deploy the actual platform-compatible binary.
    • I have thus loosened lemmy-pict-rs.service systemd restrictions.

I think it is a dirty solution. We might need to find something system-wide, like the AppImage daemon.

tituspijean commented 1 year ago

!testme

yunohost-bot commented 1 year ago

Alrighty! Test Badge

yalh76 commented 1 year ago

Status as of 8757210

Works! 🎉 Though... some explanations are required here.

1. Pict-RS requirese ImageMagick 7, which is not available on Debian

2. ImageMagick offers a nice AppImage binary, which makes is compatible on multiple platforms.

   * The ImageMagick download page does not offer a checksum, nor a versioning system, so I ended up directly `wget`ting it.

3. To avoid running an unchecked binary running as root, I installed it in `$install_dir/pict-rs/magick`, and `chown`ed it to `lemmy` system user

4. However, I discover the AppImages require some privileges, as they need FUSE to do whatever they need to do to deploy the actual platform-compatible binary.

   * I have thus loosened `lemmy-pict-rs.service` systemd restrictions.

I think it is a dirty solution. We might need to find something system-wide, like the AppImage daemon.

* [x]  Downloaded file of ImageMagick AppImage is always empty

If it could help ImageMagick was implemented and build in lemmy_ynh and removed at https://github.com/YunoHost-Apps/lemmy_ynh/pull/30

tituspijean commented 1 year ago

If it could help ImageMagick was implemented and build in lemmy_ynh and removed at #30

Fixed by unpacking the AppImage :)

Ready to merge!

yalh76 commented 1 year ago

!testme !bookwormtestme

yunohost-bot commented 1 year ago

:books: :worm: Test Badge

yunohost-bot commented 1 year ago

:rocket: Test Badge

tituspijean commented 1 year ago

CI test on YunoHost 11 is experiencing the issue mentioned there: https://github.com/YunoHost-Apps/lemmy_ynh/pull/69#issuecomment-1658588298

D4llo commented 10 months ago

!testme

ericgaspar commented 10 months ago

!testme

yunohost-bot commented 10 months ago

Meow :cat2: Test Badge