Zurg Mount is messing up Docker Desktop in M1 Max MBP running MacOS

akhanalcs commented 3 days ago

Expected Behavior

Zurg and Rclone working without any issues.

Actual Behavior

Whenever I mount the zurg webdav in M1 Max MBP running macOS, Docker Desktop freezes and containers accessing the zurg mount refuse to start up, ultimately crashing Docker Desktop.

Detailed Steps

First take a look at my folder structure:

home-media-server
├── docker
│   └── appdata <--- appdata for containers live here
│       ├── radarr
│       ├── zurg
│       ├── rdtclient
│       └── prowlarr
├── data
│   ├── symlinks
│   │   └── radarr <--- RDTClient downloads symlinks here
│   ├── realdebrid  <--- RD mounted folder
│   │   └── torrents <--- RD content shows up here
│   ├── zurg-tmp <--- Zurg temp directory
│   └── media
│       ├── movies <--- Radarr brings movies from symlinks/radarr to here. Plex also points here.
│       └── tv
└── non-docker-apps
    └── rclone     <--- I run `./rclone serve webdav zurg: --addr 0.0.0.0:8080 --dir-cache-time 20s --vfs-cache-mode full --vfs-cache-max-size 20G --cache-dir=“/Users/fastesthorse/home-media-server/data/zurg-tmp”` command here. And mount it using `/sbin/mount_webdav http://[::]:8080 /Users/fastesthorse/home-media-server/data/realdebrid`
        ├── rclone <--- downloaded binary
        └── rclone.conf

My compose looks like this:

# CFG_ROOT_DIR=/Users/fastesthorse/home-media-server
# DATA_ROOT_DIR=/Users/fastesthorse/home-media-server
services:
  radarr:
    container_name: radarr
    image: ghcr.io/hotio/radarr:latest
    restart: unless-stopped
    logging:
      driver: json-file
    ports:
      - 7878:7878
    environment:
      - PUID=1000
      - PGID=1000
      - UMASK=002
      - TZ=${TIMEZONE}
    volumes:
      - ${CFG_ROOT_DIR}/docker/appdata/radarr:/config
      - ${DATA_ROOT_DIR}/data:/data
      - ${DATA_ROOT_DIR}/data/symlinks/radarr:/downloads/radarr
      - ${DATA_ROOT_DIR}/data/realdebrid/torrents:/data/realdebrid/torrents
  zurg:
    image: ghcr.io/debridmediamanager/zurg-testing:v0.9.3-final
    container_name: zurg
    restart: unless-stopped
    healthcheck:
      test: curl -f localhost:9999/dav/version.txt || exit 1
    ports:
      - 9999:9999
    volumes:
      - ${CFG_ROOT_DIR}/docker/appdata/zurg/config.yml:/app/config.yml
      - ${CFG_ROOT_DIR}/docker/appdata/zurg/logs:/app/logs
      - ${CFG_ROOT_DIR}/docker/appdata/zurg/data:/app/data
  rdtclient:
    image: rogerfar/rdtclient
    container_name: rdtclient
    restart: unless-stopped
    ports:
      - 6500:6500
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=${TIMEZONE}
    volumes:
      - ${CFG_ROOT_DIR}/docker/appdata/rdtclient:/data/db
      - ${DATA_ROOT_DIR}/data:/data
      - ${DATA_ROOT_DIR}/data/realdebrid/torrents:/data/realdebrid/torrents
      - ${DATA_ROOT_DIR}/data/symlinks:/data/downloads

First I start zurg: docker compose up -d zurg
I run rclone natively to expose webdav to host os ./rclone serve webdav zurg: --addr 0.0.0.0:8080 --dir-cache-time 20s --vfs-cache-mode full --vfs-cache-max-size 20G --cache-dir=“/Users/fastesthorse/home-media-server/data/zurg-tmp”
Mount webdav exposed by rclone /sbin/mount_webdav http://[::]:8080 /Users/fastesthorse/home-media-server/data/realdebrid
I then run the remaining containers: docker compose up -d
At this point rdtclient, zurg and radarr apps will refuse to load in the browser. The docker desktop will hang.

Whenever I quit (^ C) step 2 and unmount step 3 above, the computer comes back to normal and rdtclient and radarr load normally.

My guess is that rdtclient and radarr run into issues when they're trying to access remote volume mount?

Operating System

macOS Sonoma 14.6.1. Rclone version: v1.68.1. Zurg version: v0.9.3-final.

Environment Setup

I'm using rclone binary, but all others are running on Docker. The compose file is above.

Logs

RDTClient:

zurg:

After it stopped hanging, I got this:

Radarr:

rclone (redacted):

2024/10/05 13:44:40 ERROR : torrents/The.Simpsons.S35.1080p.DSNP.WEB-DL.DDP5.1.H.264-NTb/._The.Simpsons.S35E01.Homers.Crossing.1080p.DSNP.WEB-DL.DDP5.1.H.264-NTb.mkv: Failed to copy: Method not allowed: 405 Method Not Allowed
2024/10/05 13:44:40 ERROR : torrents/The.Simpsons.S35.1080p.DSNP.WEB-DL.DDP5.1.H.264-NTb/._The.Simpsons.S35E02.A.Mid-Childhood.Nights.Dream.1080p.DSNP.WEB-DL.DDP5.1.H.264-NTb.mkv: Failed to copy: Method not allowed: 405 Method Not Allowed
2024/10/05 13:44:40 ERROR : torrents/The.Simpsons.S35.1080p.DSNP.WEB-DL.DDP5.1.H.264-NTb/._The.Simpsons.S35E01.Homers.Crossing.1080p.DSNP.WEB-DL.DDP5.1.H.264-NTb.mkv: vfs cache: failed to upload try #4, will retry in 1m20s: vfs cache: failed to transfer file from cache to remote: Method not allowed: 405 Method Not Allowed
2024/10/05 13:44:40 ERROR : torrents/The.Simpsons.S35.1080p.DSNP.WEB-DL.DDP5.1.H.264-NTb/._The.Simpsons.S35E02.A.Mid-Childhood.Nights.Dream.1080p.DSNP.WEB-DL.DDP5.1.H.264-NTb.mkv: vfs cache: failed to upload try #4, will retry in 1m20s: vfs cache: failed to transfer file from cache to remote: Method not allowed: 405 Method Not Allowed
2024/10/05 13:49:25 ERROR : torrents/.DS_Store: Failed to copy: 404 page not found: 404 Not Found
2024/10/05 13:49:25 ERROR : ._.DS_Store: Failed to copy: 404 page not found: 404 Not Found
2024/10/05 13:49:25 ERROR : torrents/.DS_Store: vfs cache: failed to upload try #4, will retry in 1m20s: vfs cache: failed to transfer file from cache to remote: 404 page not found: 404 Not Found
2024/10/05 13:49:25 ERROR : .DS_Store: Failed to copy: 404 page not found: 404 Not Found
2024/10/05 13:49:25 ERROR : .DS_Store: vfs cache: failed to upload try #5, will retry in 2m40s: vfs cache: failed to transfer file from cache to remote: 404 page not found: 404 Not Found
2024/10/05 13:49:25 ERROR : .___unplayable__: Failed to copy: 404 page not found: 404 Not Found
2024/10/05 13:49:25 ERROR : .___unplayable__: vfs cache: failed to upload try #5, will retry in 2m40s: vfs cache: failed to transfer file from cache to remote: 404 page not found: 404 Not Found
2024/10/05 13:49:25 ERROR : ._.DS_Store: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: 404 page not found: 404 Not Found

Rclone Configuration

[zurg]
type = webdav
url = http://localhost:9999/dav
vendor = other
pacer_min_sleep = 0

Zurg Configuration

# Zurg configuration version
zurg: v1
token: <my-token-here> # https://real-debrid.com/apitoken
# host: "[::]"
# port: 9999
# username:
# password:
# proxy:
api_rate_limit_per_minute: 60
torrents_rate_limit_per_minute: 25
concurrent_workers: 32
check_for_changes_every_secs: 10
# repair_every_mins: 60
ignore_renames: true
retain_rd_torrent_name: true
retain_folder_name_extension: true
enable_repair: false
auto_delete_rar_torrents: false
get_torrents_count: 5000
# api_timeout_secs: 15
# download_timeout_secs: 10
# enable_download_mount: false
# rate_limit_sleep_secs: 6
# retries_until_failed: 2
# network_buffer_size: 4194304 # 4MB
# pass the download link to rclone instead of zurg streaming it for you.
# Dev recommended to comment out this line if you are not sure what you are doing.
# serve_from_rclone: false
# verify_download_link: false
# force_ipv6: false
directories:
  torrents:
    group: 1
    filters:
      - regex: /.*/

JurajNyiri commented 3 days ago

I believe I have encountered something similar in the past. What helped was mounting one folder above the actual Zurg mount instead. So in your case realdebrid instead of realdebrid/torrents

akhanalcs commented 3 days ago

It somehow recovered which is strange. And I added movie from Radarr and RDTClient, but RDTClient created a folder with extension which is preventing Radarr from finding the correct folder.

Added movie in Radarr
RDTClient finished downloading

At /Users/fastesthorse/home-media-server/data/symlinks/radarr. Notice .mkv extension in folder.
Radarr getting stuck

akhanalcs commented 3 days ago

I believe I have encountered something similar in the past. What helped was mounting one folder above the actual Zurg mount instead. So in your case realdebrid instead of realdebrid/torrents

@JurajNyiri I'm doing whatever you're recommending, isn't it? Or am I missing something. Because this is the command I used:

/sbin/mount_webdav http://[::]:8080 /Users/fastesthorse/home-media-server/data/realdebrid

torrents comes from zurg config.yml

# redacted
# ...
directories:
  torrents:
    group: 1
    filters:
      - regex: /.*/

JurajNyiri commented 3 days ago

${DATA_ROOT_DIR}/data/realdebrid/torrents:/data/realdebrid/torrents

I see this in your docker mount. If torrents is the exact space where zurg puts in files it needs to be one way above. Basically if you have a remote directory, I found for docker you need to mount one level up.

Also of note, I was experiencing instability issues after the WebDAV native mount so since then I have migrated back to native zurg lib outside of docker (zurg in docker had other issues - running OOM) and a rclone http mount.

JurajNyiri commented 3 days ago

… now that I think about it, i encountered issues with that as well. So finally I migrated my media stack to rpi4 and run 2 instances of zurg (one without repair). One for Plex on m1 (both without docker, it was causing too many issues) and one on rpi with the whole stack (all in docker). That has been rock stable, so much that I almost forgot to mention this before as I was writing previous comment :)

akhanalcs commented 3 days ago

Oh nice.

I'll probably run zurg without docker too.

So you have it setup and working on Plex on M1 too?

The rock solid stable one is the one running on RPi? Sorry, I'm not understanding the 2 setup thing.

Would be great if I could take a look at your folder strucutre (the way I created in the post above), your compose file, your RDTClient web ui settings etc.

If I could make it work, I'd probably write a guide so other folks don't have to pull hair to get this thing working, lol.

JurajNyiri commented 3 days ago

Setup 1 on m1:

Zurg native binary Plex server native install Rclone native binary Script that keeps track of whether the folder is mounted or not and mounts zurg http endpoint if not. Running every minute. Script that launches zurg inside screen unless it is already running, runs every minute. Plex simply pointed to the rclone folders and folders where sonarr and radarr import files (see setup 2)

Setup 2:

rpi4 Radarr, sonarr, rdtclient etc with symlink setup. All via docker with custom script that copies files into m1 on import via rsync. What is important is that the symlink points to the expected all folder path in m1 so that symlink still works. This is possible because I set it to the same path inside the docker containers so what all containers see is the same path as it would be in m1.

It’s pretty complex but hopefully this helps. If I were you I would take it in small steps first. First get Plex and zurg working via native. Then find Linux device (not Mac) to run the media stack. Only then start playing with the import scripts (fun stuff to look out for there are weird characters in paths / file names which break rsync copy - make sure to either properly escape them in script or with the rename function in sonarr and radarr.

akhanalcs commented 3 days ago

@JurajNyiri That sounds really interesting but I don't think I could do all that without some guide/ help/ examples. So, any chance you could write a guide for your setup here in this thread or at some github pages blog please?

And I tried native zurg and rclone, but the supporting containers (radarr, rdtclient) like to hang when remote volume is mounted to them.

Sometimes they recover (the hang recovers after 3-5 minutes) but symlinks aren't working, so I'm probably doing something else wrong too. So making Plex, zurg and rclone work on a Mac seems impossible at this point.

Can you see what I'm doing wrong here?

I request the movie in Radarr
It goes to RDTClient, and it downloads it to - ${DATA_ROOT_DIR}/data/symlinks:/data/downloads. Sometimes it shows up there, sometimes it doesn't, it's so inconsistent.

Now it's showing up at ${DATA_ROOT_DIR}/data/downloads/radarr where it shouldn't show up because Radarr will be looking at ${DATA_ROOT_DIR}/data/symlinks/radarr.
At this point, Radarr is stuck.
Now just for testing the symlink, if I try to play the file at ${DATA_ROOT_DIR}/data/downloads/radarr, it doesn't even work:

So there's myriad of problems here that I can't figure out which is incredibly frustrating.

My compose looks like this now:

services:
  radarr:
    container_name: radarr
    image: ghcr.io/hotio/radarr:latest
    restart: unless-stopped
    logging:
      driver: json-file
    ports:
      - 7878:7878
    environment:
      - PUID=1000
      - PGID=1000
      - UMASK=002
      - TZ=${TIMEZONE}
    volumes:
      - ${CFG_ROOT_DIR}/docker/appdata/radarr:/config
      - ${DATA_ROOT_DIR}/data:/data
      - ${DATA_ROOT_DIR}/data/symlinks:/downloads
  zurg:
    image: ghcr.io/debridmediamanager/zurg-testing:v0.9.3-final
    container_name: zurg
    restart: unless-stopped
    healthcheck:
      test: curl -f localhost:9999/dav/version.txt || exit 1
    ports:
      - 9999:9999
    volumes:
      - ${CFG_ROOT_DIR}/docker/appdata/zurg/config.yml:/app/config.yml
      - ${CFG_ROOT_DIR}/docker/appdata/zurg/logs:/app/logs
      - ${CFG_ROOT_DIR}/docker/appdata/zurg/data:/app/data
  rdtclient:
    image: rogerfar/rdtclient
    container_name: rdtclient
    restart: unless-stopped
    ports:
      - 6500:6500
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=${TIMEZONE}
    volumes:
      - ${CFG_ROOT_DIR}/docker/appdata/rdtclient:/data/db
      - ${DATA_ROOT_DIR}/data:/data
      - ${DATA_ROOT_DIR}/data/realdebrid:/data/realdebrid
      - ${DATA_ROOT_DIR}/data/symlinks:/data/downloads
    logging:
      driver: json-file
      options:
        max-size: 10m

The settings in my RDTClient look like this:

I tried changing the path to data/realdebrid for rclone mount below, the app kept hanging constantly so I changed it back to data/realdebrid/torrents

JurajNyiri commented 3 days ago

And I tried native zurg and rclone, but the supporting containers (radarr, rdtclient) like to hang when remote volume is mounted to them.

In my experience, you will always have some kind of issues with docker on mac, it doesn't play well with remote mounts, I suspect due to Linux VM overhead. I tried all the ways. I found that you simply need to run the whole thing on linux, or the part with docker on linux.

As for the guide, check this page https://savvyguides.wiki/sailarrsguide/ (not written, or contributed to, by me).

unfixa1 commented 2 days ago

Docker image does not support ARM architecture It is recommended that you use Servbay to replace docker to deploy the development environment. Servbay has fewer bugs and supports the arm architecture.

akhanalcs commented 11 hours ago

@JurajNyiri Actually I was following that guide too. 😀

After reading your answer, I've decided to just put Linux on this Mac Mini to make this work. I watched some videos and tried installing Ubuntu Server on it. But I'm unable to mount the efi volume when I use LVM. Do you know of a solution to this:

https://askubuntu.com/q/1529376/1967188

I've put all the details in the question.

akhanalcs commented 11 hours ago

@unfixa1 Thank you for the response but I'm working on 2018 Intel Mac Mini, so I'm not working with ARM arch.

debridmediamanager / zurg-testing