No update of directories after adding a new one with relevant filters/regex

[alexdelprete]

Expected Behavior I added an "Adult" directory, and applied a couple of filters to match the filenames, I restarted zurg to trigger the rescan and let it update the directories, but nothing happens after the initial scan.

Actual Behavior Nothing happens. After the initial scan, where everything is under the same directory, no matter what I do there's no change to the dir structure.

Detailed Steps I edited config.yaml adding the new directory and the relevant filter

Operating System Linux - Debian 12

Environment Setup Docker with compose:

version: '3.8'

services:
  zurg:
    image: ghcr.io/debridmediamanager/zurg-testing:latest
    container_name: zurg
    restart: unless-stopped
    environment:
      TZ: Europe/Rome
      PUID: 1000
      PGID: 1000
    volumes:
      - $PWD/script/plex_update.sh:/app/plex_update.sh:ro
      - $PWD/config/config.yml:/app/config.yml
      - $PWD/zurgdata:/app/data
    ports:
      - 9999:9999

  rclone:
    image: rclone/rclone:latest
    container_name: rclone
    restart: unless-stopped
    command: "mount zurg: /data --allow-non-empty --allow-other --uid 1000 --gid 1000 --dir-cache-time 10s"
    environment:
      TZ: Europe/Rome
      PUID: 1000
      PGID: 1000
    cap_add:
      - SYS_ADMIN
    security_opt:
      - apparmor:unconfined
    devices:
      - /dev/fuse:/dev/fuse:rwm
    depends_on:
      - zurg
    volumes:
      - $PWD/config/rclone.conf:/config/rclone/rclone.conf
      - /mnt/nas/zurg:/data:rshared

Logs

Logging to logs/zurg-2024-01-02.log
2024-01-02T19:02:26.368Z        INFO    manager Compiled into 36 torrents, 0 were missing info
2024-01-02T19:02:26.447Z        INFO    manager Finished initializing torrent manager
2024-01-02T19:02:26.447Z        INFO    zurg    Starting server on [::]:9999
2024-01-02T19:02:26.448Z        INFO    manager Starting periodic refresh

Rclone Configuration

[zurg]
type = webdav
url = http://zurg:9999/dav
vendor = other
pacer_min_sleep = 0

[zurghttp]
type = http
url = http://zurg:9999/http
no_head = false
no_slash = false

Zurg Configuration

# Zurg configuration version
zurg: v1
token: abcdefghTOKEN # https://real-debrid.com/apitoken

# basic functionality
host: "[::]" # do not change this if you are running it inside a docker container
port: 9999 # do not change this if you are running it inside a docker container
concurrent_workers: 20
check_for_changes_every_secs: 15

# misc configs
retain_folder_name_extension: true # if true, zurg won't modify the filenames from real-debrid
retain_rd_torrent_name: true # if true, it will strictly follow RD API torrent name property w/c should make this more compatible with rdt-client
auto_delete_rar_torrents: true # if true, zurg will delete unstreamable rar files (these torrents will always be compressed in a rar archive no matter what files you select)
use_download_cache: true # if true, during zurg initialization, it will fetch all downloads to unrestrict links faster
enable_repair: true # BEWARE! THERE CAN ONLY BE 1 INSTANCE OF ZURG THAT SHOULD REPAIR YOUR TORRENTS
#on_library_update: sh plex_update.sh "$@"
on_library_update: |
  token="abcdefghTOKEN"
  plex_url="http://plex.axel.dom:32400"
  email="my@email.com"

  # Log the updated directories and send refresh request
  for arg in "$@"
  do
      echo "Detected update on: $arg"
      # URL encode the directory path
      encoded_arg=$(python -c "import urllib.parse; print(urllib.parse.quote_plus('$arg'))")
      curl -s -X GET "$plex_url/library/sections/all/refresh?path=$encoded_arg" -H "X-Plex-Token: $token"
  done

  echo "All updated sections refreshed."

  # Send an email notification
  echo "Plex library has been updated with these content: $@" | mail -s "zurg: Library Update" $email

# network configs
network_buffer_size: 1048576 # 1 MiB
serve_from_rclone: false # serve file data from rclone, not from zurg (zurg will only provide rclone the link to download)
verify_download_link: true # if true, zurg will check if the link is truly streamable; only relevant if serve_from_rclone is set to true (as it already does this all the time if serve_from_rclone is false)
force_ipv6: false # force connect to real-debrid ipv6 addresses
rate_limit_sleep_secs: 6 # wait time after getting a 429 from Real-Debrid API
realdebrid_timeout_secs: 60 # api timeout
retries_until_failed: 5 # api failures until considered failed
preferred_hosts: # Run ./zurg network-test
  - 44.download.real-debrid.com # hops: 11 latency: 10.331ms
  - 42.download.real-debrid.cloud # hops: 11 latency: 10.764ms
  - 50.download.real-debrid.cloud # hops: 11 latency: 10.886ms
  - 43.download.real-debrid.cloud # hops: 11 latency: 10.98ms
  - 51.download.real-debrid.cloud # hops: 11 latency: 10.998ms
  - 56.download.real-debrid.cloud # hops: 11 latency: 11.043ms
  - 45.download.real-debrid.cloud # hops: 11 latency: 11.045ms
  - 59.download.real-debrid.cloud # hops: 17 latency: 11.077ms
  - 41.download.real-debrid.cloud # hops: 11 latency: 11.08ms
  - 64.download.real-debrid.com # hops: 11 latency: 11.101ms
  - hkg1.download.real-debrid.com # hops: 11 latency: 11.128ms
  - 43.download.real-debrid.com # hops: 11 latency: 11.214ms
  - 69.download.real-debrid.com # hops: 11 latency: 11.255ms
  - 58.download.real-debrid.com # hops: 11 latency: 11.276ms
  - 67.download.real-debrid.com # hops: 11 latency: 11.284ms
  - 66.download.real-debrid.com # hops: 11 latency: 11.31ms
  - 58.download.real-debrid.cloud # hops: 11 latency: 11.331ms
  - 68.download.real-debrid.com # hops: 11 latency: 11.34ms
  - 31.download.real-debrid.com # hops: 11 latency: 11.35ms
  - 42.download.real-debrid.com # hops: 11 latency: 11.385ms

# List of directory definitions and their filtering rules
directories:
  # Configuration for anime shows
  anime:
    group: media # directories on different groups have duplicates of the same torrent
    group_order: 10 # group order = priority, it defines who eats first on a group
    filters:
      - and: # you can use nested 'and' & 'or' conditions
        - has_episodes: true # intelligent detection of episode files inside a torrent
        - any_file_inside_regex: /^\[/ # usually anime starts with [ e.g. [SubsPlease]
        - any_file_inside_not_regex: /s\d\de\d\d/i # and usually anime doesn't use SxxExx

  adult:
    group: media
    group_order: 15
    filters:
      - or:
        - any_files_inside_regex: /xxx/i
        - any_files_inside_regex: /sex/i
        - any_files_inside_regex: /adult/i

  shows:
    group: media
    group_order: 20
    filters:
      - has_episodes: true  # intelligent detection of episode files inside a torrent

  movies:
    group: media  # because anime, shows and movies are in the same group,
    group_order: 30 # and anime and shows has a lower group_order number than movies, all torrents that doesn't fall into the previous 2 will fall into movies
    only_show_the_biggest_file: true # let's not show the other files besides the movie itself
    filters:
      - regex: /.*/ # you cannot leave a directory without filters because it will not have any torrents in it

[yowmamasita]

did you happen to check http://zurgip:port on a browser and see if that new directory contains anything?

[alexdelprete]

did you happen to check http://zurgip:port on a browser and see if that new directory contains anything?

Yes, it was the first thing I did, and it's empty.

Maybe it's the regexp used, it doesn't match maybe? In the log I don't see anything to debug the problem. If you could add some debugging output it would be helpful.

I used regexp syntax provided in the docs examples, even though they're not standard regexp, I tried them in regex101.com and it says they're not valid. Could that be the problem?

[yowmamasita]

ah right yes the config is wrong

any_file_inside_regex instead of any_files_inside_regex

you can also remove the or condition because it's or on the first level anw

adult:
    group: media
    group_order: 15
    filters:
      - any_file_inside_regex: /xxx/i
      - any_file_inside_regex: /sex/i
      - any_file_inside_regex: /adult/i

[alexdelprete]

any_file_inside_regex instead of any_files_inside_regex

ok, now something is finally working: with /xxx/i it moved in adult/ all files with xxx (lowercase) but not the ones with XXX (uppercase) or mixed-case (xXx).

Are you sure about that regexp nonstandard syntax? What standard/library do you use for regexp parsing?

[yowmamasita]

try it with /mkv/i or /mkv/ or /MKV/ to test it

it works fine for me

[alexdelprete]

Solved it with: regex: /xxx/i. But then I found out that contains: xxx is even simpler, and it's case-insensitive. :)

Question: when you update the config.yml file, does it refresh automatically or I have to restart the container every time I update it? Would be cool if it was dynamic.

I hope we have some more verbose logging to troubleshoot things in the future, but you've made an amazing work with zurg, thank you very much also for the great support.

Now I have to understand how/if it works with rdt-client.

[yowmamasita]

you have to restart. docker doesn't play well with signaling mounted file changes to the container and I don't want to continuously read the file and check for changes because that stresses up drives.

sure, what was missing on the logs for you that will help?

[alexdelprete]

I have Traefik and other services (Homepage for example) that are docker based and react dynamically to config file changes. I read it's not based on polling but a notification service...but I didn't look into it too much, if you want I can take a deeper look at it.

what was missing on the logs for you that will help?

for example when it processes the torrents, how it applies the filters, etc. when the regex didn't work I had no clue if it was doing something at all, if the filters were being applied, if the filters were wrong, etc.

[yowmamasita]

ah cool, if you can send the image here that you're specifically looking, i'll take a look

the filter evaluation im planning on doing differently for debugging. i'll share more when i start developing it.

[alexdelprete]

Traefik (dynamic configuration file automatic reload): https://github.com/traefik/traefik Homepage (any config file change gets automatically reloaded): https://github.com/gethomepage/homepage

the filter evaluation im planning on doing differently for debugging. i'll share more when i start developing it.

ok, great.