Odd Behavior With Omnimount

[Misterobots]

I have done my best to try and troubleshoot the odd behavior happening. Testing scenarios: Full OmniStream Stack - Fresh rebuild Omnimount only - Fresh rebuild Pre-setup of Rclone Mount Fresh Rclone drive/remotes Imported Rclone drive/remotes

I am having an odd issue with Omnimount/Rclone. Going through setup, there are no errors in any of the logs (omni or rclone), but when setting up Omnimount, only the "cloud" mount is visible. I can see the subdirectories (Downloads: empty, Media: empty), but no "Media", "unsynched", or "uploadcache".

When I exec into the omnimount container, I can see all of the directories, and I can see that Media is successfully linked to the rclone mount ( I can see drive contents), but that's as far as it gets.

I thought maybe some odd issue with mount binding and contents being "obscured" but everything seems set correctly. I am happy to provide any info or additional logs you may want. I appreciate that assistance as always.

Omnimount container

Host

[Misterobots]

Omnimout log

omnimount | rclone v1.64.0 omnimount | - os/version: debian 12.1 (64 bit) omnimount | - os/kernel: 5.15.0-84-generic (x86_64) omnimount | - os/type: linux omnimount | - os/arch: amd64 omnimount | - go/version: go1.21.1 omnimount | - go/linking: static omnimount | - go/tags: none omnimount | omnimount | mergerfs v2.37.1 omnimount | omnimount | https://github.com/trapexit/mergerfs omnimount | https://github.com/trapexit/support omnimount | omnimount | ISC License (ISC) omnimount | omnimount | Copyright 2023, Antonio SJ Musumeci trapexit@spawn.link omnimount | omnimount | Permission to use, copy, modify, and/or distribute this software for omnimount | any purpose with or without fee is hereby granted, provided that the omnimount | above copyright notice and this permission notice appear in all omnimount | copies. omnimount | omnimount | THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL omnimount | WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED omnimount | WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE omnimount | AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL omnimount | DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR omnimount | PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER omnimount | TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR omnimount | PERFORMANCE OF THIS SOFTWARE. omnimount | omnimount | omnimount | Starting vnstat omnimount | No interfaces found in database, adding available interfaces... omnimount | Interface "eth0" added with 10000 Mbit bandwidth limit. omnimount | -> 1 new interface found. omnimount | Limits can be modified using the configuration file. See "man vnstat.conf". omnimount | Unwanted interfaces can be removed from monitoring with "vnstat --remove". omnimount | omnimount | Configuration: omnimount | MERGEMOUNT=cloud omnimount | RCLONESERVICE=Media omnimount | RCLONEMOUNT=Media omnimount | UNSYNCED=unsynced omnimount | UPLOADCACHE=uploadcache omnimount | USENFS=false omnimount | NFSREMOTE= omnimount | NFSLOCAL= omnimount | MEDIA=Media omnimount | TURBOMAX=20 omnimount | LOCAL=Media omnimount | Adding group omniuser' (GID 1000) ... omnimount | Done. omnimount | Adding useromniuser' ... omnimount | Adding new user omniuser' (1000) with groupomniuser (1000)' ... omnimount | Creating home directory /home/omniuser' ... omnimount | Copying files from/etc/skel' ... omnimount | Adding new user omniuser' to supplemental / extra groupsusers' ... omnimount | Adding user omniuser' to groupusers' ... omnimount | Cleaning up leftovers omnimount | Starting services omnimount | VFSMAX=100G omnimount | VFSAGE=48h omnimount | VFSPOLL=5m omnimount | VFSREAD=2G omnimount | VFSCACHE=yes omnimount | DIRCACHE=96h omnimount | NFS Disabled omnimount | mkdir -p /mnt/Media omnimount | OmniMount Caching: enabled omnimount | { omnimount | "jobid": 1 omnimount | } omnimount | omnimount | Startup complpete

[kelinger]

I was traveling last week but finally got a chance to read through your post this weekend.

That's very odd. Towards the end of the log file, you can see the "jobid = 1" which tells me that the remotely mounted service is up and that the directory scan has started. If the rclone mount failed, it would mean that the rclone daemon isn't running and this background task would have failed.

Based on the data in the logs, I'm assuming that the entry in your rclone config for your cloud service (Google, Dropbox, etc.) is called "Media" correct?

[meharrington90]

This is the same issue I'm having from the other ticket.

[kelinger]

OK, but in the other thread, your settings show as:

omnimount | MERGEMOUNT=cloud
omnimount | RCLONESERVICE=google
omnimount | RCLONEMOUNT=google
omnimount | UNSYNCED=unsynced
omnimount | UPLOADCACHE=uploadcache

and in this thread they show as

omnimount | MERGEMOUNT=cloud
omnimount | RCLONESERVICE=Media
omnimount | RCLONEMOUNT=Media
omnimount | UNSYNCED=unsynced
omnimount | UPLOADCACHE=uploadcache

So I wanted to check. If this is correct, then I'll move on. If not, the first thing I'd check is to make sure the rclone.conf in the /configs for OmniMount is accurate.

[meharrington90]

Yep @Misterobots is having the same issue as me, I just piped in on this ticket. I'm assuming his rclone mount is just named "Media" while I named mine "google"

[kelinger]

OK, duh... missed that.

I just recompiled OmniMount with the newer Debian software and an updated Rclone. It's a long shot, but could you try to refresh it (omni update) ?

[meharrington90]

Nogo. Did a full sudo apt update && sudo apt upgrade and reboot as well.

omnimount      | Configuration:
omnimount      | MERGEMOUNT=cloud
omnimount      | RCLONESERVICE=google
omnimount      | RCLONEMOUNT=google
omnimount      | UNSYNCED=unsynced
omnimount      | UPLOADCACHE=uploadcache
omnimount      | USENFS=false
omnimount      | NFSREMOTE=
omnimount      | NFSLOCAL=
omnimount      | MEDIA=Media
omnimount      | TURBOMAX=20
omnimount      | LOCAL=google
omnimount      | Adding group `omniuser' (GID 1000) ...
omnimount      | Done.
omnimount      | Adding user `omniuser' ...
omnimount      | Adding new user `omniuser' (1000) with group `omniuser (1000)' ...
omnimount      | Creating home directory `/home/omniuser' ...
omnimount      | Copying files from `/etc/skel' ...
omnimount      | Adding new user `omniuser' to supplemental / extra groups `users' ...
omnimount      | Adding user `omniuser' to group `users' ...
omnimount      | Cleaning up leftovers
omnimount      | Starting services
omnimount      | VFSMAX=100G
omnimount      | VFSAGE=48h
omnimount      | VFSPOLL=5m
omnimount      | VFSREAD=2G
omnimount      | VFSCACHE=yes
omnimount      | DIRCACHE=96h
omnimount      | NFS Disabled
omnimount      | mkdir -p /mnt/google
omnimount      | OmniMount Caching: enabled
omnimount      | {
omnimount      |        "jobid": 1
omnimount      | }
omnimount      | 
omnimount      | Startup complpete
^Ccanceled
plexabyte@Plexabyte:~/OmniStream/mnt$ ls
cloud
plexabyte@Plexabyte:~/OmniStream/mnt$

[Misterobots]

I was traveling last week but finally got a chance to read through your post this weekend.

That's very odd. Towards the end of the log file, you can see the "jobid = 1" which tells me that the remotely mounted service is up and that the directory scan has started. If the rclone mount failed, it would mean that the rclone daemon isn't running and this background task would have failed.

Based on the data in the logs, I'm assuming that the entry in your rclone config for your cloud service (Google, Dropbox, etc.) is called "Media" correct?

That's correct. Media is the remote name.

You are also correct that within the Omnimount container, the storage Remote is fully accessible, visibly correct, and operates as I normally see. Outside the container, the "Media" directory is all that shows, and it's empty from the host view.

[robnester]

I'm seeing similar behavior with a fresh installation on a fresh installation of Debian.

$ omni logs omnimounts
omnimount  | rclone v1.64.2
omnimount  | - os/version: debian 12.2 (64 bit)
omnimount  | - os/kernel: 6.1.0-13-amd64 (x86_64)
omnimount  | - os/type: linux
omnimount  | - os/arch: amd64
omnimount  | - go/version: go1.21.3
omnimount  | - go/linking: static
omnimount  | - go/tags: none
omnimount  | 
omnimount  | mergerfs v2.37.1
omnimount  | 
omnimount  | https://github.com/trapexit/mergerfs
omnimount  | https://github.com/trapexit/support
omnimount  | 
omnimount  | ISC License (ISC)
omnimount  | 
omnimount  | Copyright 2023, Antonio SJ Musumeci <trapexit@spawn.link>
omnimount  | 
omnimount  | Permission to use, copy, modify, and/or distribute this software for
omnimount  | any purpose with or without fee is hereby granted, provided that the
omnimount  | above copyright notice and this permission notice appear in all
omnimount  | copies.
omnimount  | 
omnimount  | THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
omnimount  | WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
omnimount  | WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE
omnimount  | AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL
omnimount  | DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR
omnimount  | PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
omnimount  | TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
omnimount  | PERFORMANCE OF THIS SOFTWARE.
omnimount  | 
omnimount  | 
omnimount  | Starting vnstat
omnimount  | No interfaces found in database, adding available interfaces...
omnimount  | Interface "eth0" added with 10000 Mbit bandwidth limit.
omnimount  | -> 1 new interface found.
omnimount  | Limits can be modified using the configuration file. See "man vnstat.conf".
omnimount  | Unwanted interfaces can be removed from monitoring with "vnstat --remove".
omnimount  | 
omnimount  | Configuration:
omnimount  | MERGEMOUNT=cloud
omnimount  | RCLONESERVICE=gdrive
omnimount  | RCLONEMOUNT=gdrive
omnimount  | UNSYNCED=unsynced
omnimount  | UPLOADCACHE=uploadcache
omnimount  | USENFS=false
omnimount  | NFSREMOTE=
omnimount  | NFSLOCAL=
omnimount  | MEDIA=Media
omnimount  | TURBOMAX=20
omnimount  | LOCAL=gdrive
omnimount  | Adding group `omniuser' (GID 1000) ...
omnimount  | Done.
omnimount  | Adding user `omniuser' ...
omnimount  | Adding new user `omniuser' (1000) with group `omniuser (1000)' ...
omnimount  | Creating home directory `/home/omniuser' ...
omnimount  | Copying files from `/etc/skel' ...
omnimount  | Adding new user `omniuser' to supplemental / extra groups `users' ...
omnimount  | Adding user `omniuser' to group `users' ...
omnimount  | Cleaning up leftovers
omnimount  | Starting services
omnimount  | VFSMAX=100G
omnimount  | VFSAGE=48h
omnimount  | VFSPOLL=5m
omnimount  | VFSREAD=2G
omnimount  | VFSCACHE=yes
omnimount  | DIRCACHE=96h
omnimount  | NFS Disabled
omnimount  | mkdir -p /mnt/gdrive
omnimount  | OmniMount Caching: enabled
omnimount  | {
omnimount  |    "jobid": 1
omnimount  | }
omnimount  | 
omnimount  | Startup complpete

On the server:

user@host:~/OmniStream/mnt$ ls
cloud
user@host:~/OmniStream/mnt$ ls cloud/
Downloads

One the omnimount container:

root@omnimount:~# cd /mnt/
root@omnimount:/mnt# ls -alh
total 16K
drwxr-xr-x 6 omniuser omniuser 4.0K Oct 24 14:57 .
drwxr-xr-x 1 root     root     4.0K Oct 24 14:57 ..
drwxrwxrwx 1 omniuser omniuser    0 Oct 24 14:57 cloud
drwxrwxrwx 1 omniuser omniuser    0 Oct 24 14:57 gdrive
drwxr-xr-x 2 omniuser omniuser 4.0K Oct 24 14:57 unsynced
drwxr-xr-x 3 omniuser omniuser 4.0K Oct 24 14:57 uploadcache
root@omnimount:/mnt# ls -alh gdrive/
total 4.0K
drwxrwxrwx 1 omniuser omniuser    0 Oct 24 14:57 .
drwxr-xr-x 6 omniuser omniuser 4.0K Oct 24 14:57 ..
drwxrwxrwx 1 omniuser omniuser    0 Oct 24 14:57 Media
root@omnimount:/mnt# ls -alh cloud/
total 8.0K
drwxrwxrwx 1 omniuser omniuser    0 Oct 24 14:57 .
drwxr-xr-x 6 omniuser omniuser 4.0K Oct 24 14:57 ..
drwxr-xr-x 2 omniuser omniuser 4.0K Oct 24 14:57 Downloads
drwxrwxrwx 1 omniuser omniuser    0 Oct 24 14:57 Media

[Misterobots]

I will add that mounting Remotes with Rclone, outside of the container, work as expected. There is just no "tie" in to the core function with Omnimount. All signs point to an issue with the container implementation, but if there is, I cannot find it at all.

[meharrington90]

Yep I was able to mount it outside of the container as well. If I had to guess, it would be an issue with the config for the bind from container to host. Although I'm not skilled enough in Docker to troubleshoot that.

This section is worth a look though, in the 003-omnimount.yaml file:

    volumes:
      - ${CONFIGS}/omnimount:/config
      - ${LOGS}:/logs
      - ${OMNIHOME}/healthchecks/omnimountcheck:/omnimountcheck
      - type: bind
        source: ${OMNIHOME}/mnt
        target: /mnt
        bind:
          propagation: shared

[Misterobots]

Yep I was able to mount it outside of the container as well. If I had to guess, it would be an issue with the config for the bind from container to host. Although I'm not skilled enough in Docker to troubleshoot that.

This section is worth a look though, in the 003-omnimount.yaml file:

    volumes:
      - ${CONFIGS}/omnimount:/config
      - ${LOGS}:/logs
      - ${OMNIHOME}/healthchecks/omnimountcheck:/omnimountcheck
      - type: bind
        source: ${OMNIHOME}/mnt
        target: /mnt
        bind:
          propagation: shared

I did some investigation into exactly that. Unfortunately, I wasn't able to come to any conclusions. The shared propagation designation and binding to a directory should allow both the host and container to see and interact with all directories in "/mnt" (with proper permission set of course). Initially I played around with that bind, but I was not able to affect any positive change.

[kelinger]

I'm creating a test box. I've downloaded the latest Debian Bookworm DVD (12.2) and will install a fresh system from it. I'll be running apt update and installing sudo, but that's the only prep I'm doing before trying the OmniStream installation.

Does anyone in this thread believe this will NOT mirror their starting environment?

[robnester]

This mirrors my process.

[Misterobots]

This mirrors my process.

Same. Please let us know if we can do anything to assist further.

[kelinger]

Well, if there's good news, its that I was able to replicate the issue on my test box.

The bad news is that there is no logical reason why this should be happening. My "production" box and the test box run Debian 12.2. Both are running identical versions of Docker and Docker Compose. Both are also running the same version of Rclone and fuse. The only real difference is that my production system started on Debian 12.0 and gradually upgraded to 12.2 after months of regular updates.

This is going to take more time to solve.

[kelinger]

For those interested in more details, the mount between the host and OmniMount is coming up and it is indeed showing that propagation of the sub-directories of "/mnt" is enabled. It's just not... propagating.

            {
                "Type": "bind",
                "Source": "/home/kelinger/OmniStream/mnt",
                "Destination": "/mnt",
                "Mode": "rw,rshared",
                "RW": true,
                "Propagation": "rshared"
            }

[meharrington90]

@kelinger Hmmm. Do you think this is related to the change you made last month regarding NFS somehow? My installation broke (on both servers) when that update was posted. It was in the previous ticket I commented on.

[kelinger]

You'd think that, right? But rclone is mounting things inside the container and so is NFS. The difference is that NFSs mounts make it back to the host while Rclone's do not.

I'm looking into it being potentially fuse-related right now.

[kelinger]

@meharrington90 Can you try the following on your host?

omni down
sudo apt remove fuse
sudo apt update
sudo apt install -y fuse3
omni up

[mahogl]

Just to add, i also did experienced this bug before i switched to the NFS config.

Did a fresh install and could not get the mount the to work on the fresh install. So did a restore of my old server, with working mount and used that instead of the fresh install since i could not work out why it did not work.

[robnester]

@meharrington90 Can you try the following on your host?

omni down
sudo apt remove fuse
sudo apt update
sudo apt install -y fuse3
omni up

I tried this, with no success. I'll await results from @meharrington90 to see if they're similar.

[kelinger]

Interesting because after I upgraded my fuse on the host, this eliminated the problem.

@robnester can you verify on the host that fuse3 is running? fusermount --v (note there are two dashes)

You may also try rebooting to make sure the previous version was properly unloaded (I didn't need to reboot but fuse wasn't in use when I upgraded it, either)

[robnester]

$ fusermount --v
fusermount3 version: 3.14.0

As an additional note, I had rebooted after uninstalling fuse / installing fuse3.

Did you see it immediately fix things or did you do a re-installation?

[kelinger]

As soon as it was installed, the next omni up worked without any reboots. I didn't have to rerun any of the setup processes.

[robnester]

Strange. That didn't work here. I'll wipe this server and try again.

[kelinger]

Well, please note that I have not rolled out that change to the setup process so its still going to install the older version. However, on my pristine system, that didn't matter (once I upgraded it after the setup).

[robnester]

Process:

Fresh install of Debian 12.2 Installed Omni

removed fuse: $ sudo apt remove fuse

updated system: $sudo apt update

installed fuse3:

$ sudo apt install fuse3
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libfuse3-3
The following NEW packages will be installed:
  fuse3 libfuse3-3
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 124 kB of archives.
After this operation, 415 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://deb.debian.org/debian bookworm/main amd64 libfuse3-3 amd64 3.14.0-4 [88.1 kB]
Get:2 http://deb.debian.org/debian bookworm/main amd64 fuse3 amd64 3.14.0-4 [35.9 kB]
Fetched 124 kB in 0s (638 kB/s)  
Selecting previously unselected package libfuse3-3:amd64.
(Reading database ... 43733 files and directories currently installed.)
Preparing to unpack .../libfuse3-3_3.14.0-4_amd64.deb ...
Unpacking libfuse3-3:amd64 (3.14.0-4) ...
Selecting previously unselected package fuse3.
Preparing to unpack .../fuse3_3.14.0-4_amd64.deb ...
Unpacking fuse3 (3.14.0-4) ...
Setting up libfuse3-3:amd64 (3.14.0-4) ...
Setting up fuse3 (3.14.0-4) ...

Configuration file '/etc/fuse.conf'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** fuse.conf (Y/I/N/O/D/Z) [default=N] ? 
update-initramfs: deferring update (trigger activated)
Processing triggers for libc-bin (2.36-9+deb12u3) ...
Processing triggers for man-db (2.11.2-2) ...
Processing triggers for initramfs-tools (0.142) ...
update-initramfs: Generating /boot/initrd.img-6.1.0-13-amd64

Rebooted

Started Omni:

$ omni up

Still seeing the same issue:

On the server:

user@host:~/OmniStream/mnt$ ls
cloud
user@host:~/OmniStream/mnt$ ls cloud/
Downloads

On the omnimount container

root@omnimount:~# cd /mnt/
root@omnimount:/mnt# ls -alh
total 16K
drwxr-xr-x 6 omniuser omniuser 4.0K Oct 24 14:57 .
drwxr-xr-x 1 root     root     4.0K Oct 24 14:57 ..
drwxrwxrwx 1 omniuser omniuser    0 Oct 24 14:57 cloud
drwxrwxrwx 1 omniuser omniuser    0 Oct 24 14:57 gdrive
drwxr-xr-x 2 omniuser omniuser 4.0K Oct 24 14:57 unsynced
drwxr-xr-x 3 omniuser omniuser 4.0K Oct 24 14:57 uploadcache
root@omnimount:/mnt# ls -alh gdrive/
total 4.0K
drwxrwxrwx 1 omniuser omniuser    0 Oct 24 14:57 .
drwxr-xr-x 6 omniuser omniuser 4.0K Oct 24 14:57 ..
drwxrwxrwx 1 omniuser omniuser    0 Oct 24 14:57 Media
root@omnimount:/mnt# ls -alh cloud/
total 8.0K
drwxrwxrwx 1 omniuser omniuser    0 Oct 24 14:57 .
drwxr-xr-x 6 omniuser omniuser 4.0K Oct 24 14:57 ..
drwxr-xr-x 2 omniuser omniuser 4.0K Oct 24 14:57 Downloads
drwxrwxrwx 1 omniuser omniuser    0 Oct 24 14:57 Media

[meharrington90]

@kelinger @robnester Nogo on installing fuse3 either. Did a full omni clean, reboot, and still only listing the cloud folder.

[kelinger]

@meharrington90 @robnester - Can you try omni update and see if it works now?

Anyone else: please update fuse before trying this (see earlier discussion)

[robnester]

@meharrington90 @robnester - Can you try omni update and see if it works now?

Anyone else: please update fuse before trying this (see earlier discussion)

That works. I'd love to know what fixed it?

[meharrington90]

Nope, still nogo on the other folders. I wonder if using an older distro would be ideal. @robnester, what version of debian are you using?

Also, put the command you use to to verify it is mounting properly in the container.

[meharrington90]

Seems I am using Ubuntu 22.04.3 with FUSE 3.10. I'm going to attempt a Debian reinstall.

plexabyte@Plexabyte:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy
plexabyte@Plexabyte:~$ fusermount --v
fusermount3 version: 3.10.5

[meharrington90]

IT WORKED!! I gotta run the full restore, but the system is back online with a Debian reinstall and manually installing fuse3. Good work @kelinger!!!

I'd say the only thing that needs fixed next is the install script, changing fuse to fuse3.

[kelinger]

Right; I've already updated the installer but didn't want to push that out until we knew it, you know, worked 😄 !

I'm also going to put an updater in there so everyone else (assuming there's at least one person not in this thread who uses OmniStream) can get the Fuse upgrade.

[kelinger]

Updated installer with fuse3 as well as new upgrade process added. System will now check for upgrades (takes milliseconds unless critical component, like fuse, needs a forced upgrade) upon omni up and omni update processes.

@Misterobots - this was your post so let us know if you're working now. Feel free to reopen if not.

[Misterobots]

Wanted to comment on this and say THANK YOU! This set of steps resolved my issue. Thank you everyone for the team work and dedication.