search

kelinger / OmniStream

Deployment and management tools for an entire streaming platform that can reside on a server (local, remote, hosted, VPS) with media files stored on cloud services like Google Drive or Dropbox.
MIT License
33 stars 10 forks source link

Omnimount bouncing #11

Closed shadowsbane0 closed 2 years ago

shadowsbane0 commented 2 years ago

Hello again!

Finally finished pulling all of my data down. Quick off topic: NetData issue resolved itself some time back I just didn’t get a chance to mention it. Can’t say if it was a system update or something you did.

Back on topic. Omnimount is bouncing for some reason. It will get to startup complete and then issue a shutdown command.

My setup is now simplified. I have all of my content local on a raid’d enclosure. The enclosure is a mount point called “Media” in my user home directory. I ran the rclone setup again and created a new local remote. I chose it as my media directory and exited leaving it to default to /Media.

This isn’t ideal for me but after creating a local remote and choosing it. OmniStream assumes the user home directory as default. I gathered this after multiple attempts of mounting locations. Omni Menu setups defaults the the user home if you select to chose a media folder other than /Media (which I have anyway now in the user home). You are only allowed to select from a single directory in whatever your rclone remote directory happens to be. There is no option to select a subdirectory from any of the directories listed. Example. I had my mountpoint mounted at ~/OmniStream/mnt/mount but creating a local rclone remote forced me to ~/ and if I selected OmniStream to get to the ~/OmniStream/mnt/mount directory my Media mountpoint would be set as ~/OmniStream.

The main reason it isn’t ideal is because when Omni Backup script kicks off it tries to backup my Media mountpoint because it isn’t included in the ~/OmniStream/mnt exclusion. I’m not sure how long it would take to try and backup and compress 52TB of content but I decided to kill the process - not to mention my host system doesn’t have the space to hold that file anyway.

Omnimount is correctly creating the rclone remote, cloud, unsynched, and uploadcache folders but is removing them when it bounces. There is a mention of a “fusermount extra argument after mount point” error in the logs.

Here is log output:

Starting vnstat

Configuration:

MERGEMOUNT=cloud RCLONESERVICE=Junker RCLONEMOUNT=Junker UNSYNCED=unsynced UPLOADCACHE=uploadcache MEDIA=Media TURBOMAX=20 addgroup: The group omniuser' already exists. adduser: The useromniuser' already exists. Starting services { "jobid": 1 }

Startup complpete Received request to shutdown fusermount: extra arguments after the mountpoint rmdir: failed to remove '/mnt/uploadcache': Directory not empty rclone v1.58.0

mergerfs version: 2.33.5

TechPerplexed commented 2 years ago

Welcome back @shadowsbane0 😄 So, I'm probably completely misunderstanding what you are asking, but if I read you correctly, you are wanting to use the mounts for local storage, not so much Google/Dropbox/etc right?

So one idea could be to use the "unsynced" mount for that purpose. For example I have a test server at home with two hard drives - one small SSD and a large HDD drive with a number of movies/TV shows. I have simply mounted the HDD to ..../mnt/unsynced and that is what the test server uses for its content - not the stuff I have in Google. In fact it doesn't touch Google at all.

Again, I might misunderstand what you're saying, if so, please eloborate 😆

shadowsbane0 commented 2 years ago

No @kelinger, you understood correctly. I was trying to figure the easiest way to get a local mount available within the Omni environment as the /Media and /cloud mount my containers would have access to my content. Mostly to not have to adjust my current content structure within all of my containers.

I thought I would be able to do it easiest by just creating a rclone “local remote” (currently option 24) but ran into the complications mentioned with omnimount.

On Apr 27, 2022, at 9:00 PM, Tech Perplexed @.***> wrote:



Welcome back @shadowsbane0https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fshadowsbane0&data=05%7C01%7C%7Ca52893fcf86145f2f34008da28b26c16%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637867044027479573%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CYjj4aRXY9pkHGAHHvaHFm2sTZAkpAGwsgFoTPrTnZQ%3D&reserved=0 😄 So, I'm probably completely misunderstanding what you are asking, but if I read you correctly, you are wanting to use the mounts for local storage, not so much Google/Dropbox/etc right?

So one idea could be to use the "unsynced" mount for that purpose. For example I have a test server at home with two hard drives - one small SSD and a large HDD drive with a number of movies/TV shows. I have simply mounted the HDD to ..../mnt/unsynced and that is what the test server uses for its content - not the stuff I have in Google. In fact it doesn't touch Google at all.

Again, I might misunderstand what you're saying, if so, please eloborate 😆

— Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkelinger%2FOmniStream%2Fissues%2F11%23issuecomment-1111624064&data=05%7C01%7C%7Ca52893fcf86145f2f34008da28b26c16%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637867044027479573%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=wxIsJvqbHdlcWXrfoT8w4qMGehexDej4WvGPDkfEIzY%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAGWGOGAUVW66YBEF275B7WDVHHPJBANCNFSM5UP3XMCQ&data=05%7C01%7C%7Ca52893fcf86145f2f34008da28b26c16%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637867044027479573%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hbVJqmF2VauBDcFio4ShyfZDTivIQ7hFLzvhNeEdl%2Bg%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

shadowsbane0 commented 2 years ago

@kelinger @TechPerplexed

I don’t think the issue is with the rclone “local remote”. I deleted my config and started created a new remote pointing to my GDrive. I’m still getting the “Fusermount: extra arguments after mountpoint” error and omnimount is in a boot loop.

shadowsbane0 commented 2 years ago

@kelinger @TechPerplexed

Boot loop fixed for GDrive config. I reinstalled rclone and ran setup-again script to recreate my rclone.conf. It now works with the GDrive settings. I tried to create a second “local” remote and selected it as my media location - Omnimount bootloops. It also bootloops if it is just the local remote option only - both with the Fusermount error mentioned previously. (Local remote is now storage option 19 in the rclone config). Gooby used to create a “local” remote on its own. I never saw what it was used for since I was only using GDrive at the time.

Can you expound on how you are setting up to use the unsynced directory? It seems to be created and deleted when omnimount is up and down. If I can easily go down that path it will likely save me some time.

TechPerplexed commented 2 years ago

Right, the localfiles in Gooby and unsynced in Omni are the same thing :)

I'm puzzled why yours would disappear. When I take omni down (unsurprisingly, with the command omni down) whatever I have stored in unsynced stays in unsynced. Be aware though that in order for your containers to see this mount, you need to manually create the Media folder yourself (if you named it differently, adjust accordingly).

In my case I have actually used it all along. I used to run in a lot of Google API bans in the early days of scanning etc. That would result in us not being able to watch anything that evening, so I placed a number of "essential watch items" in mnt/unsynced/Media/

As I said, if you mount your local storage to that folder it should remain up and your content shouldn't vanish. I even created a script in the plugins folder that checks for new items on the Google mount and keeps them up to date locally, but that's probably going one step further. Can't say we aren't flexible 😆

Not sure what's up with your bootloops - I can't say I have ever experienced anything like that... but then I don't experiment THAT much since Omni does everything I need ootb, haha.

kelinger commented 2 years ago

A couple notes and answers (in random order) I'd like to add:

I use a Synology locally and I have one of its shares mounted on the same server I use for OmniStream. This is not where I store my media so I'm just mounting it outside of OmniStream using Linux's nfs-mount capabilities. In other words, I'm NOT mounting into ~/OmniStream/mnt but rather somewhere in the /mnt off the root of the server. However, it is still technically available to OmniStream functions using the plugins. For example, I have a script in the "backup-end" plugin that takes the files recently created by today's backup process and copies them to my NAS (since this script is for me and me alone, there's no need to use variables in it and I just sync the directories from ~/OmniStream/mnt/cloud/Backups to /mnt/Synology/Backups. While this may not be your use case, it is an example of using a local mount without subjecting it to OmniStream's removal or being wiped out by a restore or down/up process.

There are four directories used by OmniMount. One for the Rclone mount (typically the cloud drive with direct access), one for unsynced (default name is "cache" and it is not currently implemented but used for overlaying files that you never want to have synced back to the cloud), one for files pending upload (default name is "UploadCache" and these are files that have YET to be uploaded to the cloud), and finally the "cloud" directory which is the overlay of the other 3 using MergerFS.

Since "cloud" and the "Rclone mount" are mount targets, they need to be empty directories for the mount to be successful. While yes, it is technically possible to have the mount process ignore directories that are not empty, this override could easily cause a lot of troubleshooting issues that way Linux handles mounts. For example, when a mount fails or unmounts expectedly, the empty folder is indistinguishable (to Linux apps) from a mounted empty directory. Thus, there is the potential for some files to be written into the local folder and NOT to the mount's source. At best, this could cause the drive to fill up over time and, more likely, make you think you're writing files to a remote when they're actually only local. To combat this behavior, I remove/clear those mount targets whenever OmniMount starts since the expectation is that the local target of those mounts is always an empty directory.

The other two (cache and UploadCache) are designed to remain local. While UploadCache should ultimately be empty after TurboSync runs, the fact is that it may not be empty if a sync hasn't finished or the power went out or there was a crash (system or OmniStream). This way, when OmniStream starts up again, those unsynced files will be acted upon with the next TurboSync.

Typically the unsynced directory contains files pending upload to the cloud (often this is media recently downloaded from apps like Transmission or SABnzbd). The one exception here is the use of this directory for downloads (downloads are specifically ignored from the TurboSync process but having them in the same mount as the unsynced-yet media allows "move" commands to quickly move a file from downloads into the media where having downloads elsewhere causes the move command to actually perform a copy+delete which is a process that is orders of magnitude longer).

After the TurboSync process uploads the files from the upload cache, the end result is a lot of empty directories and those are purged at the end of TurboSync.

shadowsbane0 commented 2 years ago

@kelinger @TechPerplexed

I went ahead and “solved” the problem by moving my mount points into the OmniStream /mnt directory (so they would be excluded from the backup cron) and manually added it to my .yaml volumes after moving them into the 800 block. Unless you feel like tracking down why the rclone “local remote” causes boot looping of omnimount you can close this one.

kelinger commented 2 years ago

That's just it... it HAS to be in the OmniStream/mnt if it is managed by OmniStream. When you pass the variables into the containers, the paths (used by OmniStream which is managed outside of Docker) are local to the machine, not the Docker containers. Thus, I have to strip the path from the directory when it is referenced inside Docker. In other words, you may path /this/is/where/I/store/my/media as the full path to $MEDIA but OmniMount strips off all but /media and adds /mnt to it so that it sees (and accesses) it in its own root /mnt/media.