search

kelinger / OmniStream

Deployment and management tools for an entire streaming platform that can reside on a server (local, remote, hosted, VPS) with media files stored on cloud services like Google Drive or Dropbox.
MIT License
30 stars 8 forks source link

Omnimount Container Error: "Connection Refused" #49

Open Misterobots opened 1 year ago

Misterobots commented 1 year ago

I have no idea what could be happening here. I've changed nothing and everything was working a day ago. Any insight or assistance is appreciated. image

elmerfreak commented 1 year ago

Same here, happen since yesterday, no reason

kelinger commented 1 year ago

First question: have either of you tried a full stop/start of OmniStream?

omni down
omni up
elmerfreak commented 1 year ago

Yes, and still the problem.

I was looking in this issue from rclone, change my DNS but still no work.

Screenshot 2023-03-14 at 23 17 03 Screenshot 2023-03-14 at 23 17 34
kelinger commented 1 year ago

Well, that may be a problem. It looks like it is using fusermount3 which is apparently a major upgrade (ie, new commands/options/parameters).

On your host, what happens when you type fusermount -V (that's a captial V)?

Misterobots commented 1 year ago

Well, that may be a problem. It looks like it is using fusermount3 which is apparently a major upgrade (ie, new commands/options/parameters).

On your host, what happens when you type fusermount -V (that's a captial V)?

"fusermount version: 2.9.9"

What caused this? Is an update to the Omnimount container?

Thank you for the assistance.

elmerfreak commented 1 year ago

fusermount -V

fusermount version: 2.9.9

Misterobots commented 1 year ago

It does look like Rclone moved to Fuse3 in the last major update(v1.62.0). Does this coincide with the behavior?

@kelinger would the Omnimount container have picked this change up so recently?

kelinger commented 1 year ago

Definitely an issue with Rclone. See: https://github.com/rclone/rclone/issues/6844

OmniMount has a version of rclone baked into it. However, since I don't update the OmniMount container that frequently (there's not much need to) it does check for newer versions of Rclone when it is started up. It isn't looking down the beta path so the assumption was that things making it out of beta and into a brand new release have been pretty well vetted. Obviously, this one snuck through!

The reason it doesn't affect everyone is that, I'm guessing, you both performed a restart of at least OmniMount roughly the same day while I have left mine up and running for a while now.

It looks like he pushed a fix out today. I'm tempted to take down my (otherwise working) system and see if the newest release of rclone fixes it. If not, I can build a new OmniMount that either doesn't upgrade or allows you to override the upgrade. I'd rather not eliminate it, though, because this is how I can insure that things stay reasonably functional and updated in case I can't create a new build immediately after new releases.

elmerfreak commented 1 year ago

Thanks @kelinger for your help.

I installed already the rclone beta version pushed today but still no working, also downgrade rclone and no luck, install fuse3 no luck again. :(

kelinger commented 1 year ago

Right; the problem is that you probably installed it on your host, but my container uses the version installed there and that's not as easy for you to update.

I just took down my system and it's staying down so... yeah... this is personal now. I should have a fixed version of OmniMount in the next few hours.

elmerfreak commented 1 year ago

Oh God!! at least we are in the same boat now 🤣

kelinger commented 1 year ago

Fix: omni update

OmniMount has been recompiled to use Fuse3 and the latest Rclone build. The rclone beta with the patch wasn't released yet (as of this writing) so I just updated the container to use Fuse3 and it tested successfully.

To verify you're using the latest OmniMount: docker exec -ti omnimount fusermount -V The response should be something like: fusermount3 version: 3.10.3

If the "omni update" command doesn't work for you, you may try omni clean followed by omni up.

Misterobots commented 1 year ago

Confirmed working and everything back up on my server. Thank you @kelinger for the quick attention and speedy resolution!

elmerfreak commented 1 year ago

It work for me too! thanks @kelinger

Now I'm still having ban from Google apis 👎 any idea how deal with it? I don't know if open other issue 😞

I was looking in this issue from rclone, change my DNS but still no work.

TechPerplexed commented 1 year ago

I used to run into a lot of bans too, especially when I scanned a lot of content at once. I think the only solution is to ride it out - it'll probably be lifted in a few hours.

kelinger commented 1 year ago

If you run tail $LOGS/rclone.log you should see the last ~10 rclone errors. Note that this might be filled with the "fusermount" errors mentioned in this issue and if that's the last one, then you're probably not banned anymore (or not running anything).

But, if you see file operations and something like "permission denied" or "unauthorized" it likely means you're still banned. The longest I've been banned has been 24 hours (this assumes I stopped doing whatever caused it).

You could also run the command api which should show the amount of data that rclone has copied to Google in the past 24 hours (if you are accessing Google with the same user ID elsewhere, that is not included in this tabulation). If it's high, this can occur when someone refreshes a movie collection or long TV series. I've had it happen when someone reposted the complete James Bond collection and another time when they refreshed a TV series with over 200 episodes across 10+ seasons.

elmerfreak commented 1 year ago

The message is: We're sorry... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.

And its weird, the api command show something crazy Since 2023-03-15 05:29:10 Gigabytes: 49410.073131115176 Files: 23673

Its imposible transfer 49T in one day with Google 🤔

kelinger commented 1 year ago

This indicates that your system is still trying to upload files to Google. The API counter is a "guess" based on the data sent to OmniMount and if the uploads are blocked by Google and requeued, then they're just going to keep adding up in the API log.

I'd recommend you shutdown OmniStream for a few hours (or up to a day). Alternatively, you could take OmniStream down (omni down), take comment out turbosync from the cron on the host, and then bring it all back up (omni up). OmniStream will still download files and give you access to your shows/movies (assuming you're not blocked from reading, either) but won't rerun the uploading process until you re-enable it in cron.

elmerfreak commented 1 year ago

Done, hope it help

Screenshot 2023-03-16 at 19 08 27
thunderstorm654 commented 1 year ago

The message is: We're sorry... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.

And its weird, the api command show something crazy Since 2023-03-15 05:29:10 Gigabytes: 49410.073131115176 Files: 23673

Its imposible transfer 49T in one day with Google 🤔

I have the same issue with OmniStream and rclone right now when I checked the rclone logs. Nothing will play at the moment and the logs are full of these automated queries errors I've never seen before. Did shutting down your mount for a while sort it out?

Confirmed I am running latest version fusermount3 3.10.3.

Thanks!

Edit: I don't use OmniStream to download anything and there's no pending uploads.. (!) However I just tried that "api" command it and says slightly alarming numbers:

Since 2023-03-16 14:27:11 Gigabytes: 3572.4815842416137 Files: 433

Ermmm.. what's going on?! 😅

elmerfreak commented 1 year ago

Well, I disabled the cron for about 48h, enabled again but still the same, poorly it upload the backup files 😞

Maybe is the end of GDrive as it is.

Hope @kelinger has new ideas and came to rescue 😃

thunderstorm654 commented 1 year ago

Well, I disabled the cron for about 48h, enabled again but still the same, poorly it upload the backup files 😞

Maybe is the end of GDrive as it is.

Hope @kelinger has new ideas and came to rescue 😃

What hosting provider are you using? Is it Hetzner by any chance? I found that a lot of people are getting this automated queries error on their servers on and off the past few months, particularly bad the past week or so. Apparently it looks like Google may be throttling/banning Hetzner IP ranges as there is excessive traffic from them.

People have tried some workarounds like changing DNS or adding things to hosts file but these don't always work for long. Mine is working again now for about a day, but I'm not convinced it'll just break again at any point. Might move provider to be honest. Pain!!! I hope it doesn't spread to other providers and gradually Google drive is unusable. I have a lot of content..!!

Edit: another suggestion is someone is using Wireguard on another VPS to route all their Google traffic through. Sounds like a pain to me to have to do this though and extra work. Check out the post here: https://forum.rclone.org/t/429-google-drive-errors-back-for-anyone-using-hetzner-even-with-cloudflare-or-google-dns-used/36671

Currently all I did is add the entries to etc/hosts mentioned in this reply: https://forum.rclone.org/t/429-google-drive-errors-back-for-anyone-using-hetzner-even-with-cloudflare-or-google-dns-used/36671/62?u=storm

And it's been working since. But sounds like mileage may vary and only guaranteed fix people found is the Wireguard routing thing with another VPS, if wanting to stay with Hetzner.

elmerfreak commented 1 year ago

Yeah, its Hetzner, Im trying with other provider and works great, as you said hope Google don't start a witch hunt banning every provider.

Currently all I did is add the entries to etc/hosts mentioned in this reply: https://forum.rclone.org/t/429-google-drive-errors-back-for-anyone-using-hetzner-even-with-cloudflare-or-google-dns-used/36671/62?u=storm

I did this as well but no work.

Plex works as the cron is disabled and no start evary 15 min.

kelinger commented 1 year ago

@Misterobots @elmerfreak did you ever get the Hetzner issue resolved?

elmerfreak commented 1 year ago

Hi @kelinger not really, Google block completely my IP, I did everything posting on rclone and reddit forums, the only way was making wireguard and tunnelin the google traffic, so I myself decide split my providers, one for Plex other for download flow, but It became mor expence.

So, think on that, OmniStream work smoth with other storage companies like Dropbox or Mega for example?

kelinger commented 1 year ago

@elmerfreak - Did this resolve for you with Google? I haven't experienced (or read about others experiencing) that with other providers.