search

Clon1998 / mobileraker

A Flutter mobile app for Klipper/Moonraker
Other
457 stars 50 forks source link

Large Bandwidth Usage From Multiple Unclosed WebRTC Streams #339

Open theokeeler opened 4 months ago

theokeeler commented 4 months ago

Bug Report

Description

Mobileraker now uses huge amounts of data in the background when a print is running; I burned through over 40GB of mobile data in about 5 hours of printing. This behaviour first started once ProgressNotifications for Android started working. I've tried

Steps to Reproduce

  1. Start a print on your printer
  2. On an android device, connect via mobile data (you'll need some sort of remote connection)
  3. Wait
  4. Look at your mobile data usage and/or receive an angry text from your cell provider.

Expected Behavior

Mobileraker shouldn't be using this much data in the background (or even in the foreground). Ideally, whatever data is being downloaded here wouldn't be when on a mobile connection.

Version Information

Debug Logs

Not attached, as this would result in publicly posting private information about my network and printer configs that is included in your logs.

Additional Context

I'm accessing my printer remotely via a Wireguard connection to my LAN; I've tried setting up a manual "remote connection", but all that seems to let me do is change the endpoint to connect to. I've also set include_snapshot to False in my config but the behaviour has not changed. I've observed this behaviour with both very large and relatively small gcode files; The large gcode files use more data, but not to the same order of magnitude of difference as the size of the files reflect (e.g. a 9MB gcode file being printed consumes 5GB in 20 minutes, but a 200MB gcode consumes 8.5GB).

I will be unable to perform further testing on this for quite a while, as this has exhausted my data plan.

Checklist

To help us diagnose the issue, please ensure you've completed the following steps:

Clon1998 commented 4 months ago

Hey, can you attach the log files of the companion? 40 GB is a shit ton and tbh. I currently have no idea what could cause that. The progress notifications are issued in 5% steps and have a size of around 1kb as the progress notification do not include a snapshot.

The only other thing I could think of is that one of the webcams did not stop when you closed the app/ put the app in background.

theokeeler commented 4 months ago

logs-20240303-164252.zip Mobileraker Companion logs attached.

In terms of webcams, this machine has one 1280x1024@30fps webrtc feed. Looking at this on a desktop via mainsail, my system reports between 2.8 and 3.8Mbps; That should only account for ~570MB/20 minutes, so I don't see how it could be the (only) culprit here (unless the webcam feed gets reopened and left open multiple times).

Clon1998 commented 4 months ago

logs-20240303-164252.zip Mobileraker Companion logs attached.

In terms of webcams, this machine has one 1280x1024@30fps webrtc feed. Looking at this on a desktop via mainsail, my system reports between 2.8 and 3.8Mbps; That should only account for ~570MB/20 minutes, so I don't see how it could be the (only) culprit here (unless the webcam feed gets reopened and left open multiple times).

That is true. The logs of the companion also do not contain anything that stands out. It sends a notifation every 2-3 mins so nothing that would cause 40GB of traffic.

Are you using the normal WebRtc or Go2Rtc?

theokeeler commented 4 months ago

This is (as far as I know) a normal WebRTC stream from camera-streamer.

theokeeler commented 2 months ago

I've done a little additional digging here - curiously, the excess data usage survives both the uninstallation of the mobileraker app on the phone and a subsequent reboot of the phone. My router can see several continuing connections, each approximately 2.2Mbps, across various ports that look like a reasonable range for webrtc between the phone's wireguard IP and the printer's. That certainly makes me lean towards thinking this is a result of multiple connections being opened and orphaned. If I kill these active connections at the router, the same number of connections re-establish themselves; this happens even with the phone disconnected from the network entirely, so something on the printer end of things is continuously re-establishing these connections and pushing data on them.

Clon1998 commented 2 months ago

Mhhh. That's actually really weired šŸ™ƒ There should be nothing going on if the app is removed šŸ˜…. Also I double checked the webcam Implementation and can confirm that the webcams are paused/closed if the app goes into the background.

theokeeler commented 2 months ago

Yeah, my hunch here is we're looking at the client app seeing a disconnect or interruption to the connection somewhere, and requesting a new stream from camera-streamer; The old streams persist, however, because they're all UDP and thus they don't know that the other side of the conversation is gone.

I'm not conversant enough in the implementation details of webrtc to say if it's expecting a NAK here that would kill the previous streams, but I'm going to have a dig through the mobileraker source here to see if I can spot any obvious cases where a disconnect is met with an immediate request for a new connection.

The other direction I can see to tackle this would be using a TCP connection for webrtc instead of UDP; I took a look at the traffic between another printer and my phone using OctoApp (which doesn't exhibit this bandwidth-leaking behaviour) and all of its traffic is TCP, which will intrinsically let the printer side know if a given connection is down (as the phone stops sending ACKs). My understanding is that webrtc supports both modes of operation.

theokeeler commented 2 months ago

Ah, the actual webrtc code isn't available in the public repos, so I can't look at how it's handled there.

In the meantime I've set my webcam to use an mjpeg-adaptive stream, which avoids this problem (though unfortunately that change appears to affect Mainsail as well).

Clon1998 commented 2 months ago

Ah, the actual webrtc code isn't available in the public repos, so I can't look at how it's handled there.

In the meantime I've set my webcam to use an mjpeg-adaptive stream, which avoids this problem (though unfortunately that change appears to affect Mainsail as well).

I can post snippets tomorrow. But it uses the flutter webrtc lib.

Regarding the cam/effect on mainsail.

You can also add a the cam twice šŸ˜‰

Clon1998 commented 2 months ago

Here is the code that manages the webrtc.

https://gist.github.com/Clon1998/d912327938cdeaafa50df0c7e8105b44

As you might notice, it just uses the classes of the webrtc package.

If you know how to enable TCP instead of UDP for webrtc please let me know šŸ˜‰

theokeeler commented 2 months ago

Nothing immediately jumps out at me as problematic in that snippet; Unfortunately the flutter-webrtc documentation is effectively just the demo repo, so it's a little difficult to check the implementation details. I did notice that the demo app makes sure to close individual tracks when cleaning up, but that may be only for a sending connection rather than a receiving connection.

AlexanderS commented 2 months ago

I have the same issue and I can replicate it by simply closing Mobileraker via the Android Application switcher and opening it again. Each time I do this I can see an increase of the bandwidth. This may also happen when the app is in the background and is killed by the OS due to memory reasons or background process limitations.

I do not know much about Android application architecture, but it seems that it is not possible to get a callback for this event and a Service may be needed for handling this events. Maybe it can be fixed at the camera-streamer side: https://github.com/ayufan/camera-streamer/issues/123 and https://github.com/ayufan/camera-streamer/pull/91 seems to be related.

Clon1998 commented 2 months ago

I have the same issue and I can replicate it by simply closing Mobileraker via the Android Application switcher and opening it again. Each time I do this I can see an increase of the bandwidth. This may also happen when the app is in the background and is killed by the OS due to memory reasons or background process limitations.

I do not know much about Android application architecture, but it seems that it is not possible to get a callback for this event and a Service may be needed for handling this events. Maybe it can be fixed at the camera-streamer side: ayufan/camera-streamer#123 and ayufan/camera-streamer#91 seems to be related.

That is interesting and good to know. Afaik, I do everything I can to close all active connections, but if the camera streamer continues, there is not much I can do. However, did you try to reproduce this behavior via the browser of the phone rather than just the mobileraker app? If it behaves the same way, we know for sure that this is not related to my client's implementation.

Also, @theokeeler, can you confirm that you tracked it down to the WebRTC and, therefore, this issue is not related to Notifications at all? If that is the case, we might want to rename this issue.

theokeeler commented 2 months ago

Yep, it's definitely looking like webRTC from what I can see here; Most notably, if I don't open the app to look at the camera but leave it sending notifications, I don't see the same behaviour. Renamed the issue accordingly.

Clon1998 commented 2 months ago

Thanks! Also, after taking another look at it and debugging, I realized that the PeerConnection doesn't close when the app changes its AppLifecycleState to the background. Upon further investigation, I discovered that I never registered the callback to stop the WebRtc stream when the app goes into the background. I have now fixed that issue and will push a dev build for you to try out. Please let me know if you are using IOS or Android, so I can invite you to give it a try and collect some feedback. I'm hoping this update solves the problem.

theokeeler commented 2 months ago

I'm on Android.

AlexanderS commented 2 months ago

I'm on Android too.

Clon1998 commented 2 months ago

Alright, I will provide you guys an APK as soon as I am done with building it via github :)

Clon1998 commented 2 months ago

@AlexanderS, @theokeeler https://github.com/Clon1998/mobileraker/releases/tag/android2.6.15-webrtc

theokeeler commented 2 months ago

The problem persists with me for this build, though fewer streams are being created. I'm also seeing some WebRtcDisconnect events, and reconnections after those invariably produce another orphaned stream.

Clon1998 commented 2 months ago

The problem persists with me for this build, though fewer streams are being created. I'm also seeing some WebRtcDisconnect events, and reconnections after those invariably produce another orphaned stream.

Well I am out of ideas now. I am now 100% confident that my client implemention closed all connection if backgrounded.

theokeeler commented 2 months ago

After several days of testing this version, I've got a little more information about the behaviour with this patch in place:

IMO the patch is worth patching as-is even if it doesn't perfectly resolve things - it's harder to generate the extra streams without entering a failure state on this patch than on mainline.