UnchartedBull / OctoDash

OctoDash is a simple, but beautiful dashboard for OctoPrint.
https://unchartedbull.github.io/OctoDash/index.html
Apache License 2.0
1.02k stars 226 forks source link

Octodash stops reporting the print status after some time, looses connection to octoprint? #1887

Closed wseifert closed 3 years ago

wseifert commented 3 years ago

Describe the bug After some time of printing (time differs from print to print even if I print a model several times) Octodash stops reporting the progress and become stuck. If I want to cancel the print Octodash reports broken connection. This Issue is since 2.2.0, with older revisions it was never seen.

General Information:

UnchartedBull commented 3 years ago

What is being shown in the lower right hand corner as the status if the connection is lost?

wseifert commented 3 years ago

As this bug is not allways present as soon as I see him I will report what is show in lower right corner. Can take some time ...

wseifert commented 3 years ago

Did not last long to see the error again: octodash stopped on layer 1, print is now on layer 3, total print time calculated is ~ 7 hours. I attach a screenshot from octodash, only printing is show on the lower rhight hand corner. Btw, I have two printers running octodash, one Anycubic and one Prusa, on both I have seen the error. octodash

shamlian commented 3 years ago

I have a similar configuration (but with an Ender 3v2) and am experiencing the same issue. Sometimes, the screen does not update at all during a print, saying "operational" (this only happens if the print is started via the OctoPrint web interface); sometimes it stops updating during a print, as described above (this can happen during a print started by any method); sometimes, it is perfectly normal. It's unfortunate that both OctoDash and OctoPrint had a version update at about the same time; it's unclear to me which update caused the problem.

wseifert commented 3 years ago

If Octodash stops refreshing the screen I tried to restart Octodash service with sudo service getty@tty1 stop A message is displayed: Warning: Unit file of getty.service changed on disk, 'systemctl --system daemon-reload' recommended. I did sudo systemctl daemon-reload sudo service getty@tty1 start Octodash now works.

ManuelMcLure commented 3 years ago

I can report that this problem is happening to me as well. When this happens and the print is complete if I select the "Cancel" option on OctoDash I get a 409 "Conflict" error message and I'm stuck. Unfortunately since we cannot execute custom actions while OctoDash thinks a print is in progress, I can't use the [!STOPDASHBOARD] command to restart OctoDash so I need to go to another system to restart it. I think the systemctl error seen by @wseifert is a coincidence and not actually related to the loss of communication.

UnchartedBull commented 3 years ago

Thanks for all the info! Since the status doesn't change to "reconnecting ..." the socket stays connected. So it can't be due to a flaky internet connection or something. I'm going to do some investigation on this issue, but I haven't been able to replicate this on my machine yet. Are you using any other apps alongside OctoDash?

ManuelMcLure commented 3 years ago

I generally connect to OctoPrint via a browser on my PCs or through OctoPod on my iPhone and iPad. No other apps are used to connect. OctoDash is running on the same system as OctoPrint and is connected through localhost:5000. The system in question is an Intel Celeron J1900 mini-PC running fully updated Debian Buster (x64).

Crowlord commented 3 years ago

This is one of those unhelpful "I Have this too" comments. Dash stops updating. Print completes fine though.

browser.user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.214 Safari/537.36 connectivity.connection_check: 1.1.1.1:53 connectivity.connection_ok: true connectivity.enabled: true connectivity.online: true connectivity.resolution_check: octoprint.org connectivity.resolution_ok: true env.hardware.cores: 4 env.hardware.freq: 1200 env.hardware.ram: 915058688 env.os.bits: 32 env.os.id: linux env.os.platform: linux env.plugins.pi_support.model: Raspberry Pi 3 Model B Rev 1.2 env.plugins.pi_support.octopi_version: 0.17.0 env.plugins.pi_support.throttle_state: 0x0 env.python.pip: 20.2.3 env.python.version: 3.7.3 env.python.virtualenv: true octoprint.safe_mode: false octoprint.version: 1.6.1 systeminfo.generator: systemapi

wseifert commented 3 years ago

my system is a raspberry pi 4, 2021-03-04-raspios-buster-armhf-lite with all patches, OctoDash and OctoPrint 1.6.1, no other apps.

shamlian commented 3 years ago

I'm on a RPi 3B+ Rev 1.3 running the OctoPi image updated to 5.10.17-v7+. The only pieces of software running on the Pi are OctoPrint and OctoDash. I do have a handful of plugins installed other than the ones OctoDash recommends. I also use OctoPod to connect (as well as a web browser, and directly from Cura). I've attached the list, as well as some logs during prints during which this happened, if it's helpful. Also -- to @UnchartedBull -- thank you for such a beautiful and useful piece of software. octodash-config.zip octoprint-logs.zip

UnchartedBull commented 3 years ago

That's actually super helpful @shamlian! Can you remember when OctoDash stopped reporting the progress? Any chance it was 9:30ish?

And thanks for reading the docs! :)

UnchartedBull commented 3 years ago

I've created a test version: https://drive.google.com/file/d/12smcmEi_hBcXiDzgcJ1SzeDqGhTX8mkh/view?usp=sharing. If we're really lucky this straight-up just fixes the issues, if not please report whether OctoDash shows 'socket is dead' in the lower right hand part of the UI. Thanks! :)

jneilliii commented 3 years ago

will test and report back.

shamlian commented 3 years ago

Glad to help; not sure I can tell you for certain what time the reporting stopped, though. :-/ I'll also give the new build a shot.

Jack77777777 commented 3 years ago

Not sure if the below is the problem but for me it solved Octodash not updating print progress.

I had the Plugins Slicer Thumbnails and Cura Thumbnails installed and many times this issue. Deinstalled Cura Thumbnails one or two weeks ago because i don't use Cura anymore. Since then no more issues.

jneilliii commented 3 years ago

I'd have a hard time believing that either of my thumbnail plugins would cause issues with websocket communication @Jack77777777.

Jack77777777 commented 3 years ago

Like mentioned i don't know if this was the cause. I will report if the issues occur again.

jneilliii commented 3 years ago

So I have reproduced the issue with the test version and unfortunately it seems it does not report as socket is dead. In this specific case I sent a file from PrusaSlicer with the option to autostart the print and the print started fine, OctoDash showed a status of printing, but the print progress page was not displayed.

UnchartedBull commented 3 years ago

Ok that's very interesting. This means that OctoDash is still receiving messages via the socket, just not the one it expects. I guess we need another test version that logs every received message when no current status message has been send for like a minute or so.

The weird thing is, that I'm using the exact same workflow and never had this happen.

Crowlord commented 3 years ago

Just to confuse issues I have been using your test version for 4 days without a glitch!

On Wed, 16 Jun 2021 at 09:34, Timon G. @.***> wrote:

Ok that's very interesting. This means that OctoDash is still receiving messages via the socket, just not the one it expects. I guess we need another test version that logs every received message when no current status message has been send for like a minute or so.

The weird thing is, that I'm using the exact same workflow and never had this happen.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/UnchartedBull/OctoDash/issues/1887#issuecomment-862167336, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGJKPQCU7FPJS4P3BTPWVODTTBO23ANCNFSM4533R42Q .

UnchartedBull commented 3 years ago

Jack77777777 commented 3 years ago

Are you sure it can not be a plugin issue? Would it make sense that the people who have the issue report their installed plugins ?

jneilliii commented 3 years ago

I agree, in my case I do still have both thumbnail plugins and tplink smartplug. In my workflow printer is off typically and auto powers on when I send the gcode from prusa slicer. I do not have DLP plugin. I can get a more detailed list later, but I know both those have already been mentioned.

UnchartedBull commented 3 years ago

Sure I think any information would be useful from now on. I'm going to start (no issues on my side, never experienced this bug):

Jack77777777 commented 3 years ago

I had the issues as well. After deinstalling Cura Thumbnails and decreasing delay in TPLink Smartplug from 5 to 2 Sec. no more issues. image

Installed Plugins: Bed Visualizer Custom Control Editor Dashboard DisplayLayerProgress Plugin Filament Manager Firmware Updater Macro Octolapse OctoPod Plugin PrintJobHistory PrintTimeGenius Plugin Slicer Thumbnails Stateful Sidebar Top Temp TP-Link Smartplug UI Customizer WS281x LED Status

jneilliii commented 3 years ago

I need to try lowering mine more in TPLink. I started at 10 seconds, and dropped to 5, will try 2.

Backup Scheduler (0.1.0rc1) Camera Settings (0.2.0) Firmware Updater (1.11.0) Fullscreen Plugin (0.0.6) GCODE System Commands (1.0.1) Google Drive Backup (0.0.4rc2) GridSpace Plugin (0.1.7) MultiCam (0.2.8) Network Health Monitor (1.0.4) OctoDash Companion (0.0.6rc1) PrintTimeGenius Plugin (2.2.8) Slicer Thumbnails (1.0.1rc1) Tasmota (1.0.4rc4) Terminal Commands Extended (0.1.7rc3) Terminal Response (0.1.4) Top Temp (0.0.1.4) TP-Link Smartplug (1.0.1) UI Customizer (0.1.5.9)

poolrev commented 3 years ago

The same issue where Octodash stops reporting happens with my setup. It doesn't happen with every print, some work fine but I haven't been able to identify a "pattern". Listing my plugins in case it helps to understand the situation.

Plugins installed: Cancel Objects DisplayLayerProgress Plugin Dragon Order Enclosure Plugin Filament Manager Floating Navbar Fullscreen Plugin OctoDash Companion OctoPod Plugin Preheat Button PrintTimeGenius Plugin Slicer Thumbnails Tab Order Themeify TimeToFilament Plugin WebDAV Backup

wseifert commented 3 years ago

Since installing the debug version I printed only a few things but never saw the Issue.

My Plugins installed: DisplayLayerProgress PSU Control MultiCam DisplayProgress Prusa ETA override Automatic Shutdown PrettyGCode Prusa MMU2 Select Filament ipOnConnect Enclosure Plugin GPIO Shutdown Dashboard PrintTimeGenius Preheat Button Firmware Updater

Jack77777777 commented 3 years ago

If it is a plugin issue i would suspect PrintTimeGenius. Can the ones with issues report the installed version?

Mine is 2.2.8 and i deleted the compensation values in advanced settings two+ weeks ago. Perhaps someone can delete it as well and test.

dmitrimedina commented 3 years ago

today I begin a print via the octoprint web on my laptop, I have a raspberry pi 3b+ running octoprint and octodash and I notice that on the octoprint web everything its running normal with the print in this right moment but on the raspberry the octodash its only showing the home menu where its say files, filament and setting its not showing the actual print that its happening right now. but in the right corner on the bottom on the octodash its say printing

MarcWinNJ commented 3 years ago

Not sure if the below is the problem but for me it solved Octodash not updating print progress.

I had the Plugins Slicer Thumbnails and Cura Thumbnails installed and many times this issue. Deinstalled Cura Thumbnails one or two weeks ago because i don't use Cura anymore. Since then no more issues.

I removed the Cura Thumbnail plugin and it did not have an effect. I have not lost the sync but the layers reported are wrong. For example I printed an 87 layer print and it read 87 of 200 when completed. The previous print may have been 200 layers and it just didn't reset properly when the new job started.

MarcWinNJ commented 3 years ago

If it is a plugin issue i would suspect PrintTimeGenius. Can the ones with issues report the installed version?

Mine is 2.2.8 and i deleted the compensation values in advanced settings two+ weeks ago. Perhaps someone can delete it as well and test.

I deleted the plugin altogether. The issue of the screen not resetting once it completes is still there. I have not run a long enough print to see if the connection gets lot yet. The last print was one hour. Something is causing a partial or full communication failure between the two. In this case it doesn't send a Finished, reset display message r something like that.

UnchartedBull commented 3 years ago

@MarcWinNJ have you tried the linked dev build?

Has anyone else encountered this bug again, after installing the dev build except for jneilliii?

MarcWinNJ commented 3 years ago

@MarcWinNJ have you tried the linked dev build?

Has anyone else encountered this bug again, after installing the dev build except for jneilliii?

This really is a newbie thing to ask but can you post the terminal command for me to install the .deb file I downloaded for the dev build. The file name is: octodash_2.2.0_armv7l.deb located in my downloads folder (on a Mac). I usually install through the auto script. Found this" sudo dpkg -i octodash.deb but was nto sure how to ensure it pulled it from the right location and the filename syntax

shamlian commented 3 years ago

@MarcWinNJ have you tried the linked dev build? Has anyone else encountered this bug again, after installing the dev build except for jneilliii?

This really is a newbie thing to ask but can you post the terminal command for me to install the .deb file I downloaded for the dev build. The file name is: octodash_2.2.0_armv7l.deb located in my downloads folder (on a Mac). I usually install through the auto script. Found this" sudo dpkg -i octodash.deb but was nto sure how to ensure it pulled it from the right location and the filename syntax

Assuming you're using an OctoPi install to run Octoprint (if not, change octopi.local to your pi's hostname):

  1. Copy the file onto the pi (in the terminal on your Mac, type scp ~/Downloads/octodash_2.2.0_armv7l.deb pi@octopi.local:/home/pi )
  2. ssh in (ssh pi@octopi.local), and then
  3. in the directory with the .deb file (should be the one you start in), type sudo dpkg -i octodash_2.2.0_armv7l.deb
shamlian commented 3 years ago

@MarcWinNJ have you tried the linked dev build?

Has anyone else encountered this bug again, after installing the dev build except for jneilliii?

I've not noticed it since installing the dev build. That said, I've also been hanging around my printer a lot less, only checking in occasionally during builds.

As far as the people asking about various plugin versions, here are the non-bundled ones:

Creality-2x-temperature-reporting-fix (0.0.4) Creality Temperature (1.2.4) Cura Thumbnails (1.0.1) DisplayLayerProgress Plugin (1.26.0) Draggable Files (1.0.4) Enclosure Plugin (4.13.1) Exclude Region (0.3.0) Filament Manager (1.8.1) FileManager (0.1.6) GcodeEditor (0.2.12 OctoPod Plugin (0.3.0) Octoslack (2.2.0) Preheat Button (0.8.0) PrintTimeGenius Plugin (2.2.8) Slicer Thumbnails (1.0.0) Translate Model Plugin (0.2.0)

(this list was gathered using cat ~/.octoprint/logs/octoprint.log | grep "|" | grep "= /home/pi/" | sed '/bundled/d;s/ \=.*//' | sort | uniq )

wseifert commented 3 years ago

For me the dev build solved the issue, never seen again ...

MarcWinNJ commented 3 years ago
  1. scp ~/Downloads/octodash_2.2.0_armv7l.deb pi@octopi.local:/home/pi

am getting this error: Had to do a brand new clean install. Pi is up and running.

pi@octopi:~ $ scp ~/Downloads/octodash_2.2.0_armv7l.deb pi@octopi.local:/home/pi pi@octopi.local's password: /home/pi/Downloads/octodash_2.2.0_armv7l.deb: No such file or directory

shamlian commented 3 years ago
  1. scp ~/Downloads/octodash_2.2.0_armv7l.deb pi@octopi.local:/home/pi

am getting this error: Had to do a brand new clean install. Pi is up and running.

pi@octopi:~ $ scp ~/Downloads/octodash_2.2.0_armv7l.deb pi@octopi.local:/home/pi pi@octopi.local's password: /home/pi/Downloads/octodash_2.2.0_armv7l.deb: No such file or directory

That first command was to be run on your Mac, to transfer the .deb file you said you downloaded to your Downloads folder to the home folder on your Pi. If it's already there, you can proceed on the pi from the third command.

ManuelMcLure commented 3 years ago

I used my personal build with the fix (on x64) and did an 11 hour print yesterday with no issues seen.

jneilliii commented 3 years ago

I reinstalled the test version on my pi as I recently converted to USB card, and unfortunately seems to still exist for me. What seems to be the common factor when it does happen (doesn't happen all the time) is when I send a file from PrusaSlicer with TPLink/Printer powered off with the option to autostart. The printer powers on, and starts the print, main OD screen is shown with a state listed as "printing" in the bottom right corner. I use my custom action button to restart the getty service and when OD comes back up it instantly goes into the job progress screen.

UnchartedBull commented 3 years ago

That sounds like a different issue though (which should be fixable quite easily). But did you experience the hang-up during print with the beta recently?

I'm going to leave this open for another week or so and then merge that into main and release 2.2.1 with the fix included, unless someone reports the issue still exists. Thanks for all your info and patience with this!

MarcWinNJ commented 3 years ago

Appreciate all the help from everyone here with the newbie questions on getting the test version on to my pi. I ended up doing a new install on fresh card and setting up Octoprint from scratch, updated everything along the way. I did not install any plugins other than a few that came bundled with it. Specifically the list below. Ran a 3h30m print last night and the display was correct (normal) when I checked it this morning. Running it again now. Will do a long print over night tonight for a final test.

PLUGINS DisplayLayerProgress Filament Manager Firmware Check OctoDash Companion Preheat Button PrintTimeGenius Plugin Printer Dialogs Printer Notifications Slicer Thumbnails Virtual Printer

jneilliii commented 3 years ago

That sounds like a different issue though (which should be fixable quite easily). But did you experience the hang-up during print with the beta recently?

It's the same thing I was seeing before. I have not seen OctoDash go into this state outside of auto powering on the printer while auto starting the print while sending from PrusaSlicer, but that's my standard workflow, so unsure outside of that. I'm going to be away from the house until after the fourth and won't be able to do any additional testing to confirm that is the only time it happens to isolate, but was curious how is it possible to debug this on the Pi within OctoDash?

UnchartedBull commented 3 years ago

I think that the issue with the power-on occurs because OctoPrint is in the "printing" state before the printer is connected. So the websocket message PrintStarted is being send before a connection is established (thus is ignored by OctoDash). Once the connection established message is send there won't be another PrintStarted message, which leads to OctoDash staying in the main menu. Should be fairly easy to fix, I'll make sure to include this in the fix mentioned above.

UnchartedBull commented 3 years ago

fixed with https://github.com/UnchartedBull/OctoDash/commit/054f81ee7b98432d013a7810ae5291b889df5540

will create a separate problem for the one @jneilliii is having.