search

prusa3d / Prusa-Firmware-Buddy

Firmware for the Original Prusa MINI, Original Prusa MK4 and the Original Prusa XL 3D printers by Prusa Research.
Other
1.12k stars 219 forks source link

[BFW-5596] [BUG] <5T XL Printer stops during color prints without error> #3983

Open corywagner opened 4 months ago

corywagner commented 4 months ago

Please, before you create a new bug report, please make sure you searched in open and closed issues and couldn't find anything that matches.

Printer type - [Prusa XL 5 toolhead (upgraded from 2T]

Printer firmware version - [6.0]

Original or Custom firmware - [Original]

Optional upgrades - [None]

USB drive or USB/Octoprint - [USB] Please let us know if you print via USB flash drive or USB/Octoprint [USB]

Describe the bug

Prusa stops printing sometime during print. It only seems to happen with long, muti-color prints (I have a 5T XL). Stops randomly. Inducing power panic usually allows it to resume (for a while). There are no errors and the printer log does not indicate a failure. If I induce power panic, it will resume right where it left off. Sometimes it will complete the print, sometimes it will fail later.

Here is what I tried:

How to reproduce Unfortunately, I can't reproduce it at-will. It happens during long color prints. It does not see to happen with single color.

Expected behavior I expect it to complete the print, or at least throw some error so we can troubleshoot.

G-code Tried to attach bgcode, but would not let me.

Crash dump file Please send the crash dump file to Prusa by emailing it to reports@prusa3d.com. Sharing this file is important and helps us investigate the bug. Do not share the file publicly, as the crash dump contains a raw snapshot of the printer's memory and may include unencrypted sensitive information.

Unfortunately, there is no crash dump file to send because the printer does not know it crashedVideo

Please attach a video. It usually helps to solve the problem. No video because I am never in the room when iit fails. Pic attached for what it is worth. PXL_20240426_132115441 MP PXL_20240501_143907200

Mr3Deee commented 4 months ago

I have seen the same problem - BUT I didn't even need to do a multi color print for this. I had a simple PETG single color print which casued this. It happened many times (about every 30 minutes). In the end it wasn't even able to continue the print and it stopped.

My solution was to disable PrusaConnect. My feeling is/was that this was due to concurrency where multiple clients tried to access the same USB stick at the same time.

Someone else was raising the comment that there might be hardware problems. As I was assmbling the 3D printer, I know that the cabeling was a standard ribbon cable and as such it might be suceptibe to EMV. The stepper motors might therefore be feeding stray signals into the wire which will cause the USB bus to become unusable (and time out) for a short time. However I hope that this is not it. One thing to try is to use 6.0.0, turn Prusa connect and Prusa Link on buit the new "silent modes" off and see if it still happens. IF this would be USB stick related, it MIGHT be a good idea to use an empty and / or a slow (or fast) stick. If there is enough power coming out of the port you might want to try using an external SSD drive(?) If it gets even worse (faster errors) it might be a good indication for HW problems.

GeasyCreasy commented 4 months ago

I have the same issue. There is a large thread on Prusa forums that describes the same problem. (https://forum.prusa3d.com/forum/original-prusa-xl-tool-changer-hardware-firmware-and-software-help/usb-error-mid-print/paged/4/)

The issue is maddening! Upvote to get this problem addressed quickly please!

ddrl46 commented 4 months ago

I have also been running into this issue very consistently while attempting to print a fairly short single-material print. Every time the printer halts with a "USB drive or file error, reconnect the drive". Sometimes the print is recoverable from this point by reconnecting the USB stick, however quite often the USB stick is not recognized anymore until the printer is reset via the front panel.

IMG_2138

Attempting to disable phase-stepping as well as prusa connect / networking all together did not resolve the issue. I've tried multiple types of USB stick, all with the same result.

As a last ditch effort I downgraded to 6.0.0-alpha, which is was previously running without issue, and after doing so I was able to print the same g-code I was previously having issues with multiple times without any issue. Attached is the g-code in case it helps.

Quarter Baseplate HBM Ladekast v1_0.4n_0.2mm_PETG_XLIS_1h56m.zip

tsamisa commented 4 months ago

I have the same issue. I downgraded to 5.1.2 to be able to print

Blizzke commented 3 months ago

I have the same issue with 6.0

4 color / 4 hour print stopped at 33% (halfway in a layer) without any message or error. Knob was still active so I "paused". Status went to "pausing" right away, but it took almost 2 minutes before the printer actually responded by lowering the build-plate and parking the head.

Had a pretty huge blob of filament which I cut away as best as possible and then resumed. The print finished without any further issues.

(Print was via connect, if that makes a difference)

goodridm commented 3 months ago

I had this happen the other day at the 48% point of a 3 day print. The printer just stopped at what seemed to be the end of a layer before color change. I noticed the nozzle temp was dropping and Prusa Connect said it was cooling to 70C from 250C. I set it back to 250C and it seemed to heat back up, but no printing resumed. I pressed Pause on the printer and it just seemed to stick at "Pausing...". I couldn't resume from Prusa Connect. The display responded via sound to my clicking, dial moving, but nothing happened. I pulled the plug and plugged back in to induce a power panic. The print job resumed! However, I ended up with a slight layer shift.

The model was 20MB 3mf file, 11.83x11.97x12.07, 1722592 facets, two colors, 1499 tool changes

madnotangry commented 3 months ago

I have had this happen on several different prints. The XL would pause during a print with toolhead changes or single color prints alike, sit for about a minute or two, and then park itself without an error or notification at all. It would let me resume it from the LCD, but I couldn't connect to the printer with Prusa Link or Prusa Connect unless I cycled power to the printer. I tried to alleviate the problem with a Hard Reset, reflashed the 6.0.0 firmware, and had the problem continue.

I downgraded to the 6 alpha firmware and have had no problems whatsoever.

TheShuttlesworth commented 3 months ago

Currently experiencing this issue. I have so far had two stoppages, within 10 minutes of each other, on a 4 hour total print. Multi-material (2), sent through Prusaslicer via Prusa Connect, running 6.0.0 firmware, Prusaslicer 2.7.4. It seems to happen on tool changes, right before it picks up to go to the back. I have cut power and it has started back. Trying the pause, pull USB stick, etc. has not worked. Firmware 6.0.1 just released, but I don't want to lose this print. I will report back if it comes up again in 6.0.1.

Edit: I did forget one detail. I have Variable Layer Height set for this print. I don't think that would cause the issue, but I wanted to include all the detail I could.

Torx999 commented 3 months ago

Have the same issue with 6.0.1

XBurgy commented 3 months ago

Hello, I have the same problem, specifically a USB drive reading error during printing on the Prusa XL 5T. The issues started after updating to FW 6.0 (before this update, the error never occurred!) and they persist even after updating the FW to version 6.0.1. Sometimes it is possible to resume printing using the continue option, but sometimes the print is completely lost. Based on personal experience, I would therefore guess that there is a bug in the printer's firmware since version 6.0.

Torx999 commented 3 months ago

Downgrading to 6.0.0 solved this problem for me. After upgrade to 6.0.1 the error prevent me to print anything. After downgrade I didn't seen this error again.

corywagner commented 3 months ago

This is a consistant and ruinous issue, please address it. If you have any doubts, please check this reddit thread: https://forum.prusa3d.com/forum/original-prusa-xl-tool-changer-hardware-firmware-and-software-help/usb-error-mid-print/paged/5/#post-712173

nhustak commented 3 months ago

I'm on an 5 head as well and see this issue on non-multi-material prints as well.

Mr3Deee commented 3 months ago

The issue is still present. For me the rate can be reduced by clearing the USB drive and turn off PrusaConnect. However - even then it is still happening. (Also 5 head)

I would still think that it must be a polling issue where PrusaConnect, PrusaLink and the printer itself are using the drive at the same time and running into a timeout. (It could be maybe that due to cabelling a higher error rate would occur, fostering the problem)

So my guess for a fix would be: Increase fault tolerance for read timeouts by increase timeout values. Additionally it might help to decrease the priority of PrusaLink/PrisaConnect on the USB bus. Even better would be to download the entire G-code blobb before the printing starts. I assume that there must be enough memory to do so.

Mr3Deee commented 3 months ago

Okay - I am now in a totally broken state. No matter what I do (clean up the USB drive, use a new one with the same files, ..) - it claims that it cannot read the USB drive. Great. 10hrs print time, ~300g filament behind me and it's simply broken. 

Then: Press reset, and ... tatataa, USB drive is back again.

Sorry - but this needs to get fixed. I will see that I can go back to 6.0.0 alpha. I hope that that gets me into a working state again, but honestly, this HAS to be fixed.

corywagner commented 3 months ago

Beware, I went back to 6.0 Alpha and it didn't fix the problem.

On Tue, May 28, 2024 at 9:20 PM Mr3Deee @.***> wrote:

Okay - I am now in a totally broken state. No matter what I do (clean up the USB drive, use a new one with the same files, ..) - it claims that it cannot read the USB drive. Great. 10hrs print time, ~300g filament behind me and it's simply broken.

Sorry - but this SW bug needs to be fixed. I will see that I can go back to 6.0.0. alpha. But this is really sad.

— Reply to this email directly, view it on GitHub https://github.com/prusa3d/Prusa-Firmware-Buddy/issues/3983#issuecomment-2136484402, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARUJRWXX6NSRMROWEDPO2M3ZEVJSHAVCNFSM6AAAAABHHZTWGKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZWGQ4DINBQGI . You are receiving this because you authored the thread.Message ID: @.***>

Mr3Deee commented 3 months ago

Okay. Thanks for letting me know! I did think a bit more about the problem and here are some more ideas for whom ever fixes this bug:

  1. I did yesterday try to connect the printer several times from a very spotty network connection from the beach. I would think that this might have triggered a remote action (reading a file, making the printer wait, etc.) which was then never released and this caused then in turn a stall (e.g. not closing a file, hence blocking the device in some way)
  2. It was said that turning off and back on the printer would make the printer to continue printing. As the printer is going in that case through a full power / reboot / reset cycle, the only way to explain that the printing continues is that the server is triggering that. This has the smell that the server is actually interrupting the printer session and also able to resume it as the printer has lost all internal state at that time. So - the remote operation MUST be involved in all of this.
  3. It is somehow conceivable that an underlying library / kernel change around I/O / Mutex / event handling is triggering this.

I am not sure if there is anything I could supply from my side which allows you to analyze this further - but please let me / us know if there is anything we can help with! (And yes, I am also a programmer, I do know what I am talking about even though I didn't program 3d printers, but I have done low level, OS level, server side, ... programming).

sauer-m commented 3 months ago

Same here! DualHead- and SingleHead-Prints on 6.0.1. Please get this fixed!

danopernis commented 3 months ago

Hi! We are already aware of the issue, however we are still unable to track it down :disappointed:

corywagner commented 3 months ago

What can we do to help? I get the phenomenon at least once a day.

CZDanol commented 3 months ago

I've possibly found a reproducer now, so I'm trying to diagnose and fix. We might contact you later when I manage to fix it to confirm that it is indeed the issue you were having :)

Teque5 commented 3 months ago

Same issue firmware v6.0.1 PrusaXL w/5T print; screen didn't show USB icon but could detect when detached (showed no usb error).

Was able to save print via "power panic". Will downgrade to v5.x in the meantime.

peterjhl commented 3 months ago

Struggling with this problem, right now I'm trashing 2 out of 3 prints. Tried everything! Pretty frustrated right now.

TheShuttlesworth commented 3 months ago

Update from my post above. I ended up updating to 6.0.1 firmware. Also, I noticed that my problem started after I connected a co-worker to the printer via PrusaConnect. I looked into this because I lost my connection to the printer. So I set up the "Teams" on PrusaConnect per the forums recommendations. After I did this, I reconnected to the printer through the team on PrusaConnect and I have not had an issue since. Multiple prints of various times, materials, heads, etc.

This may not help anyone, but I wanted to share. I do see that there is another firmware update (6.0.2) and I plan on updating within the next week since I am stable.

tutulino commented 2 months ago

Same problem with XL 5T after update to 6.0.0, usb file or drive error message at mid print

Juan-VV commented 2 months ago

This is driving me crazy, I have wasted about 200 hours of print time and many kgs of filament on this, and it making me consider not using the printer any more until there is a solution. Contacting Prusa support tomorrow.

Team at Prusa - Unsure how to help but please let me know if I can do anything to help... test a beta fix or something.

Of course this happens randomly... finished 2 long prints, then today it happened again at 85% of a 12 hour print. I could not recover.

5T although most of the time only printing with 2 heads.

I cannot tell when the problem started, it was about 2 months ago but one month ago gotten worse. Tried many random suggestions online...

When the failure happens most often I cannot recover. Sometimes, unplugging and plugging does the trick. Occasionally, unplugging printer and restarting gets the print to start but this is not always the case.

Now I am waiting for some ferrite coils to put around LCD screen cable on the back, apparently that helped someone in facebook group...

peterjhl commented 2 months ago

I have put a ferrite choke on the LCD cable, I suspect that it is "noise" that disturbs the signal. Have been printing for 7 days now and all prints have gone without problems. Just to check if it's luck and it has no effect anyway, does anyone else want to try the same "trick"? 2024-06-11 16 07 48

corywagner commented 2 months ago

I'll try anything to get this to work. Do you have a link to the choke? Also, what are we seeing in the second picture?

Blizzke commented 2 months ago

I'll try anything to get this to work. Do you have a link to the choke? Also, what are we seeing in the second picture?

Given that it is earlier timestamp wise I suspect that would be the "before" :-) I'm sure I have some of these around somewhere, time to check the junk drawer.

peterjhl commented 2 months ago

I'll try anything to get this to work. Do you have a link to the choke? Also, what are we seeing in the second picture?

I deleted picture, I posted the wrong picture, forgot to delete it again, sorry, but it was a picture before fitting the ferrite choke. It is similar to this one. https://uk.rs-online.com/web/p/ferrite-sleeves/7740641P?gb=s

Mr3Deee commented 1 month ago

Just checking in here:

For myself - as said earlier, disabling PrusaConnect and clearing everything off the USB stick did do the trick - for now. But I'd like to activate PrusaConnect again (and I do not like the idea of constantly scrubbing the USB drive).

tsamisa commented 1 month ago

Just checking in here:

* Did anyone try yet the 6.0.3 or 6.0.4 version to see if the problem went away?

* Or did the ferrite sleeve do the trick?

For myself - as said earlier, disabling PrusaConnect and clearing everything off the USB stick did do the trick - for now. But I'd like to activate PrusaConnect again (and I do not like the idea of constantly scrubbing the USB drive).

i asked in the forum under the announcement for the new firmware if it solves the issue but again i got no response. I also ask them every time a new fw 6. version comes to at least tell us where the problem lies , so as not to go around installing ferrite chockes, fans etc with no reason but that Jan Krativcill or something that announces the updates doesn't seem to bother to answer.

TheShuttlesworth commented 1 month ago

Just checking in here:

* Did anyone try yet the 6.0.3 or 6.0.4 version to see if the problem went away?

* Or did the ferrite sleeve do the trick?

For myself - as said earlier, disabling PrusaConnect and clearing everything off the USB stick did do the trick - for now. But I'd like to activate PrusaConnect again (and I do not like the idea of constantly scrubbing the USB drive).

i asked in the forum under the announcement for the new firmware if it solves the issue but again i got no response. I also ask them every time a new fw 6. version comes to at least tell us where the problem lies , so as not to go around installing ferrite chockes, fans etc with no reason but that Jan Krativcill or something that announces the updates doesn't seem to bother to answer.

I had the same results as @Mr3Deee . Once I disconnected PrusaConnect (after setting up a secondary operator account) and reconnected to the machine with a "Team", everything worked and has worked since. I am currently running the latest firmware with no issues.

I don't have a lot of experience with this platform, but I believe that setting up a PrusaConnect connection and disrupting it by changing settings caused the issue. You could try scrubbing the printer of all external connections to PrusaConnect or other wireless systems, reconnect and start from there. That's essentially what I did.

Also, I have seen some hesitant to update to the latest firmware. It has had no issues for me doing so and if you did and have issues its not too much trouble to revert back. Just update and see what happens.

corywagner commented 1 month ago

I installed the ferrit sleeve and thought it solved the problem. I ran the printer non-stop for 5 days without an issue. I was ready to declare victory, but on day 6, it stopped with the same behavior as before. I then induced a power panic and it resumed for about 2 hours and then stopped again. I installed the newest firmware, but that made no difference. I am now exploreing the theory that this is resulting from overheating on the buddy-board. I am printing a mode to attach a fan to the buddy board enclosure. More to follow.

corywagner commented 1 month ago

Sorry to be dense TheShuttlesworth, but what do you mean by "Once I disconnected PrusaConnect (after setting up a secondary operator account) and reconnected to the machine with a "Team", everything worked and has worked since. I am currently running the latest firmware with no issues."?

tsamisa commented 1 month ago

I may sound i bit ... st pid but for me the lingering question is, if it is an overheat, noise , etc problem why it doesnt occurs with a 5. fw?

TheShuttlesworth commented 1 month ago

Sorry to be dense TheShuttlesworth, but what do you mean by "Once I disconnected PrusaConnect (after setting up a secondary operator account) and reconnected to the machine with a "Team", everything worked and has worked since. I am currently running the latest firmware with no issues."?

@corywagner Nothing dense in your question. This is going to be long, but here is a breakdown of what I did that, I THINK, caused the problem and then resolved it.

A little background. This printer is for my Engineering department at our company. After I set things up normally per the instructions and was printing off the USB drive, I connected it to our network and started using PrusaConnect to send the prints over. After I got that set up and working with no problems, I had a co-worker who wanted to set himself up to print on the same printer. I ran through everything to set him up, but in the process I lost my connection to the machine on PrusaConnect. To my understanding of the normal connection type, you can only have one person connected at a time. This is when my issues started. I reverted to using the USB because I did not have time to troubleshoot the connection issue, and it worked until I had the stopping issue we have all seen. Once that started I hopped on this chain and started testing. The more I looked into it from a multi-person access point of view, I found that PrusaConnect has an option that came out in one of the updates to allow a group of people access to the same printer through their "User Team". I went through the process of re-connecting the printer to PrusaConnect via this User Team and invited the co-worker to that. Once that was completed, I have had no issues with the stoppages. I almost solely upload prints through PrusaConnect and they have been multi-material prints on various combinations of heads. I have even updated the firmware to the latest without any issues.

I hope this gives enough detail.

corywagner commented 1 month ago

Shuttlesworth, thank you for the explanation. Since I don't work in a multi-user environment, I'm wondering if this is pertanat to my situation. Nevertheless, it does not hurt to try. I'll do anything to get this thing working reliably. Thanks again.

Mr3Deee commented 1 month ago

Thanks for all the feedback!

My thinking was that the constant polling of the USB drive from multiple "clients" (PrusaLink & PrusaConnect & Printer) + potential read error and repeats (due to cabling) - which might even be accelerated by the temperature of the printer - was causing this.

The problem with the PrusaConnect is that it is not only showing the progress, but also available jobs with images and so on. Depending on the browser and your local memory on your computer you might have then background activity from your computer to the server to the printer and the result would be constant polling - and maybe even re-loading of printer image assets (which will likely need to be computed from gcode, hence read).

As such the required throughput from the USB drive can get quite steep. And then there is the cabeling. Dependent on how the EMV between the cables adds interference and USB errors, the read process has even to be repeated (until a timeout is hit) - and on top comes the heat up of the USB stick which then also reduces the speed the stick can operate on (and who knows - maybe even the error rate goes up).

Alternatively this can all be totally wrong and it is really only a firmware issue.

Juan-VV commented 1 month ago

Hi all - to share my experience which might be anecdotal as it is a single user's. My problem was quite bad, couldn't complete 4 hour prints, most of the time couldn't recover, wasted A LOT of plastic. This was a month driving me crazy....

Notes:

Any questions let me know.... I am just.... surprised, how problem 'went away'? But happy about it, wishing the best to others struggling through this. 

m10d commented 1 month ago

It sure would be nice if prusa would add more detail about what root-cause analysis suggests is at fault here. From all the various forums (github, prusa forum, reddit) it sure seems this is EMI related due to the long unshielded ribbon cable housing usb D+/D-, SPI, and i2c. I don't have any experience with USB, but spi and i2c are (much) more commonly employed on one board or board-to-board with very short cables only. Let alone running adjacent to motors.

Prusa, would you please consider:

TheShuttlesworth commented 3 weeks ago

Update for everyone. I finally had the failure come back last night. I left a 2 extruder print (I have a 5T printer) go over night and came in this morning to the same error we have all seen. The print stopped at 82% and the total print time was around 24 hours.

After finding it in this state I did the following.

In summary, I double down on my stance that this is a PrusaConnect or PrusaLink issue. For clarity, I have not modified the printer in any way. Only updating with fully released firmware when Prusa makes it available. My theories are either they are; polling too often to the device for updates on progress or status, there is an interrupt on the processor to make this communication and this is enough of a hiccup for the machine to freeze, or some other process related to the communication to PrusaConnect/PrusaLink.

Hope this helps move things along so we can no longer have this issue with our prints.

EDIT - Print finished without additional issues after turning off PrusaConnect

corywagner commented 3 weeks ago

I thought I solved it by putting chokes on both end of the USB cable, but alas, no. I did not get a mysterious stop for couple of days . Just as I got my hopes up, the problem came back...then it came back again, and again all within 3 hours. There is no pattern that I can discern. Luckily, a power cycel restarts the print from where I left off (thank you power panic), so I can't say I'm wasting material, but I am losing a whole lot of time. I should not have to watch my printer every minute in case it stops. I wish they'd get a fix for this!

thotland commented 1 week ago

I installed chokes, reverted to older firmware (5.1.2) and it still pauses for no reason or sometimes on USB coms fault. Upgraded today to 6.1.2 and received the same USB error 30 minutes into a 24 hour print. This is ridiculous! And I'm ready to try whatever it takes to add t the community here to solve this problem.

I run Prusa XL 5-head. I'll add that I usually slice my prints with a high percentage gyroid infill (20-40% depending on application). I know that can lead to large gcode files and hence, "extra" communication needs.

CZDanol commented 1 week ago

@thotland on the 6.1.2, does it allow you to resume standardly without the need to do power panic?

TheShuttlesworth commented 1 week ago

@thotland on the 6.1.2, does it allow you to resume standardly without the need to do power panic?

@CZDanol In my experience the pause resume cycle gets hung in a please wait screen after you select pause.

Droneracer101 commented 1 week ago

I had this issue and hoped that upgrading to the new 6.1.2 firmware but it is continuing to happen. Just did. If I power cycle it it does resume. Pausing the print does not work. It never completes the pause. I will try disabling PrusaConnect on my next long print.

CZDanol commented 1 week ago

@TheShuttlesworth that's a different issue then and not the USB one - USB error should actually result in the print getting paused.

Droneracer101 commented 1 week ago

@TheShuttlesworth that's a different issue then and not the USB one - USB error should actually result in the print getting paused.

Sorry I worded this poorly. The print does pause, it stops printing and it just sits there with no error on the screen. What I meant was I try choosing pause on the screen and as other commented, it doesn't seems to do anything or ever give me the option to resume. Power cycling the printer does force it to resume. So for now I have Prusa Link/Connect disabled on the printer and have the printer connected to a Alexa manageable surge strip to make it easier to power cycle. I'm still on my first print after trying this. but 13 hours in and it hasn't stopped yet. Way to early to call though.

Torx999 commented 5 days ago

Just printed (XL 5 heads)3 days well with 6.1.2 some tiny jobs - 3-4 hours and some bigger - 17-20 hours - no problems. For now I started 22 hours print I've printed before without problems - after 1 hour I've got USB error. If I try to continue - It came again and again. I took usb drive out, reformatted it, wrote bgcode file and tried to resume - failed with the same error usb drive or file error.