Closed haarp closed 1 year ago
Hi, Thank you for feedback. Is it possible to provide us the gcode? Kangaroo
In case it's maybe related -- I observe sth like this even without printing. If I turn the printer on, connect from octoprint, then issue a longer move command (like move Y 150mm, default speed) via console, then the printer makes a small initial move (lasting a fraction of a second), stops, and then immediately makes the rest of the move in one motion. I can actually hear it pause and resume - sounds like 2 separate moves in sequence. I think it affects all axes, not just Y. Weird to see a single gcode stutter like that.
@hoppke I've seen this bug for ages and it only happens sometimes. I don't know how to reproduce the issue, but what I can tell you is that it happens when "printing" over usb (octoprint not required).
I think the first time a saw it was in 3.1.0 with the introduction of linear advanceon the MK2. I can still see the issue on my custom 3.4.1.
If I remember correctly it starts happening after the printer calibration. Try to do a full printer reset. WITHOUT calibration try the long move. It will be executed in one movement.
Unfortunately I can not reproduce this anymore. Not sure what was going on. But as others have reported similar problems, I'll keep this open for now.
Ok, I managed to reproduce this over USB. It has nothing to do with LCD updates.
When many G-codes follow each other in very quick succession, the printer eventually starts pausing briefly (milliseconds). I highly suspect a buffer underrun occuring in the command buffer.
This can happen for example when printing Slic3r's Archimedean Cords Top/Bottom infill, which consists of very short G1 moves (why don't they use G2 for that?!?). If the print speed becomes high enough, USB seems incapable of sending commands fast enough, causing the printer to very briefly halt.
@DRracer @mkbel @haarp I have also experienced this issue from when I started 3d printing on the MK2 2 years ago. It is really bad when you try to reproduce it intentionally, especially at higher speeds. I think it needs more attention. While testing, I came to the following conclusion:
I did some changes to the fw for debugging and you can clearly see in the video when it triggers. It's just for enabling and disabling mbl through gcode. XYZ calibration has no effect on this. I tried world2machine_revert_to_uncorrected() as to disable it, but it didn't help.
@wavexx You recently worked on stuff related to planner/stepper handling. Did you also look into mbl/babystepping? If yes, do you know what might trigger this stuttering?
video: https://drive.google.com/file/d/1__AzCAGrlGmKkcs3NDHgnwSeKhxBksZM/view?usp=sharing
@leptun Incidentally, yes, at low speeds even. It popped up yesterday after doing the vase tests we discussed. Looking at the video is looks very similar to the behavior I've recorded here:
https://www.thregr.org/~wavexx/tmp/VID_20190720_190425_r.mp4
As soon as I noticed I tried to fiddle around, and by lowering the acceleration you can make it very obvious. I didn't think of mbl, it was definitely active. I recorded the output of M503 and had to leave, so I turned the machine off :(. I played the entire afternoon doing all sorts of tests, so it can be a multitude of things.
But I've actually seen the same behavior in stock marlin 1.1 and also there it's incredibly hard to reproduce.
Given how it plays with acceleration, I really suspect an old bug in the planner code. I'll have a look if I can reproduce it "easily" with what I have. I took some notes of the stuff I did..
@DRracer @mkbel @haarp @wavexx I looked more into this and I found where the problem starts. Previously I said moves over 35mm are affected. I found out that any move that is over 30 (x >30) are affected. And I kept looking keeping this in mind and I found this:
void mesh_plan_buffer_line(const float &x, const float &y, const float &z, const float &e, const float &feed_rate, const uint8_t extruder) {
float dx = x - current_position[X_AXIS];
float dy = y - current_position[Y_AXIS];
float dz = z - current_position[Z_AXIS];
int n_segments = 0;
if (mbl.active) {
float len = abs(dx) + abs(dy);
if (len > 0)
// Split to 3cm segments or shorter.
n_segments = int(ceil(len / 30.f));
}
if (n_segments > 1) {
float de = e - current_position[E_AXIS];
for (int i = 1; i < n_segments; ++ i) {
float t = float(i) / float(n_segments);
if (saved_printing || (mbl.active == false)) return;
plan_buffer_line(
current_position[X_AXIS] + t * dx,
current_position[Y_AXIS] + t * dy,
current_position[Z_AXIS] + t * dz,
current_position[E_AXIS] + t * de,
feed_rate, extruder);
}
}
// The rest of the path.
plan_buffer_line(x, y, z, e, feed_rate, extruder);
current_position[X_AXIS] = x;
current_position[Y_AXIS] = y;
current_position[Z_AXIS] = z;
current_position[E_AXIS] = e;
}
In there you can clearly see why it stutters on moves longer than 30mm. If I change the split value to 50.f, the stuttering starts after 50mm.
One more thing to note: Stuttering occurs only when the buffer starts from empty (ie start of print) and only affects first move. It also happens when the buffer starts draining at the end and again it only affects the first move after buffering stops.
Also could this also affect anything? I didn't see any difference when I disabled it, but who knows:
#ifdef SLOWDOWN
//FIXME Vojtech: Why moves_queued > 1? Why not >=1?
// Can we somehow differentiate the filling of the buffer at the start of a g-code from a buffer draining situation?
if (moves_queued > 1 && moves_queued < (BLOCK_BUFFER_SIZE >> 1)) {
// segment time in micro seconds
unsigned long segment_time = lround(1000000.0/inverse_second);
if (segment_time < cs.minsegmenttime)
// buffer is draining, add extra time. The amount of time added increases if the buffer is still emptied more.
inverse_second=1000000.0/(segment_time+lround(2*(cs.minsegmenttime-segment_time)/moves_queued));
}
#endif // SLOWDOWN
I'm looking at this now
Ok - this was superhelpful indeed. This is caused by the fact the stepper isr picks up the first chunk of the line immediately. By the time we queue the second, the planner cannot recalculate the exit speed anymore as the block is already in use by the isr.
We could add a new flag to the block to allow queuing in "withheld" state, but there's some extra locking to be performed in the various block handling structures which I don't like very much. Pausing the isr is also out of the question there.
What we should do instead is allow the planner to recalculate the nominal and exit speed. There are several conditions where doing this is still safe, su as when we're still accelerating and the exit speed is increased. This will properly fix also the some of those micro-interruptions where the planner can recover.
Doing so though touches several parts that I already changed in LA1.5. I'd say it's better to fix this after LA1.5 has been tested and merged.
Also, that SLOWDOWN ifdef, mmmh... I've mixed feelings :).
@wavexx Totally agree. I also thought about this, but didn’t study the ISR and the planner enough to come up with a fix. I hope that LA1.5 gets merged without problems in 3.9 as to come up with a fix as soon as possible. Considering this issue is for as long as MBL existed in PF, then it’s about time this issue gets fixed.
Not only that, but as I was digging deeper, there are some several (broken?) assumptions about the first block being immediately tagged as busy.
I tried to pause the isr while scheduling the first 2 segments from a fullstop condition (crude fix, but still better than nothing), but it's still not enough to remove this condition.
This is going to be fun to fix.. ;)
Hi all, we're running some very thorough testing on a raspberry pi zero alternative for the MK3S, which is a banana pi m2 zero with a flipped orientation in order to expose the proc and ram to the outside and be able to passively cool them with a couple of small heatsinks. Things are looking very good, load average never goes over 1.4 [quad-core proc, so about 33% of capacity], and contrary to the zero as instructed by the docs, this board isn't exhibiting any load-related print slowdowns. However, it seems we might still be getting some stuttering in very high segments-per-second areas [sinusoidal patterns]. I've read here that the DEFAULT_MINSEGMENTTIME
setting can be used to slow down feedrate when the buffer isn't able to provide the next line of code in time, how is this handled in prusa firmware?
Thanks
Depends a lot on how the piece is printed. I guess you're using octoprint on the pi and print over the serial?
In this case two issues might be present: serial is not fast enough, or the fw cannot keep up. You might get around the first by bumping the serial speed to 250000 (which seems to work fine, but I didn't do much testing on it!). The second is harder.
More than DEFAULT_MINSEGMENTTIME, what might improve this is increasing the planner buffer size (doubling it), tweak the slowdown behavior, reduce the speed of the print so that this doesn't happen (sometimes very little is required), or reduce the geometry complexity of the piece.
The Prusa FW is still identical to Marlin 1/2 in this area. We use the same SLOWDOWN and SLOWNDOWN_DIVISOR settings, that halve the speed when the buffer gets half-empty. This is far from being optimal on circular pieces, sadly, since it tends to oscillate between full speed and half-speed, making a visible dent every time this happens.
Reducing speed and/or reducing the instructions yields better results at the moment.
@wavexx thank you so much for the insight
Depends a lot on how the piece is printed. I guess you're using octoprint on the pi and print over the serial?
exactly
In this case two issues might be present: serial is not fast enough, or the fw cannot keep up. You might get around the first by bumping the serial speed to 250000 (which seems to work fine, but I didn't do much testing on it!). The second is harder.
i can't get anything other than 115200 to work, from either board.
More than DEFAULT_MINSEGMENTTIME, what might improve this is increasing the planner buffer size (doubling it), tweak the slowdown behavior, reduce the speed of the print so that this doesn't happen (sometimes very little is required), or reduce the geometry complexity of the piece.
i'm looking for a way for either the pi or the firmware to slow down automatically, i'd like to avoid having to train everyone using the slicer to recognize very high poly areas and slow down things in prusaslicer. so reducing the speed of the print is something i'd like to automate. reducing geometry complexity would certainly help, but while speed is an acceptable tradeoff to be able to print from the pi, quality i'd like to stay the same.
very interesting about planner buffer size / slowdown behaviour. the latter is what you're talking about right below, correct? how would i change the former instead?
The Prusa FW is still identical to Marlin 1/2 in this area. We use the same SLOWDOWN and SLOWNDOWN_DIVISOR settings, that halve the speed when the buffer gets half-empty. This is far from being optimal on circular pieces, sadly, since it tends to oscillate between full speed and half-speed, making a visible dent every time this happens.
is there a way to reduce speed and leave it reduced for the entirety of the difficult zone, without constant speed oscillation?
Reducing speed and/or reducing the instructions yields better results at the moment.
is there any way to see from terminal serial log if firmware is waiting for commands / buffer is empty / is slowing down moves / etc?
thanks again!
In this case two issues might be present: serial is not fast enough, or the fw cannot keep up. You might get around the first by bumping the serial speed to 250000 (which seems to work fine, but I didn't do much testing on it!). The second is harder.
i can't get anything other than 115200 to work, from either board.
You need to tweak the FW for this of course. The default is 115200.
i'm looking for a way for either the pi or the firmware to slow down automatically, i'd like to avoid having to train everyone using the slicer to recognize very high poly areas and slow down things in prusaslicer. so reducing the speed of the print is something i'd like to automate. reducing geometry complexity would certainly help, but while speed is an acceptable tradeoff to be able to print from the pi, quality i'd like to stay the same.
I use octoprint occasionally, but I admit I'm not too experienced with it (I mostly use it "as is" ;)), so I cannot suggest much there.
Did you try to see if setting a limit on resolution in prusaslicer helps? If the issue is really massive models, going to print settings -> advanced -> resolution -> 0.01 (which is pretty much the effective resolution anyway) might be an easy tweak.
very interesting about planner buffer size / slowdown behaviour. the latter is what you're talking about right below, correct? how would i change the former instead?
yes
is there a way to reduce speed and leave it reduced for the entirety of the difficult zone, without constant speed oscillation?
Improving this behavior is something I'd love to do at some point. What you describe is one approach. Another I was thinking about was to perform a proportional slowdown, which should also reduce the effect of the speed bumb, and maybe "self-balance" itself if done right.
I suspect that changing the divider at the moment simply moves the point where this happens depending on the current perimeter speed, so it might solve the problem for some prints, but then again it will make it appear in other spots. I didn't test this, so if you want to experiment search for SLOWNDOWN in the Firmware/planner.cpp file.
is there any way to see from terminal serial log if firmware is waiting for commands / buffer is empty / is slowing down moves / etc?
Defining PLANNER_DIAGNOSTICS
in Configuration_adv.h
will enable some extra info on the serial to debug issues such as these
In this case two issues might be present: serial is not fast enough, or the fw cannot keep up. You might get around the first by bumping the serial speed to 250000 (which seems to work fine, but I didn't do much testing on it!). The second is harder.
i can't get anything other than 115200 to work, from either board.
You need to tweak the FW for this of course. The default is 115200.
ok sorry i read that elsewhere but hadn't updated my comment yet, i've done that, flashed it on my MK3S, octoprint works great but i haven't been able to really bump the feedrate higher than before on the same curves without incurring in the exact same very visible slowdowns.
i'm looking for a way for either the pi or the firmware to slow down automatically, i'd like to avoid having to train everyone using the slicer to recognize very high poly areas and slow down things in prusaslicer. so reducing the speed of the print is something i'd like to automate. reducing geometry complexity would certainly help, but while speed is an acceptable tradeoff to be able to print from the pi, quality i'd like to stay the same.
I use octoprint occasionally, but I admit I'm not too experienced with it (I mostly use it "as is" ;)), so I cannot suggest much there.
Did you try to see if setting a limit on resolution in prusaslicer helps? If the issue is really massive models, going to print settings -> advanced -> resolution -> 0.01 (which is pretty much the effective resolution anyway) might be an easy tweak.
very interesting about planner buffer size / slowdown behaviour. the latter is what you're talking about right below, correct? how would i change the former instead?
yes
is there a way to reduce speed and leave it reduced for the entirety of the difficult zone, without constant speed oscillation?
Improving this behavior is something I'd love to do at some point. What you describe is one approach. Another I was thinking about was to perform a proportional slowdown, which should also reduce the effect of the speed bumb, and maybe "self-balance" itself if done right.
I suspect that changing the divider at the moment simply moves the point where this happens depending on the current perimeter speed, so it might solve the problem for some prints, but then again it will make it appear in other spots. I didn't test this, so if you want to experiment search for SLOWNDOWN in the Firmware/planner.cpp file.
great! i will definitely perform some testing with this. is there any way i can help fast-tracking this problem? as in, perform some rigorous testing to offload some of the work? i could ask one of our devs to devote some time but he would need some time / help getting up to speed before being able to contribute i suppose.
is there any way to see from terminal serial log if firmware is waiting for commands / buffer is empty / is slowing down moves / etc?
Defining
PLANNER_DIAGNOSTICS
inConfiguration_adv.h
will enable some extra info on the serial to debug issues such as these
ok, i'll try playing around with SLOWDOWN
and PLANNER_DIAGNOSTICS
, is there a "smart" way to do this, and / or to change values in a way they are more likely to produce a good result?
Thank you so much for your help!
On Tue, Sep 01 2020, Nicolas North wrote:
ok sorry i read that elsewhere but hadn't updated my comment yet, i've done that, flashed it on my MK3S, octoprint works great but i haven't been able to really bump the feedrate higher than before on the same curves
Then serial i/o is not really the bottleneck in this case!
ok, i'll try playing around with
SLOWDOWN
andPLANNER_DIAGNOSTICS
, is there a "smart" way to do this, and / or to change values in a way they are more likely to produce a good result?
There's no guidebook ;), but I think having a good source gcode where the problem is persistent and shows clear quality issues in the outer perimeter would be more helpful to see the results later.
Bonus points if the same model has different faces with different resolution (and/or different speed, resulting in the same behavior), so that you can see the "bouncing" effect on the same layer when using a fixed speed divisor.
I've seen these behaviors in isolation, but having a torture test would be nice =)
ok, will report back with some test results!
What about applying gradual slowdown/speedup to all segments, not just short segments? What about using nearly whole buffer for that to make it more smooth? (not just half of the buffer) https://github.com/prusa3d/Prusa-Firmware/pull/2820 I haven't tested it. I would expect strange slow movement at the beginning of the print and then smooth operation. (Buffer is normally always full except for the case of high poly arches)
We need some buffer, I'd favor the gcode speed above everything. While I was debugging the LCD stuff it wasn't uncommon for the buffer to be around 60-70% full on a normal print. Doing what you did on 50% but on half the buffer as before is perhaps safer.
Honestly, I was wondering for quite a while if we can't just double the planner buffer as well.
just to renew my availability to run tests on our store's 6 MK3Ss, if anyone has interesting ideas. I'm currently testing with the two aforementioned parameters, but am open to any other improvement anyone would like to bulk test.
Then serial i/o is not really the bottleneck in this case!
@wavexx re-reading your comment, just to make sure i have my terms correct, if the quality issue [blobs because of stuttering] is only showing via serial, this should be a serial i/o bottleneck issue, no? [my board isn't having any load / temp issues like raspberry pi zeros]
@mkbel i've noticed your recent development on #2820 and your branch, can I ask you if this also helps in cases of serial-induced slowdowns, or, if that term is incorrect, "quality issues showing up when printing not directly from sd card"?
Thanks to both!
@nordurljosahvida Both current state of firmware and also my experiment #2820 relies on fact, that printer is capable to execute segments with duration "Minimum segment time (µs)" sustainably. If buffer is draining due to shorter segments printer starts slowing down up to the moment planed segments are stretched in time to "Minimum segment time (µs)". It doesn't slow down more, so if parameter "Minimum segment time (µs)" is set to the smaller value than the printer is really capable it can lead to buffer underrun. Default value of this parameter is 20000. I was printing from SD card my torture gcode attached to #2820 and object starts to be scarred when I set that parameter to lower than 15000. So for SD printing there is great margin (it may depend on SD card and gcode flavour). When you print from USB (which is connected to UART with baud rate 115200 and each line needs to wait for OK reply) sustained "Minimum segment time (µs)" may need to be higher than for SD printing. You can tune that parameter with M205 B\<number> e.g. M205 B30000. You can store that setting to printer's EEPROM by M500 so it is remembered to next power cycle. If you are using both USB and SD card printing you can add M205 Bxxx command in the beginning of file to be printed via USB and leave default value for SD prints. Maybe there is also some way in Octoprint to add some custom gcode before each print...
@mkbel thank you so much for the explanation. this is really interesting as i could apply a hotfix to all prints going through octoprint with a simple start gcode. however, based on this comment from wavexx:
More than DEFAULT_MINSEGMENTTIME, what might improve this is increasing the planner buffer size (doubling it), tweak the slowdown behavior, reduce the speed of the print so that this doesn't happen (sometimes very little is required), or reduce the geometry complexity of the piece.
it seems that maybe minsegmenttime
is not the whole story here? could your work on #2820 correlate with increasing planner buffer size or are these separate issues?
2820 is probably unrelated to your problem. Current firmware algorithm slows down up to Minimum segment time. Because there is some computation power margin buffer starts to be refilled in that moment up to moment printer speeds up, drains buffer and slows down again. So if you are printing high polygon count cylinder in high speed, printer is periodically slowing down and speeding up and it has some impact on surface finish. My proposed change is to lock speed to maximum sustained rate when the buffer is drained and don't speedup until segments with higher sustained speed are to be planned regardless of buffer filled earlier.
If I understand correctly this is exactly what I was wondering, why we don't keep maximum speed lower to avoid constant speed oscillations, but I might be missing something here. Anyway, great work. I've tried compiling with your diff inserted but flashing fails, but works without code from your tree. Will have to investigate.
Anyway, I'm now running a very slightly modified version of 3.9.1-RC1
[great timing btw!] with:
--- a/Firmware/Configuration.h
+++ b/Firmware/Configuration.h
@@ -78,7 +78,7 @@ extern PGM_P sPrinterName;
#define SERIAL_PORT 0
// This determines the communication speed of the printer
-#define BAUDRATE 115200
+#define BAUDRATE 250000
--- a/Firmware/Configuration_adv.h
+++ b/Firmware/Configuration_adv.h
@@ -177,6 +177,12 @@
// If defined the movements slow down when the look ahead buffer is only half full
#define SLOWDOWN
+// enable extra info on serial to debug slowdown issues
+// https://github.com/prusa3d/Prusa-Firmware/issues/1406#issuecomment-684990545
+// 2020-09-04 tillverka nz
+
+#define PLANNER_DIAGNOSTICS
And I got a beautiful little corner on my LCD showing me buffer size [Q?] on a scale from 1 to 15 I think, very very useful. I'd love to be able to read those states from serial, in order to catch low buffer statuses from octoprint's logs, any idea?
I'll try re-implementing your code and flashing it. I'm obviously interested independently of whether this is directly related to serial printing or not.
Hi again! If and when you have time, could I ask you for some insight related to my latest answer? I've done some experimenting, Q values still get really low as expected with 250000 baudrate. I'm still not sure how to tweak SLOWDOWN
or implement #2820 without flashing failing. I see some progress on it, which is very good to see. Thanks again for all the work
@DRracer @mkbel @haarp @wavexx I looked more into this and I found where the problem starts. Previously I said moves over 35mm are affected. I found out that any move that is over 30 (x >30) are affected. And I kept looking keeping this in mind and I found this:
void mesh_plan_buffer_line(const float &x, const float &y, const float &z, const float &e, const float &feed_rate, const uint8_t extruder) { float dx = x - current_position[X_AXIS]; float dy = y - current_position[Y_AXIS]; float dz = z - current_position[Z_AXIS]; int n_segments = 0; if (mbl.active) { float len = abs(dx) + abs(dy); if (len > 0) // Split to 3cm segments or shorter. n_segments = int(ceil(len / 30.f)); } if (n_segments > 1) { float de = e - current_position[E_AXIS]; for (int i = 1; i < n_segments; ++ i) { float t = float(i) / float(n_segments); if (saved_printing || (mbl.active == false)) return; plan_buffer_line( current_position[X_AXIS] + t * dx, current_position[Y_AXIS] + t * dy, current_position[Z_AXIS] + t * dz, current_position[E_AXIS] + t * de, feed_rate, extruder); } } // The rest of the path. plan_buffer_line(x, y, z, e, feed_rate, extruder); current_position[X_AXIS] = x; current_position[Y_AXIS] = y; current_position[Z_AXIS] = z; current_position[E_AXIS] = e; }
In there you can clearly see why it stutters on moves longer than 30mm. If I change the split value to 50.f, the stuttering starts after 50mm.
One more thing to note: Stuttering occurs only when the buffer starts from empty (ie start of print) and only affects first move. It also happens when the buffer starts draining at the end and again it only affects the first move after buffering stops.
Also could this also affect anything? I didn't see any difference when I disabled it, but who knows:
#ifdef SLOWDOWN //FIXME Vojtech: Why moves_queued > 1? Why not >=1? // Can we somehow differentiate the filling of the buffer at the start of a g-code from a buffer draining situation? if (moves_queued > 1 && moves_queued < (BLOCK_BUFFER_SIZE >> 1)) { // segment time in micro seconds unsigned long segment_time = lround(1000000.0/inverse_second); if (segment_time < cs.minsegmenttime) // buffer is draining, add extra time. The amount of time added increases if the buffer is still emptied more. inverse_second=1000000.0/(segment_time+lround(2*(cs.minsegmenttime-segment_time)/moves_queued)); } #endif // SLOWDOWN
Are there any updates on this issue? Can it be fixed now that LA1.5 is merged? Also I feel like there are at least 2 completely different issues in this issue, so perhaps a new issue should be created?
@nulaft there are two/three separate causes of stutters.
Short segments (which are read fast enough) cannot be planned fast enough. This is this issue. The code from @mkbel should work-around this issue by improving the current slow-down logic to something that doesn't introduce additional artifacts. I was busy @ work in the months so I didn't get around to test this yet.
Sometimes small segments cannot be sent fast enough across the serial, causing planner starvation. This is what #2657 and #2955 try to address. On top of that, the LA stepping code now is robust enough that we could also double the serial speed to 256000. I've been using it without issues since some months now.. but I'm not sure how easy is for others to change the serial speed.
There is a third source of "random" stutters which is caused by current planning code. The first segment received after a drained buffer will not be reprocessed correctly by the planner as new segments are received. This causes the printer to always accelerate and decelerate on the first segment, irregardless of length. This is mostly visible when using MBL and planning a segment which exceeds 3mm as shown initially in your code snippet, but it can also be triggered currently by a combination of the above two. I can't find the issue number about this one at the moment, but it's there somewhere (since I've commented on it ;))!
I've been working on the last one on&off using a couple of approaches. My wish is to fix it by modifying the segment in-flight when possible, and improving the MBL function to always schedule segments in a batch (which would also reduce the scheduling overhead)...
@wavexx -- do you have a working branch for the solution to the third source of stutters which you are working on?
This issue has been flagged as stale because it has been open for 60 days with no activity. The issue will be closed in 7 days unless someone removes the "stale" label or adds a comment.
This issue has been closed due to lack of recent activity.
Hello,
during fast and frequent head moves while printing from the SD card, I can often hear the head pausing for the fraction of a second. This seems to coincide with LCD updates. I suspect that the LCD update routine takes long enoguh on the tiny CPU to delay print commands for a couple of milliseconds.