Printer reboots when attempting to start print with PCFan firmware

[Adam-Beaumont]

Hi, I installed the AC fan version of the newest firmware and was able to print successfully. When I flashed the PC fan version, I am now unable to start any print successfully. After flashing the firmware, I've tried restoring factory settings as per instructions, rerunning autocalibrate, etc.

After starting a print, it will successfully come up to temperature, home itself, then hang for a second or two before the printer reboots. I am able to control the fan manually through Gcode, and can manually move print head and extrude filament through repetier but am unable to do an actual print. I have tried through both usb printing and SD card to no avail. The log in repetier doesn't really reveal any information about whats happening besides a simple "communication timeout - reset send buffer block" error.

Any ideas?

[see3d]

I just came here to report a similar sounding issue. I am also running the PC Fan version. Everything's the same problem as the Author, except I suspect a thermal runaway error reboot. Sometimes I could not get beyond the G29 P0, and sometimes I could get partially through the first layer. The screenshot is Repetier-Host Mac. The Log is up to the point of failure. The failure on the graph is right at the hot end temp peak. I thought it was because my hot end PID needed to be retuned, because of that funny dip just before it hit set temperature (240C). I tuned the PID. It gave me slightly different results, but as you can see that dip is still there. In M4MPMD, the same PID numbers gave a perfect chart coming up to temp.

I had just finished calibrating my M666/5 for dimensional accuracy and flatness for the first time. I was ready to try an easy first test print to verify the bed was indeed level and I could print anything before tuning the Linear Advance K. After half a dozen attempts, I am getting nowhere. I have a production run to print this weekend, so I may have to go back to M4MPMD for now. :-(

[aegean-odyssey]

Thanks for putting your printer through its paces -- the tests are helpful. Aside from a little more code, the fan control in the PC version should have little impact on the execution (the PWM control is implemented in hardware). In the latest version of firmware, though, the linear advance feature almost doubles the execution time of the (stepper motor) motion interrupt. If not directly the culprit, I think it can exacerbate the actual cause. There are a few thoughts that come to mind:

• the reset you describe is likely a watchdog timeout. It should never occur and it means the code is "stuck" somewhere;
• the "tickle" for the watchdog timer is in the same loop as the temperature control, so that's why they may seem related;
• printing from the SD card without the usb cable attached is the least demanding communication-wise;
• SD card disk errors, can slow things down and generate the fault as well. A dos disk scan can verify the SD;
• to eliminate "linear advance" as a suspect, you can see if the issue disappears with an earlier version of firmware, say the 119r10 version;
• another way to lighten the CPU load is via the M665 Sn code. Pre-"linear advance" it was set ridiculously high at M665 S500, and now the default is a high (and chosen arbitrarily) M665 S200. I think typical Marlin values range from S80 - S120.

I'll try to get set up here over the next few days to see if I can reproduce the issue. Please report anything you find.

[aegean-odyssey]

@see3d, are you using the 10A (as opposed to the 05A) flavor of the firmware? Also, for any grave error condition or fault that the firmware checks under normal operation (e.g thermal runaway, communication errors, etc.), the firmware halts the machine and flashes the red led, but it should not reboot until the user presses the led push button switch. Any CPU fault (these should never happen), though, traps and eventually reboots the machine via a watchdog timeout. So in the mpmd_marlin_1.1.x firmware, a reboot indicates a fairly serious (though not unsafe) issue.

[see3d]

I did not have linear advance enabled yet (K=0). I can try setting M65 S200 to S120, but it was failing even before it started to print most of the time. I can try without the USB connected. I usually have it connected while debugging and initial setups and to monitor the temperatures. Can the watchdog time interval be set longer to test? Can the watchdog get tickled in another compute intensive place also to help figure out where the problem is?

Oh, that reminds me. I was not able to start printing my SD card file from the USB interface to Repetier. It gave me an error that the file could not be opened (even though it could see it). No biggie for now, but just in case you have not tested it out yet.

[see3d]

Yes, I have the 10A flavor. When it fails, the screen goes to something that looks kind of like the screen that comes up when pressing the LED switch, but it is on for only a fraction of a second, so I could not read it, then it comes back to the normal home screen (but not the FW version screen). The internal state is scrambled though, because if I try to print the file again at that point the printer acts in different bizarre ways. I tried to press the LED button twice for a reboot, but the only way I can start clean again is to power cycle it. I did not notice if the LED switch was Red. It is facing away from me and I usually just reach around to press it. I will look out for the color next time. It was probably not red though, because it was not waiting for me to do anything, it was back running the home UI (including the USB as long as I hit the connect button again).

[aegean-odyssey]

Unfortunately, even with K=0 the linear advance motion code still requires much more execution time than the code without it. The watchdog timeout is set for about 6 to 8 seconds -- under even busy conditions, 2 to 3 seconds should be sufficient. Early on I went down the path of extending the watchdog timeout; it never helped. In all cases, the issues were not marginal timing or delays -- actual problems were the code ended up bouncing between interrupts, or goes belly-up from a dangling pointer or garbage values in the settings. There also have been a fair number of compiler optimization issues with the marlin (and my) code. The compiler optimizes aggressively to keep the code small enough to fit in the available flash memory, so some lax coding practices produces unintended optimization consequences.

Thanks for the report. It sure does sound like something is buggered. When the internal state is "scrambled", does the M503 command reflect this? (show garbage?). Anyway, I'll take a look a the PCfan code.

[see3d]

The M503 command does not show damage. I think the code was running then just everything stops. Then after a few seconds it resets. So it must be a code crash as opposed to just being slow. I will have to look in the morning at it closer when it crashes.

[Adam-Beaumont]

Just to add my log reaches the same point as see3d before failure, although i was not able to even get a partial first layer. I am also running the 10amp variant, 0001 if I remember correctly.

Edit: Also I'm curious why the K linear advance might be related, since the AC fan firmware was able to print without issue. I haven't played with K values yet (so mine is still set to 0)

[see3d]

I can confirm that the AC fan firmware prints just fine for me on SD card only or with with Repetier-Host over USB active and S200. There must be a clever little bug hiding in those few lines of PC fan code. Sometimes just a typo in my code takes forever for me to see.

On a related note, the AC version does not work the same as the stock or M4MPMD versions of the FW. The AC version seems to have the fan on 100% if the hot end is above about 60C. The stock version does not allow the fan speed to be below about 40% on when the hot end is above about 60C, but can still be controlled by the Gcodes in the 40% to 100% range. I do not know if this is a bug or you just missed this subtlety of stock operation.

I do have a separate PC fan, but I would actually want it to run like the AC fan from about 33% on up when the hot end is running. My air distribution ring is printed out of PETG and right against the hot end block. It would melt if it did not have a little cooling air going through it whenever the hot end is above 100C. I can do that with Gcodes, but it is safer in FW,

[Adam-Beaumont]

That Ac fan behaviour in stock firmware is interesting, didn't know that. I added a command at the beginning of my start gcode to start the fan during preheat but when a reset happens the fan turns off even though the temperature of the hot end is still quite high, which could cause some heat creep issues.

[see3d]

@Adam-Beaumont When my stock FW machine would crap out or I power cycled it with a hot nozzle, the fan would immediately start up. Right now my hot end cooling fan is connected to the +12V (13V actually), so it always runs with power. It was noisy though, so I put a 100 Ohm resistor in series with the ground lead that dropped the noise to just below the noticeable level. That seems to be enough. If it does get heat creep on a long print, I will add a switch to let it run full speed during a print. I like peace and quiet, because it sits on my desk. My part cooling fan is another thing, but it is controlled so it can be turned off when not printing.

[aegean-odyssey]

Yes, @Adam-Beaumont, the ACfan flavors keep the fan on whenever the nozzle temperature exceeds 50°C. Early on I thought I could combine both types of fan control into one firmware flavor, but decided against it -- mostly because the Marlin code really doesn't want the two types of fans sharing the same I/O; plus I was trying to change as little of the original Marlin code as possible; and my solution would result in printer behavior that could be confusing, difficult to document, and a pain to test and validate.

Of course now I just need to get the PCfan firmware to work :)

[Adam-Beaumont]

@aegean-odyssey interesting. Just a small note I did flash the second most recent PC fan firmware and it works perfectly, so the problem is only with the newest release

[aegean-odyssey]

Thank you @Adam-Beaumont for the update! Your bit of information (and @see3d's report) narrows down the search considerably -- something to do with the linear advance code.

I also think that there is an underlying race condition in Marlin's endstop code with ENDSTOPS_INTERRUPT enabled (as it is in our firmware). I've been pondering it for a couple of months; trying to get a clearer picture of the set of real-life conditions that would expose the issue.

[see3d]

@aegean-odyssey I understand your reasoning for the PC vs AC fan operation, but I am not in agreement with it. In my mind, the object should be to put Marlin on the MPMD to expand its capabilities, not to add MPMD to expand the number of Marlin users. I see this AC operation as a downgrade of the basic factory capability of the MPMD. The feature is there because that is what is needed for the stock machine as shipped to give a better printing performance.

Safety of the machine is another factor. I have had to "pull the plug" in the middle of printing more often that I would like to admit. The hot end needs to have cooling restored as a default condition if the hot end is over 50C-100C. If a separate PC is added and the supply nozzle is a printed plastic piece close to the nozzle, it still needs a minimum airflow to keep the plastic from melting or warping.

If I wanted to have a single code base, I would expand the capability so that there would be one EEPROM setting that was the minimum fan on % if the hot end is over 50C. Set to 0% would be your current PC implementation, and 100% for your current AC operation. Think of the MPMD hardware as having a PC fan that steals some air to keep the hot end cool.

So far my experiments show that I need very little air to prevent heat creep. On the other hand, too much part cooling can have bad effects for vase mode prints. The layers don't stick well and the cooling is uneven, causing differences in appearance around the part. Before I added a separate PC fan, I needed to run many parts at 50% cooling.

To get the desired functionality in hardware to overcome the FW limitation I would need to have a resistor and diode from the 12V supply to supply a minimum amount of current to the fan all the time. Your PC FW option would be required. That may be asking a bit much of a typical user who wants to upgrade to the Marlin 1.1.x FW.

If I were going to stay with a single fan (I have a second MPMD with just a single fan), I would design and print a new fan housing that made the PC function more symmetrical. I would still want to control the PC airflow between the minimum needed for hot end cooling and full blast. Some users take the approach of adding external fans to blast the part printing area with a couple of external fans. Unfortunately that ends up cooling the baseplate, which is already underpowered without a PS upgrade. Needless to say, part cooling is a big issue to printing a good part.

I have run many continuous hours of prints over the last two days on my dual fan machine with 135C, 140C, 145C and 70C bed using PETG with M900 K2.0 and 2.5. I connected the factory hot end fan through a 100 Ohm resistor to the +12V (13V). That makes the fan which runs all the time almost silent. I did the same thing to the bottom case fan with a 150 Ohm resistor to make it almost silent, since it also runs all the time and is very noisy as shipped. Right now, using your AC FW, since the PC FW is broke, I found that I had to clamp my air supply hose to restrict half the air from the PC fan to make a successful print. I feel this illustrates why the AC fan function still needs to have speed control like the factory FW.

[aegean-odyssey]

@see3d, I appreciate your comments. There is an ACfan and PCfan option because Marlin really does not mix the two ideas in a machine with a single fan. I can't say if one option is better than the other; your opinion is certainly more credible here.

Combining the temperature controlled and manually controlled fan is a thorny issue:

If I wanted to have a single code base, I would expand the capability so that there would be one EEPROM setting that was the minimum fan on % if the hot end is over 50C. Set to 0% would be your current PC implementation, and 100% for your current AC operation. Think of the MPMD hardware as having a PC fan that steals some air to keep the hot end cool.

So what happens during a print, and M106 0 (turn the fan off) is issued? With your strategy, the fan speed would be set to the minimum fan speed for ACfan operation; negating the fine control that's desired with the PCfan operation. On the other hand, overriding the minimum fan speed defeats the purpose of the ACfan operation as a safety mechanism. I do think there is a way to make it work, but I'm holding out until there's a strategy that can be easily presented as an "improved" PCfan operation and can replace the current PCfan operation.

The mpmd_marlin_1.1.x firmware mantra is "improve the every-user experience with the stock Monoprice MP Mini Delta printer hardware, but don't hinder (maybe even encourage) the modder, tinkerer, and power-user". I agree that the ACfan option falls short on the latter part of the vision.

[see3d]

So what happens during a print, and M106 0 (turn the fan off) is issued? With your strategy, the fan speed would be set to the minimum fan speed for ACfan operation; negating the fine control that's desired with the PCfan operation. On the other hand, overriding the minimum fan speed defeats the purpose of the ACfan operation as a safety mechanism. I do think there is a way to make it work, but I'm holding out until there's a strategy that can be easily presented as an "improved" PCfan operation and can replace the current PCfan operation.

With my suggestion, M106 0 would turn the fan off once the hot end dropped below 50C or whatever minimum is decided. This is the way that the stock FW does it. I am not familiar with the Marlin code structure to say how it could be implemented. All that is really needed is for both the fan control module and the hot end module to each have access to the current commanded fan speed and the hot end temperature and the minimum fan speed for a hot hot end. Then if the temperature drops below the threshold (or conversely rises above it), the temperature module can command a new fan speed, either the last commanded one, or the minimum one. I am assuming that the temperature module get called a lot and the fan one might only get called when a new fan speed is commanded.

Alternatively, the control of this feature might reside in a safety check module that determines things like thermal runaway. The point is that it is a temporary safety override of fan speed when the temperature is above a threshold and the fan speed is below a minimum threshold. I don't know of a better strategy for a single fan.

[aegean-odyssey]

Bear with me, I want to make sure I understand your change.

I think you're suggesting a fan control that prevents the actual fan speed from being lower than some minimum value if the hotend temp exceeds some threshold. Succint, easy enough to implement:

actual_fan_speed =
    max(
       (hotend_temp < MIN_HOTEND_TEMP) ? 0 : MIN_FAN_SPEED ,
       desired_fan_speed
    )

This control, though, with a MIN_HOTEND_TEMP of say 50°C, means that the fan speed can never be set lower than the MIN_FAN_SPEED while printing.

Is this really the desired behavior? I think part cooling is often turned off completely for the first layer or so of a print, but is this a no-no for a printer that combines part cooling and hotend fans into one fan?

Also, what is a proper MIN_FAN_SPEED? MIN_HOTEND_TEMP?

And you're correct about Marlin, the ACfan control occurs in a slower temperature control loop, while the PCfan control is implemented in the planner. For the scheme you suggest, both sections would need to be modified; not too difficult, though there are some complications with an ACfan speed (MIN_FAN_SPEED) less than full on.

Of course, a prerequiste is the PCfan option working with linear advance enabled. :)

[Adam-Beaumont]

I do agree that some sort of minimum fan speed would be nice in the PC fan firmware. I have my end gcode turn the fan on to prevent heat creep after a print is done but I've been aborting many prints over the past couple days while calibrating and have to quickly run a command to turn the fan back on every time. I think normally this might not be an issue but especially because the LCD control does not have fan controls, it could be problematic in SD printing away from a computer.

Edit: but just an aside, the work you've done on this Marlin port is awesome and greatly expands this printers capabilities. Many thanks

[see3d]

@aegean-odyssey Yes, that is the algorithm that the stock FW uses.
Well, since we are on the subject in earnest, I will start making measurements on my stock V44 machine and my dual fan machine to get actual numbers.
I think a MIN_HOTEND_TEMP of 70C would be safe. That is what the factory is set to.
When I command the fan off, it goes off for about 1 second and then turns back on again.

I know in the distant past, when the hot end was cooling off, the fan would turn off when it dropped below some min temperature after a print. I remember because I noticed it when it happened and took a mental note. However in my testing right now, the fan stays on until it is commanded to turn off again after the hot end has cooled. I don't know if this was a FW change in this newer machine (my original machine was the second batch after the Kickstarter), or if there was something in the standard ending gcode that waited for the temp to drop and then sent the fan off gcode.

MIN_FAN_SPEED is a bit trickier. It depends on the configuration, which is why I suggested it be a settable number. Stock measurement looks like setting fan % is setting the PWM_ON duty cycle, except at the low end. It could be my waveform estimations are off from reality, but this is what I am measuring as I type this:

The PWM frequency is 76Hz. 100% on fan is 200ma draw. From 50% to 100%, the command and PWM_ON % match up. From 1% to 40%, the PWM_ON % seems to be 5% higher than commanded.

When heating the nozzle, the fan turns on about 60C with a 65% PWM_ON %.

So that should give you all you need to know for duplicating the factory single fan arrangement.

My dual fan machine with the factory hot end fan shows 170ma at 12V (it was 180ma at 13V). Interesting that it is only 85% of the current of my other machine for the 12V supply across each fan. Perhaps manufacturing tolerances.

I added a 100 Ohm resistor for my 13V supply to make it quiet. I just measured 130ma for that configuration. If I assume that the PWM_ON % of the factory machine is the same as the average current, then that puts both machines running at 130ma for the default cooling setting.

All these numbers are a little different that what I thought I remembered. Perhaps I did not measure all of them in the past (my little portable scope is new this year). My old one is a 1970's era HP beast that I did not like to get out unless absolutely needed (it cost more than a new car when I bought it new).

For a dual fan machine, the hot end fan could be set up like mine (easy to do). The PC fan though depends on how it is configured. No auto minimum part cooling is an option for some. Others like mine might need a small amount of cooling when the hot end is on, perhaps 25%. It depends on the fan motor and the air channel arrangement. I hope it is practical to make it settable. A different hot end and adding a PC fan is a popular mod.

[aegean-odyssey]

@Adam-Beaumont, I see what you mean. I've been testing with the PCfan firmware, and it's a bit disconcerting to watch the hotend heat up without the fan on. Btw, no joy yet for PCfan and linear advance.

[aegean-odyssey]

@see3d, @Adam-Beaumont

It seems that we want the single fan to operate in two distinct ways:

1. as a part cooling fan (mostly) while the printer is printing; and
2. as an automatic temperature-controlled fan when not printing.

IF the firmware could determine a "printing" state, then it seems that the fan control could be fairly straightforward:

• if not printing (ACfan): the fan is on 100% whenever the temperature exceed 50°C;
• during a print (PCfan): the fan is controlled via M106, M107 commands.

The trick is knowing whether the printer is "printing" or not. Printing from the micro SD card is easy; as long as the machine is reading from a file, it's "printing". But, identifying a "printing" state when printing via the USB port is not so cut and dry. For this case, I think we can take advantage of M73, Set Print Progress command, with the idea being that whenever the progress is set to less than 100%, the printer is "printing".

I toss this idea up for review. I like it because there is no need to determine a minimum fan speed and the one fan is kind of "multitasking"; switching between two roles. My thinking (now) is that whatever we do for the "combination" fan control will become the ACfan flavor of firmware. The PCfan flavor will stay the same for those who want to add a second fan.

[aegean-odyssey]

@see3d,

MIN_FAN_TEMP: 60-70°C MIN_FAN_SPEED: 25-65% (and possibly configurable)

Got it, good working numbers.

The mpmd_marlin_1.1.x firmware implements the PWM in hardware; it uses very little code space and there's virtually no execution overhead. In early versions, I used a PWM frequency of ~23kHz. It worked, but I think the fan is probably not designed to be switched like that. Currently, the PWM frequency is 33Hz and seems to be good. It's a simple matter to configure a wide range of frequencies.

In early testing, I noticed that the fan had trouble starting at low speeds, and that the threshold speed for starting the fan was quite a bit higher than the minimum speed setting that would keep the fan blades turning.

I ended up enabling Marlin's KICKSTART feature which "gooses" the fan on start -- when setting the fan, Marlin runs it at 100% for 100ms before settling in to the desired speed. The M106 command works much better with the KICKSTART feature enabled.

And every few months or so, I find myself window shopping for a digital storage scope; I've no particular need, but it would be fun to have one. I'm sure I'll give in to the temptation. I marvel at how affordable and accessible test equipment, development systems, prototyping services, etc. are these days. Makes me want to build something.

[see3d]

@aegean-odyssey The KICKSTART feature looks good. Often portable room fans make you go from high speed when first turning on before getting to the desired speed. Interesting that you were choosing a PWM frequency above or below hearing. The 76Hz in stock FW might be too low for the mechanical system to respond for noise. I did not notice a hum from it.

Interesting take on the two operating modes. Default on reset/power up to temp controlled covers the emergency stop with a hot hot end. It is a little harder for me to imagine all the scenarios for printing operations.

Taking one step back (I know, I often argue both sides) -- primary goal for unmodified single fan hardware is a minimum fan speed with a hot hot end for machine safety. Heat creep is a safety feature if it clogs the machine and requires taking the machine apart to fix it. Perhaps it belongs in the same module as the temperature runaway or in the watchdog -- If temperature is above the minimum and current fan PWM is below the minimum, set the fan PWM to the minimum. This will not auto turn off the fan after the temp falls below the minimum, but as my tests yesterday showed, neither does the stock FW. Secondary goal is full fan control for modified hardware with a PC fan. If the fan PWM is gcode RAM settable (not M500 saved), that also takes care of this. Perhaps a fan setting parameter: M106 S50 M25. Fan PWM minimum gets defaulted to 65% on reset or power up. Where else it might get reset is the only other question I have. Perhaps home (G28), just to start in a known state on every print. Normal users would never change the fan minimum PWM, because it would be a gcode that none of the slicers would generate. It would have to be in the start gcode on purpose. If zero or very low fan speed is desired for the first layer in the slicer, it would be up to the user to put a low number at the end of his start gcode. The slicer machine configuration would be unique (not stock), as the hardware is also unique.

I am sensitive to the desire to have this simple to implement and document.

I looked for a portable digital scope for a few years. I finally sprung for a Hantek 2D42 when I found it on sale at Amazon.com for $113. Not the best UI, but has a useful top frequency. My old HP scope is 200Mhz.

[aegean-odyssey]

I finally tracked down the crashing issue in the firmware -- stack overflow. And the offending command, G29 P0, which translates into G33 P1; the G33 command uses a lot of stack space. I manage to trim it back a bit along with the some other functions, and the crashing seems to be abated. Fixed quite a few other little issues along the way while searching for the cause. So, not directly related to Linear Advance or the -PCfan code.

I'm little surprised by the stack overflow, since the firmware uses a stack of 2.5k bytes. This should be plenty of space, though I'm not sure how the optimizer affects the stack usage.

[aegean-odyssey]

With 119r12:

-PCfan firmware controls the part cooling fan directly via M106/M107 codes. I think this is the firmware to use if your machine has a separate hotend fan.

-ACfan firmware combines the control of both a hotend fan and a part cooling fan into one fan. It operates in two ways:

• when the fan is disabled M106 S0/M107, the fan functions as an automatic hotend fan; on at 100% whenever the hotend temperature exceeds 50°C. The fan is turned off otherwise.
• with M106 S<n> ; where n is 1-255, the fan functions as a part cooling fan, with its speed controlled by the M106 code.

This new scheme for the -ACfan firmware should not negatively impact users of the previous releases of -ACfan firmware, as the change will likely go unnoticed. For users that want to control the fan speed during a print, this new scheme should accommodate as well. The catch, though, is how to turn the fan off for the first layers of a print, since the slicer will probably set M107 early on in the gcode. I think this can be accomplished by setting the fan speed (e.g. M106 S1) at the end of the start gcode to effectively turn the fan off until the slicer adjusts the fan speed later in the gcode. At the end of a print, disabling the fan M107 returns the fan to its automatic hotend fan function. All in all, I think this is a workable solution.

[see3d]

I just did a little test with the -ACfan on my machine with a separate part cooling fan. It worked exactly the way I needed. I heated it up to 240C, 40% fan and power cycled the printer. It came back up with the fan at 100% (AC mode). My part fan is close to the heater block, so it needs the protection of some air flow. The fan automatically turned off as the hot end dropped below 50C. Perfect functionality for me. Thanks.

[Adam-Beaumont]

@aegean-odyssey I flashed the new firmware yesterday (AC 10 amp variant) and have been successfully printing with it, fan behaviour is perfect for the compromise have having only a single fan. Linear advance is a great feature to have as well.

[aegean-odyssey]

@Adam-Beaumont, thank you for following up. Your issue, the ensuing discussion, and the varied perspectives and opinions led to a good result.

I think we can close this issue. If not, please feel free to re-open it.