[FR] Idea for resuming prints after power failure.

[marcio-ao]

Prusa has recently released a printer with the ability to restart a print that has been interrupted due to a power failure. AFAIK, they implemented this feature by adding additional electronics that determines a brownout on the 120V AC side of the power supply and immediately causes the FW to write the current layer information to the EEPROM before all the energy in the power supply caps are drained.

In the past, folks have tried to implement this using only FW, by writing the current layer to EEPROM, but this has the downside of dramatically shortening the life of the EEPROM. Prusa's technique solves this problem, albeit in a very hardware intensive way.

I had a thought about an alternative software-only implementation. Perhaps a simple flag could be written to EEPROM at the start of the print, indicating that a print was in progress, and at the end of the print, that flag would be cleared. If a power failure occurred, the printer could use that flag to determine that a print had failed, but not where. The printer would then lift the Z axis all the way to the Z_MAX endstops and measure the distance covered. This could be used to determine the approximate location of the last printed layer and resume the interrupted print.

Does anyone have any thoughts on whether this might work? Obviously it would be less accurate than the other methods, but perhaps it may be good enough for some users?

-- Marcio

[marcio-ao]

Perhaps the EEPROM would not be needed at all. The user could manually invoke a resume operation, select the file they were printing, and then the printer would measure the Z distance to Z_MAX and resume printing from that layer.

[AnHardt]

Almost a complete layer could be printed twice - or not. Not very promising to me.

[JerryDurand]

That assumes you have Z-max stops.

There's a couple of  OLD technologies that I used to use, one is a shadow EEPROM.  On power up it copies everything to a mirror SRAM on the chip.  You can read and write this all you want.  On power down a small capacitor hooked to a dedicated pin powers a bulk copy of all the SRAM to the EEPROM.

Then there's magnetic RAM, first ran into this when I was designing hardware for SEGA.  Works just like an EEPROM, only it has a 1 billion cycle wearout.

Jump ahead to present time and I'll often use a small serial FLASH memory.  Data for it is held in a STRUCT.  Every couple of seconds I check for changes in the STRUCT, if there are any, I write it to the FLASH in the next available sector.  On power up I scan from the top down to find the last page written.

If the FLASH fills up, only then do I execute a bulk erase and then write the STRUCT to the beginning of the FLASH.

The above will get you many years of constant use, my first use of this was for a power metering device for a power company.

On 10/20/2017 11:30 AM, Marcio Teixeira wrote:

Prusa has recently released a printer with the ability to restart a print that has been interrupted due to a power failure. AFAIK, they implemented this feature by adding additional electronics that determines a brownout on the 120V AC side of the power supply and immediately causes the FW to write the current layer information to the EEPROM before all the energy in the power supply caps are drained.

In the past, folks have tried to implement this using only FW, by writing the current layer to EEPROM, but this has the downside of dramatically shortening the life of the EEPROM. Prusa's technique solves this problem, albeit in a very hardware intensive way.

I had a thought about an alternative software-only implementation. Perhaps a simple flag could be written to EEPROM at the start of the print, indicating that a print was in progress, and at the end of the print, that flag would be cleared. If a power failure occurred, the printer could use that flag to determine that a print had failed, but not where. The printer would then lift the Z axis all the way to the Z_MAX endstops and measure the distance covered. This could be used to determine the approximate location of the last printed layer and resume the interrupted print.

Does anyone have any thoughts on whether this might work? Obviously it would be less accurate than the other methods, but perhaps it may be good enough for some users?

-- Marcio

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/MarlinFirmware/Marlin/issues/8037, or mute the thread https://github.com/notifications/unsubscribe-auth/AHfZY7Lfq7dTDKPFHoYfKj2UNBtbiHNYks5suOa1gaJpZM4QBDff.

-- Jerry Durand, Durand Interstellar, Inc. www.interstellar.com tel: +1 408 356-3886 @DurandInterstel

[JerryDurand]

I just remembered another problem, on my Sunhokey not-a-Prusa i4 if I lose power the nozzle tends to glue itself to the print.

Also, your stepping motors might not align the same through a power cycle.  It's a curse of microstepping that it can be hard to get back to where you were.

-- Jerry Durand, Durand Interstellar, Inc. www.interstellar.com tel: +1 408 356-3886 @DurandInterstel

[marcio-ao]

@AnHardt: Yes, I am assuming that reprinting a layer or two may for certain prints be preferable than restarting from scratch. It probably would leave a visible seam or a weakened layer, so definitely not a perfect print.

@Jerry: You are correct that it would only work on printers with Z_MAX. Basically any sort of resume feature is limited to a printer that has Z_MAX; as in the case of resuming a print, homing to Z_MIN would never be an option. As for micro-stepping repeatability, Prusa's solution would also be affected, although their method is probably accurate down to a single layer while measuring Z travel could also be made worse by any flexing of the endstops.

[JerryDurand]

Time for a calculation (assuming you know exactly where you were when the power died):

My Z-axis uses 2mm pitch threaded rod.

A stepping motor has 200 full steps per revolution.  On power off it may go to the previous or next full step depending on where it was and any load on the shaft and momentum.  If it's moving fast it could jump a couple of steps.

So, 2mm / 200 = an error of +/- 0.01mm.  Not too bad, half a layer off.  As long as it's only off by half a layer it might work ok.

This is assuming the X and Y can be safely homed.

There will also be nozzle ooze to deal with, mine has quite a runny nose while idle but hot.

On 10/20/2017 12:50 PM, Marcio Teixeira wrote:

@jerry https://github.com/jerry: You are correct that it would only work on printers with Z_MAX. Basically any sort of resume feature is limited to a printer that has Z_MAX; as in the case of resuming a print, homing to Z_MIN would never be an option. As for micro-stepping repeatability, Prusa's solution would also be affected, although their method is probably accurate down to a single layer while measuring Z travel could also be made worse by any flexing of the endstops.

-- Jerry Durand, Durand Interstellar, Inc. www.interstellar.com tel: +1 408 356-3886 @DurandInterstel

[marcio-ao]

@JerryDurand: Yeah. Nozzle ooze is quite a problem. Prior to addition of the head park feature in the pause feature, I used to consistently see blobs in prints after stopping and resuming a print. I wonder if Prusa has enough power in the caps to park the head or retract -- I highly doubt it. If it does neither of those things, I suspect the feature may be considerably less useful than they make it out to be.

[vMeph]

sense you recovering a file and do a resume couldnt the machine performe before a homing to get the point 0.0.0 position and then go resume the file?

[marcio-ao]

@vMeph: Yes, a home is necessary, but in order to resume the print you need to know what position the print stopped. There are really only two ways to do this. Either you save the position to the EEPROM while printing, or you try to work it out based on where the print head is at once you restore power.

[teemuatlut]

I wonder if Prusa has enough power in the caps to park the head or retract

He has stated in interviews that they're able to lift the nozzle a few millimeters.

[marcio-ao]

@teemuatlut: Ah. That's probably what makes it feasible then.

I guess I'll close this FR out, since it doesn't sound likely that the feature could be emulated in software without their specialized sensing hardware.

[AnHardt]

For the curios of you: https://www.youtube.com/watch?v=VzHbmHPPYX8 Tom's interview with the printerman about the MK3 https://www.youtube.com/watch?v=RuvaSTg33r4 Tom's interview with the printerman about the EINSY. https://www.youtube.com/watch?v=hwNIzQLtHnU Josephs introduction of the MK3.

[fiveangle]

I wonder if Prusa has enough power in the caps to park the head or retract

He has stated in interviews that they're able to lift the nozzle a few millimeters.

Yes, Joeseph said in an interview they have routed P_FAIL signal out the PS to an interrupt-capable pin on the Einsy so that the moment it's asserted, they kill hotend and bed heaters, shutdown XY stepper drivers, write print progress to EEPROM, initiate a Z-lift then let it execute that until the caps run out of power. I think he mentioned something like 200ms.

Without smart power I don't see any way to do this. It could be "hacked" by a simple supercap added to the board power but by then you're already into the weeds.

[fiveangle]

Looks like 300ms: https://www.youtube.com/watch?v=yPoiT41WtsM&feature=youtu.be&t=4m02s

[Grogyan]

What you need is a low pass filter to an adc channel from the AC (high voltage) input, and have the firmware monitor this as part of it's primary ISR.

Mind you, I haven't taken apart one of these power supplies to see if a hack could be done. However, as a safe method, with a single io pin that asserts a logic level is a safer hack than opening a power supply.

On 21 Oct 2017 10:22 am, "Dave Johnson" notifications@github.com wrote:

https://www.youtube.com/watch?v=yPoiT41WtsM&feature=youtu.be&t=4m02s

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/MarlinFirmware/Marlin/issues/8037#issuecomment-338325488, or mute the thread https://github.com/notifications/unsubscribe-auth/AGULGgJv-WSHeDGx-cQem59Foh9mJ4N3ks5suQ8igaJpZM4QBDff .

[AnHardt]

More information --> less wrong guessing. https://github.com/MarlinFirmware/Marlin/issues/8037#issuecomment-338323492 and follow ups.

EDIT: Sorry. This was the wrong tread. #8042 (IDEAS FOR POWER OUTAGE FEATURE) was meant.

[Grogyan]

Mains Power Loss Detector Simple circuit to alert a board that Mains Power has been lost. Just use a a LPF to a micros' IO pin from the buzzer Though this version has been "tested"

[dot-bob]

A simple solution that should work on most boards is to use a DS1307. They are cheap, use I2C, are very low power, and have 56 bytes of battery backed ram with unlimited writes. They would be very easy to wire in. https://www.aliexpress.com/item/RTC-Real-Time-Clock-DS1307-module-with-battery/32530897478.html

You could store last completed move, x, y, and z coordinates. You could then detect an unfinished print by reading the battery backed ram and allow the user the option to recover. Most printers will hold their z position when power is removed so just home X and Y (probably after reheating).

[Grogyan]

@dot-bob On 32bit boards there is an SD card for the firmware that can be used for storing bed mesh, X-Y-Z, and last known position of the end effector. On 8bit boards, the SD card for gcodes could be used. Essentially, not requirring additional hardware above the need for loss-of-power detection. In my previous post, I mentioned a potential solution.

As a "hack", one could crack open a 5V power supply adapter, soldering a few components rats nest style. And feed the signal, AND ground from that to the processor via an available header connector.

The main bulk of the problem is to know where in the gcode the printer was before power was lost, save that, move the effector away a bit, then once power has been restored, load the saved data, home X and Y, and resume print. If the effector was raised, that amount would also need to be taken to account when resuming

Edit: Also need to realize that gcodes don't have line numbers, so, in RAM need a counter for the line number - trivial thing

[dot-bob]

@Grogyan The main reason I suggest the DS1307 is that it is lightweight and wouldn't require much CPU time which is important, especially for the AVR 8bit. The idea would be to continually update the RAM during printing. Writing to the SD card is slow and expensive. The SPI bus on the printer is pretty congested with the SD card and display. If it is bitbang then it is even worse. On 32bit that may have SDIO it might be another story.

There are many power supplies out there that include brownout detection. My meanwell supply on my primary printer has this. Another option would be to interface with an UPS like a computer does for brownout detection. I think there are still those with serial ports or a digital IO interface out there.

Another simple solution would be to use a battery backed RPI and have support in Octoprint and or Repetier Server. They know what command was last acked and the z-location. Just heat, home X and Y, then resume. If anything this could be used when a print fails because of communication interruption due to the buggy RPI usb stack.

[Grogyan]

I'll leave the debate as to what slow or fast in this scenario to those who know more than I.

What I do know is that seeing as the remote display is already being utilized with an SD card, through the hardware SPI. So to switch from read to write, is pretty insignificant, then write a few bytes. I2C on the other hand requires a 64bit address (8 bytes) followed by the the rest.

Loss of power is different from brownout. A UPS is expensive Having a RPI would also be useless, even with a backup battery, unless you have a home router that also has a backup battery.

In fact, if the devs can put factors in firmware as known values such as Z-Clearance, and bed mesh that is RAM already pushed to the SD when probing has finished (those of us that still do bed leveling the old way, like me, ie non-UBL and non-manual-Mesh), then all that needs to be written is literally just the line number from RAM, which would be counted, I guess, from the ISR routine. So I don't know, 8 bytes + 2 bytes ish for I2C vs just 2 bytes on Hardware SPI?

Maybe Josef is right with sacrificing a few write cycles on EEPROM, I mean, realistically, how often do blackouts occur?

[AnHardt]

The main bulk of the problem is to know where in the gcode the printer was before power was lost

Right. At first there are two cases. Reading from sdcard and from the serial. For the serial i have no good idea how to point at the last/next gcode. Line numbering is not usable. Likely we have to store the complete line of gcode. With the sdcard we could point to the position in the file. Then we have to think about the planer buffer. I guess we have to finish at least the current move in the planer buffer, better the complete g-code. On systems with move subdivisions (DELTA, SCARA), or printing arcs this is not the same. When we only remember the last parsed code, we lose the complete content of the planer buffer. Alone printing a long straight over the complete bed could last several seconds. In this case there is no chance to finish a single line in the planer buffer.

We'll see what Prusa does, when the source is open then.

[dot-bob]

Lets say we don't know when the power is going to go out so we store every time we process a line of gcode. With I2C and the DS1307 there is two bytes of overhead per command (one byte i2c address, one byte command address). Lets say we simplify things store a file number (one byte) and file position (8 bytes or 64 bits). This can be accomplished with a single write command on I2C so 11 bytes. At 100kbs this is 110uS.

Let's look at SPI and the sd card. Writing to the SD card has a lot of overhead as it is a multi layered operation, we have commands to the SD card and reads and writes to the filesystem. Here is a simple example and bear with me it has been a while since I have written a driver for a MMC/SD card, if recall correctly the overhead for single MMC command is approx 6 bytes (preamble (1), command (4), ecc(1)). Since we don't know when the power is going to be lost we need to open and close the file every time we want to update it else it may be corrupt the fat upon power loss. So we need to initiate a fat read, supply a filename/handle i.e. open a file, seek (all this 512 bytes of overhead possibly more). Say we write the same data as above to be efficient (9 bytes), close the file and update the fat (512 bytes or more of overhead depending on implementation). We then need to wait until the card reports the write is complete so we can turn around and read more data from the card to continue to fill the planner buffer. Just this can be between 500uS to 35mS depending what the SD controller is doing and depending on the speed of the card. So neglecting command overhead as it is insignificant to filesystem reads and write we have 1024 bytes/4000000 MHZ=256uS -> Overhead 255uS + 500uS = 750uS best case. Yes this is oversimplified but when looking at efficiencies the SD card and filesystem it is nearly one to two orders of magnitude slower.

Now if you have brown out detection (I refer to it as brown out detection as that is what it is called in the microcontroller world) you could use the SD card as you wouldn't need to write every move to disk during run time as long as you have enough prewarning to finish the write to the SD card.

[marcio-ao]

@AnHardt : It would seem to me as if Marlin would only have to handle the case in which SD card printing was resumed. In the case in which printing was done via serial, I would imagine the resume functionality would be implemented in the host software. Likely there would need to be an extra GCODE to ask the printer the position at which the print was interrupted and the host software would have to interrogate that to find out where to resume printing from.

In any event, it seems like the two big issues to solve are 1) to store the position in non-volatile RAM and 2) to move the head away from the print. The latter is the much harder problem and seems to necessitate the power supply tricks Prusa is doing. The seems to be no other way, unless it maybe involved a hacky mechanical solution, such as a spring loaded print head with a solenoid to keep it engaged until the power failed...

[Grogyan]

A mechanical solution would be to have an inline relay on the AC mains side, still requires a battery or supercap to hold enough charge to signal the controller of loss of power.

@dot-bob I think you are still confused with brownout vs LOP. Brownout refers to the minimal amount of power required for stable operation, slightly below this limit, and the device an potentially fall into an instability state, this state is called brownout. LOP detection will NEED to be triggered much earlier before a brownout can occur.

[dot-bob]

@Grogyan I am referring to the brown out detection peripheral that most micro's have for voltage supply monitoring or a de-asserted power good signal. They often give the option to fire an interrupt to allow you to perform some cleanup before all power is lost (like write to flash, toggle gpio, switch to backup or alternate power like a battery).

If you are wanting something for mains detection why not use a cheap off the shelf power monitor? You could measure voltage or current and trigger when it drops below a threshold. Option 1 Option 2 Option 3 It would be simple to tie these onto a GPIO interrupt, tune it to detect a power drop, and perform the same function as Prusa MK3 (cut power to the heaters, move z, and save position). That's as long as they react fast enough to voltage drop and your supply has decent caps. I actually might have some of these modules kicking around here in my lab.

[JerryDurand]

A typical SMPS (standard modern power supply) has a guaranteed hold-up time of only 20mS but most are good for longer, especially when partially loaded. A UPS will normally switch in half that time.

The simplest seems to be a UPS with a status output (often one pin used on an RS-232 line for the ones that don't use USB).  On my system I just have a UPS and hope the power isn't out that long. You wouldn't need a large UPS as long as you could safely part the print in say 5 minutes.

We had three outages in a month ending a few weeks ago, all quite long (one was all night).  Unusual for us but we lost 2 out of 3 prints due to the UPS going dead each time.  Our neighborhood has two new transformers. Hopefully they accounted for all the new electric cars plugged in around here or I may need to install a jumper wire from the neighbor's car to keep the prints running.  :)

On 10/23/2017 10:36 PM, dot-bob wrote:

@Grogyan https://github.com/grogyan I am referring to the brown out detection peripheral that most micro's have for voltage supply monitoring or a de-asserted power good signal. They often give the option to fire an interrupt to allow you to perform some cleanup before all power is lost (like write to flash, toggle gpio, switch to backup or alternate power like a battery).

If you are wanting something for mains detection why not use a cheap off the shelf power monitor? You could measure voltage or current and trigger when it drops below a threshold. Option 1 https://www.aliexpress.com/item/XD-93B-5A-range-Single-phase-AC-Active-output-Current-transformer-module-Current-sensor-free-shipping/32259914518.html Option 2 https://www.aliexpress.com/item/A01B-Single-phase-ac-active-output-voltage-transformer-module-Voltage-sensor-module-free-shipping/32260479182.html Option 3 https://www.aliexpress.com/item/1PCS-WCS1800-Hall-Current-Sensor-35A-Short-Over-Current-Detector-Protection-Module/32803032597.html It would be simple to tie these onto a GPIO interrupt, tune it to detect a power drop, and perform the same function as Prusa MK3 (cut power to the heaters, move z, and save position). That's as long as they react fast enough to voltage drop and your supply has decent caps. I actually might have some of these modules kicking around here in my lab.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MarlinFirmware/Marlin/issues/8037#issuecomment-338879229, or mute the thread https://github.com/notifications/unsubscribe-auth/AHfZYw_2rimH0RqUHF_Bz3-8f7hqd8Trks5svXdGgaJpZM4QBDff.

-- Jerry Durand, Durand Interstellar, Inc. www.interstellar.com tel: +1 408 356-3886 @DurandInterstel

[AnHardt]

Option 2 looks most promising to me. We want to detect voltage, not current. When heaters, motors, fans, light is off, we don't draw much current. We definitely don't want false positives. We want a digital output, not an analogue. We have to trigger an interrupt. Until we had polled the pin half of the energy could have been gone. The ADCs convert slowly and you have to poll them.

Edit: On the other hand all digital inputs act as a "schmitt trigger". So, if the analog signal is falling deep enough and is never above the max input voltage - no problem.

[ljmarent]

I've sent a cute little PCB to SEEED that will be located inside my printers PSU cover. It uses a voltage detector (1662-1219-1-ND) to sense when the +12V supply has dropped below +9V. When this happens, it cuts power to the heated bed using a (BTS50055-1TMA) and makes a logic level signal available that indicates a brown out or power fail is in progress. There is quite a bit of energy stored in the bulk capacitors of the SMPS, which if the MCU catches the signal in time could enable the writing to EEPROM and possibly moving the hotend away from the print, before complete powerloss. One other thing I'd like to add..aside from the Dallas DS1307 mentioned above, which requires a battery, one could use a small I2C FRAM device like (865-1274-1-ND) which for practical purposes has the same endurance and ease of use as SRAM. No erasing, wear-levelling, block management, etc. like floating gate based storage devices require. SRAM like behaviour without a battery. While the data sheet lists the endurance of these as 1e10^13, I've seen these go to 1e10^14.5 before giving up the testing. You could write every move the printer ever makes and you'd still get no where near the endurance limit.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.