search

openwisp / openwisp-firmware-upgrader

Firmware upgrade solution for OpenWRT with possibility to add support for other embedded OSes. Provides features like automatic retry for network failures, mass upgrades, REST API and more.
https://openwisp.io/docs/dev/firmware-upgrader/
Other
53 stars 60 forks source link

[Feature] Enable cancelation of stuck "in progress" mass firmware updates #208

Open tricksol opened 1 year ago

tricksol commented 1 year ago

On the mass update history tab where you see the running updates, there should be a button to cancel updates that seem to be stuck.

kosli commented 1 year ago

I have run into the same "issue" :-)

nemesifier commented 1 year ago

This is a rare corner case which shouldn't happen thanks to OPENWISP_FIRMWARE_UPGRADER_TASK_TIMEOUT (see relevant section in the readme).

It can happen though, if while an upgrade is in progress, a cold shut down is performed.

Now, how can we deal with this is the question.

Allow canceling or abort an in-progress upgrade can create other problems and more corner cases to handle. The upgrade can be canceled if the firmware is not being flashed yet, but once the flashing starts, there's no point in aborting, it's too late.

I think the problem here is that while an upgrade is in progress users cannot do new upgrades, which is very annoying. A quick fix is to open the django shell and execute this SQL query, just make sure there's no other real upgrade in progress!

UpgradeOperation.objects.filter(status='in-progress').update(status='failed')

A definitive solution could be the following: when a new upgraded operation is created if we detect another one in progress, let's check if it's old enough to be timed out, if it is we assume this corner case is happening, flag it as failed automatically and move on.

Now, I cannot promise we'll invest our resources in fixing this since we can easily work around it with 1 minute and it never happens to the OpenWISP instances we manage, but if anyone wants to contribute, feel free to implement this solution and send a patch.

mips171 commented 1 year ago

Hey, I opened a draft PR, haven't tested it yet though as I don't have my env set up. https://github.com/openwisp/openwisp-firmware-upgrader/pull/238/files