Open bradtpt opened 4 years ago
Thanks for gathering the debug logs, and taking the time to check against the vendor software.
I'm scratching my head on this one, too. From a quick glance at git diff -w v2.6.3.. -- drivers/bcmxcp*
, it looks like the majority of the changes were enhancements to read and parse additional fields from the UPS. It is certainly possible that nobody scrutinized the output of the driver connected to a Powerware 3105 after those changes were made (and the changes seem oriented towards other models, including multi-phase UPSes).
There are a few different angles you can take. I think a lot of them involve being able to recompile the driver, so I would recommend picking one of the boxes as your development machine, and making sure you can rebuild from the latest code on the GitHub master branch. (I don't expect it will magically work; it just means we can debug the compilation and driver fixes in two independent steps.) For pfsense, I think you should be able to adapt the FreeBSD instructions on the wiki. I am less familiar with BSD in general, but I might be able to help you muddle through common problems. The Pi should work with the Debian instructions. Aside from the initial compilation, the speed of the Pi should not be too much of a limiting factor.
Note also that we have some protocol documents linked here: https://networkupstools.org/ups-protocols.html#_eaton
The workhorse of any driver is the upsdrv_updateinfo()
function, which is in drivers/bcmxcp.c
for your UPS. Since your UPS seems to be working well with the vendor software, we can probably assume that the received bytes are valid. (However, I don't know anything about LanSafe, so if there is another protocol besides XCP that LanSafe could be using, that assumption is not necessarily valid.) Either way, it might make sense to work through the parsing code, maybe inserting some extra debug statements as needed. I will say it is also possible that a combination of the underlying NUT and OS USB code is at fault, since OS distributions have been making subtle changes to the USB stack that NUT seems particularly sensitive to (see,, for instance, the fallout from issue #300).
@clepple thank you so much for the quick and detailed reply. Turns out I have a few of these Eaton x-slot series UPS laying around and as curiosity got the better of me I ducked into work and grabbed a couple of different models I had tucked away. I now have the following models as test subjects sitting under my home office desk to play with:
bcmxcp_usb
reports false alarms and other inaccurate data with all 3 and I've confirmed that all 3 test and behave as they should using the vendor's Lansafe v6.06
software which is period accurate for these, I note Eaton don't even list the 5110 and 3105 series on their website any more for support. I've had intermittent results with ups.status
simultaneously reporting OL & OB when the UPS has been mains powered and should be reporting OL only, after power cycling the UPS via its power switch a couple of times I've found this will report correctly albeit still with a persistent ALARM state due to the huge list that it always has. Another interesting 'feature' I found today with the 5110/500VA unit is when it's OL and I plug a small load into it (just a laptop) the UPS would for some reason report LB (low battery) and immediately nut would do its thing and initiate a system shutdown on my 'test pi' which I have powered independently. I had this happen multiple times so there's clearly something not right here. I won't worry at this stage with posting debug logs from the other models as it'll only serve to clutter the thread and not provide anything useful. If one of the 5110's had of worked correctly I think another log would be relevant so we could compare.
I had already started to follow the directions for compiling nut on my pi earlier today as I figured we'd have to go down this road so I'll finish doing that and start digging. Thank you again.
Would also like to confirm this issue on my POWERWARE UPS 1500i (5110)
Hey Guys, Late to the party, but, what kernels are you using? 5.x? 32bit/64bit?
I recently upgraded a machine from slackware 14 to slackware -current (aka 14.2"plus" aka 15.0), I used the same 2.7.4 nut version built the same way I always have, that worked flawlessly, until the new OS/kernel was installed, so perhaps this is a compat issue somewhere with kernel?
Here, it runs on a 32bit using 5.10.3-smp - but this is a recent upgrade, it also had same false reports including the false "replace battery" alerts and false constant cycling on-battery to online with earlier 5.4.x kernel
did anyone manage to resolve this issue?
I guess no one ever got this fixed?
Just to add another data point- running an Eaton Powerware 5110 with fresh batteries. No alarms reported by Intelligent Power Manager when connected over local USB. UPS supports load no problem, it's even kept my server and router up during actual power outages a number of times already. Nut is however reporting all the faults. UPSC dump below:
battery.charge.low: 22
battery.charge.restart: 244
battery.charger.status: charging
battery.energysave.delay: 76
battery.energysave.load: 204
battery.packs: 1
battery.runtime.low: 1
battery.voltage: 27.2
battery.voltage.low: 20
device.mfr: Eaton
device.model: POWERWARE UPS 1000i
device.part: 103004268-5591
device.type: ups
driver.name: bcmxcp_usb
driver.parameter.pollinterval: 2
driver.parameter.port: auto
driver.parameter.synchronous: no
driver.version: 2.7.4
driver.version.internal: 0.31
input.frequency: 49.9
input.frequency.high: 55
input.frequency.low: 45
input.frequency.nominal: 50
input.quality: 0
input.transfer.boost.high: 216
input.transfer.high: 280
input.transfer.low: 186
input.transfer.trim.low: 260
input.voltage: 244
input.voltage.nominal: 240
output.current: 0.0
output.current.nominal: 2.5
output.frequency: 49.9
output.frequency.nominal: 50
output.phases: 1
output.voltage: 242
output.voltage.nominal: 240
ups.alarm: STATIC_SWITCH_OVER_TEMP CHARGER_LOGIC_PWR_FAIL DC_LINK_UNDER_VOLTAGE STATIC_SWITCH_FAILED BATTERY_CURRENT_UNCALIBRATED INVERTER_AD_SELFTEST_FAIL INVERTER_RAM_SELFTEST_FAIL INVERTER_CPU_SELFTEST_FAIL INVERTER_PHASE_BIAS_ERROR DC_LINK_BLEED_COMPLETE LARGE_C
ups.beeper.status: enabled
ups.description: Line-Interactive UPS, Single Phase
ups.firmware: Cont:00.50 Inve:01.50
ups.load: 7
ups.mfr: Eaton
ups.model: POWERWARE UPS 1000i
ups.power.nominal: 1000
ups.status: ALARM OL RB
ups.test.result: Done and passed
Apart from the fault reporting, everything seems to work OK.
@clepple thank you so much for the quick and detailed reply. Turns out I have a few of these Eaton x-slot series UPS laying around and as curiosity got the better of me I ducked into work and grabbed a couple of different models I had tucked away. I now have the following models as test subjects sitting under my home office desk to play with:
- Powerware 3105 700VA
- Powerware 5110 500VA
- Powerware 5110 1500VA
bcmxcp_usb
reports false alarms and other inaccurate data with all 3 and I've confirmed that all 3 test and behave as they should using the vendor'sLansafe v6.06
software which is period accurate for these, I note Eaton don't even list the 5110 and 3105 series on their website any more for support. I've had intermittent results withups.status
simultaneously reporting OL & OB when the UPS has been mains powered and should be reporting OL only, after power cycling the UPS via its power switch a couple of times I've found this will report correctly albeit still with a persistent ALARM state due to the huge list that it always has. Another interesting 'feature' I found today with the 5110/500VA unit is when it's OL and I plug a small load into it (just a laptop) the UPS would for some reason report LB (low battery) and immediately nut would do its thing and initiate a system shutdown on my 'test pi' which I have powered independently. I had this happen multiple times so there's clearly something not right here. I won't worry at this stage with posting debug logs from the other models as it'll only serve to clutter the thread and not provide anything useful. If one of the 5110's had of worked correctly I think another log would be relevant so we could compare.I had already started to follow the directions for compiling nut on my pi earlier today as I figured we'd have to go down this road so I'll finish doing that and start digging. Thank you again.
I've had intermittent results with
ups.status
simultaneously reporting OL & OB when the UPS has been mains powered and should be reporting OL only,
I would also like to confirm I am seeing this issue with a Powerware 5110 500VA but only on Raspberry Pi's. I have even gone back to Debian Stretch and see the same issue. Any other architectures have no issues. I am monitoring here using a Banana Pi R2 with Debian and Kernel 4.19 and have also tested on Debian installed on a desktop pc here and see no issues at all. Only on Raspberry Pi's Model A, 2, 3 and also 4 which I have here. I see OL & OB at the same time and also got OL OB RB and a bunch of others all at the same time. Sometimes everything is good but after restarting the driver it all falls over. usbhid-ups works fine on Raspberry Pi as we also have a Dell UPS1920W which we use a Pi to monitor. It seems to just be this driver. I have also compiled and tested up branch libusb1.0+0.1 thinking libusb might be the issue but there was no change with the ups.status showing ALARM OL OB RB all at the same time.
I am also seeing this on a 5110 500VA, on x86_64 ubuntu 24.04. Happy to do some debugging if needed since it isn't in a production state at the moment
I think something like this was addressed (or at least discussed) in the past couple of years, in some other ticket. IIRC @arnaudquette-eaton wanted to discuss it with the HW/FW team internally, whether it was a bug on that level.
FWIW, @davidmonro : can you check which NUT version you have? Distros tend to lag behind, you might want to try testing with current codebase (not necessarily installing it over the packaged binaries, unless you like the custom build better) - see https://github.com/networkupstools/nut/wiki/Building-NUT-for-in%E2%80%90place-upgrades-or-non%E2%80%90disruptive-tests
Latest NUT release was 2.8.2 and some months of improvements have accumulated since then on master branch.
Looks like I currently have 2.8.1. I'll build from master and see what happens.
For kicks, I wonder if it is worth pointing other drivers at the device, e.g. if usbhid-ups
or nutdrv_qx
would find something they can understand. You may have to explicitly specify a set of options including vendorid
, productid
and subdriver
(and/or protocol
in QX) if the IDs are not hard-coded as "known" in any of those subdrivers.
I have found the bcmxcp_usb driver reporting invalid / implausible values plus alarms that don't exist for the two Eaton Powerware 3105 UPSs I have.
I have two of these UPS units on different systems, one as per the default package install of pfsense-2.4.4, (nut version 2.7.4) and the second on a raspberry pi I setup for testing with Raspbian Buster Lite running nut 2.7.4 installed from the package repository (ie just apt install nut), I'm getting similar results from both.
I have highlighted the values which I believe are incorrect in the rest of this post.
Here's the output from upsc ups on the pfsense machine:
and from the pi we see this:
This last one is clearly incorrect as it's reporting the UPS status as OL and OB simultaneously! This same UPS plugged into Eaton's own Lansafe software on a windows box reports no issues, plus the UPS behaves as it should and will support the load when power is removed quite happily.
Debug level 5 output from the driver shows us further details of the alarms that are allegedly active. Is this something I'm likely to be able to fix if I could get into the driver and make some changes? I'd be happy to start looking at the code but I have limited time and experience with nut for my time to be effective I think.
I find it surprising as this exact model has an entry on the supported hardware list yet both units are functionally ok, perhaps there's a different firmware version in the UPSs I have here compared to the ones used to validate the driver as per the link.
Any direction or input would be appreciated, I'd really like to be able to monitor and control these with nut and as stated earlier if I can be pointed to the relevant bits of the code I'll put in some effort myself if need be.