Charging slow on Werner

[gestom]

Hi,

since yesterday, Werner's charging rate is very low. The charging current is typically below 2A and is fluctuating wildly. The 'charger_status' field internal_error_flag is True and 'const_current_mode' is fluctuating between True and False. Battery level was about 77% when I checked. We recorded the 'battery_status' for a few hours - the rosbag is available here: https://lcas.lincoln.ac.uk/owncloud/index.php/s/FbRBWMZnZQQjVcQ.

Can we do anything to figure out why is the charging slow ? normally it's about 10A.

Best,

Tom

[marc-hanheide]

@creuther as our valuable MetraLabs contact, could you forward this to the relevant people, please?

[creuther]

@marc-hanheide Of course, thanks for notifying me. Without knowing any details, I think the charger would need repair or even replacement so can you give me a c/o address so we can expedite the whole process a bit and get a new charger sent out? At least that is what I suspect will happen, I can only confirm it tomorrow morning though.

[marc-hanheide]

Thanks for coming back to us so quickly. If there's any diagnosis we can do, please let us know.

[gestom]

Maybe that the reason for slow charging is caused by the additionally installed PCs inside of the robot, which might cause the robot to overheat. We have never observed the charging slowdown with a robot that had it's skirt off. To test it, I let the other robot (LINDA) patrol 'naked' to deplete its batteries to 20%, put the robot skirt on, started the deployment systems, placed a digital thermometer inside of the robot (at the top part of the skirt area, approximately at the level of the PTU controller) and let the robot charge in a room with controlled 30 °C temperature. First, the robot would charge at 10 A, but as the temperature inside of it would increase, the charging current would drop to +-2A. After the battery was charged to 35%, I removed the skirt and set the room air condition to 19°C. The charging current started to slowly increase again, see the attached graphs.

[marc-hanheide]

Wow, thanks, guys. This is on Linda, so the question is if the effect is the same on Werner. How did you measure the temperature? Have we got any idea what the temperature profile is on Werner? Also, have we got a chance to measure the temperature in the battery cells themselves? It would now be interesting to see if we start charging without skirt and at 19℃ and obtain a very different profile...

[creuther]

Great analysis, thanks for that! I have created a service ticket and have some diagnostics instructions for you.

First of all, we suggest flashing the latest firmware of the charger as it does improve the stability of the charging "arc", but obviously it won't solve the temperature dependency if that turns out to be the cause. We don't have the best track record with firmwares and usually believe in not changing running systems, but I personally feel very confident about our latest firmware bunch. It includes a couple of fixes, among others it noticeably improves the odometry stability (thanks to the great investigative work by @cburbridge). I will write an email about this to the STRANDS mailing list within the next hour.

As for diagnostics, it would be more helpful for us if you could create a MIRA tape of the /robot/charger/BatteryState channel (topic) while this phenomenon is happening. Doing that is fairly easy, you can just connect to the locally running MIRA instance (it should by default be port 1234) by using miracenter -k 127.0.0.1:1234 and then see whether there is already a "Recorder" tab in the bottom center workspace area. If not, just add the view part by going "Window -> Show View -> Tape -> Recorder". In the widget, press the green "+" button and select the /robot/charger/BatteryState channel. After that you just need to press the red "Record" button and specify a filename, then it'll record the channel. Exactly the same as ROS bags, but readable by our diagnostics tools ;-) Also you need to press "Stop" when done for the tape to finish writing.

There is also a way to access the persistently stored errors of the charger, by using the RPC /robot/Robot.savePersistentErrors(std::string filename) This can be called from miracenter using the RPC Console view (if not already added, add in the same way as above). This persistent error file, if there are any stored, would be helpful as well for the diagnostics.

[creuther]

Just in case, here is the file sent to the STRANDS mailing list. scitos_firmware_upgrade_17032016.tar.gz

[gestom]

How did you measure the temperature? Using the Temper1F sensor, see the metallic piece in front of the router.

Have we got any idea what the temperature profile is on Werner? No. But hospitals might have some (contact less) thermometers and people who can use them. I am not sure it these work on robot skin as well.

Also, have we got a chance to measure the temperature in the battery cells themselves? It would now be interesting to see if we start charging without skirt and at 19℃ and obtain a very different profile... We might try. However, this might not be the case of the batteries only, but the other charging circuitry as well.

[gestom]

I am about to try the firmware upgrade. However, how do I backup the previous firmware ?

[marc-hanheide]

I recall haven't read somewhere "you can't", but I can't find it...

[gestom]

so we might end up with a broken system ?

[gestom]

I will do some outstanding experiments before then :)

[gestom]

OK, I did the firmware and run the robot around with and without the cover. Navigation was working fine and the charging current fluctuations were reduced. The average charging current was slightly higher than before the update, but one could still see that with the cover on, the charging took much longer time.

[marc-hanheide]

So, would you then recommend we update Henry? We have an easter break at the moment anyway...

[gestom]

Actually, I would not recommend it. Although the charging current is not fluctuating after the update, its average value did not change much and the overheating problem persists. Rather than updating firmware, I recommend upgrading hardware and cut some additional venting hole(s) in the robot shell. I am currently running another batch of tests with the lower skirt removed. This might give us a hint where to cut.

[gestom]

So, removal of the lower skirt also helps, so IMHO we might start trying to drill a venting venting hole in the rear part of the robot 'lower' skirt. The question is if Metralabs can provide us with a few replacement plastic parts of the robot's shell and for how much ?

[creuther]

I can find out for you, what exactly do you need? Which parts? How many of them? In the same colour as Werner or just some generic colour?

Besides, what I find a bit odd is that the problem arose so suddenly. I can see from your measurements that there is some correlation with temperature, but before it was charging fine despite the exact same situation (shell on) right? If so, then I wouldn't rule out a problem with the charger yet. If you could do the diagnostics I posted above, we might be able to find out more.

[gestom]

Maybe the problems are not observed that much because when not used in deployments, the robots were running half-naked. Moreover, I think that we observed the overheating problems last year as well, but this time it might be more severe. I have plenty of rosbags from the experiments, cannot they substitute the MIRA tape ? The topic is the same.

[gestom]

miracenter -k 127.0.0.1:1234 reports that it 'could not connect to the framework'

[creuther]

I had a look at the scitos_mira node, apparently the MIRA framework there doesn't start with a port open for incoming connections unless you specify the server_portvariable. But either way, you can also run a separate MIRA instance when the scitos_mira node is not running (as only one connection to the CAN bus is allowed).

miracenter SCITOSConfigs:etc/SCITOS-mapping.xml --variables robot=SCITOS-A5,canType=MLCAN

The reason we cannot easily use the ROS bags instead is that we have tools that read the tape files (and channels within) for analysis. I guess it would be possible to build a converter but as this can't be done generically, we have never put too much priority into it.

[gestom]

OK, the tape recorded by @Jailander is at labe.felk.cvut.cz/~tkrajnik/miratapebattery.tape. The persistent errors of the charger are at labe.felk.cvut.cz/~tkrajnik/persistent.errors. We will probably need one 'bottom' skirt and one 'back-middle' skirt with Linda-blue color.

[creuther]

Thanks, we are analyzing the tape and error files and will get back to you. As for the case parts / skirts, I talked to my boss about it. Unfortunately we can only sell the large middle parts as a whole / together as they are commissioned in that way. So the two middle parts and the bottom part including paintwork would cost 750€ net altogether. Let me know whether you want a formal quote or how you want to continue. Otherwise I'll be back in touch about the charging irregularities.

[marc-hanheide]

OK, as this tape was on Linda, we shall create one from Werner as well.

[cdondrup]

I recorded a bit of data: https://lcas.lincoln.ac.uk/owncloud/index.php/s/VwGbXIiOWIBqhk0

This contains 4 files:

• persistenterrors.txt which is the persistent error log
• a rosbag file which contains recordings of the charging topic while the robot docks and undocks
• charger.tape a mira tape of the robot being on the charger
• drop.tape a mira tape that hopefully shows the drop in current when docking. This was created by sending the robot to the charger and when it docked, immediately killing the scitos node and starting mira centre. It does not have the docking in it but should start with the high current we observed and then drop.

[marc-hanheide]

Here is the plot of the rosbag @cdondrup created:

Sorry, about the x axes... it's samples, so roughly it's second * 10.

I plotted the current (as before), and we see a similar behaviour. indeed we went through a number of docking and undocking cycles, where we always observed pretty much the same behaviour. After docking the charging started at a high rate (so negative current, red line, at about -8A, one such docking starts at 2362 in the plot). At that point we also see the Voltage (green line) to steeply go up, but interestingly slowly, and not directly with the current. At 3537 we see the dropping occurring when the charger seems to go give less power to the robot, until levelling out at about -3A from 4524 onwards. We have then repeated this procedure three times, quickly undocking and re-docking to see if it makes any difference in the speed it declines. One hypothesis was that if we charged before, the unit might still be warm and the decline in charging speed should be faster than at the beginning when the robot was away from the docking station for longer... this seems to be the case as indeed in the last cycle the "good" charging duration was only about half long as in he first cycle.

Note: This time it continued charging because we had a very low load on the robot, not runnng anything but the bare necessary things to run the system. On full load, those 3A would be eaten by the computer running stuff. That's what we currently see: The current is just about enough to power the processing, but not to charge. It would be perfectly enough if the higher charge we see at the beginning would be maintained.

[marc-hanheide]

So, @creuther is this helpful? @gestom, I think this all confirms what we were suspecting, right?

[marc-hanheide]

One last note... the voltage profile is a little surprising to me, which is why I also included it, maybe it help diagnosis.

[creuther]

@marc-hanheide @TobKoer We sent you a new charger today, I'll provide you with instructions on how to change it soon.

The additional information you provided is quite interesting as well, I forwarded it to our hardware guys. I'll keep you posted!

[marc-hanheide]

I forgot to add, that this was all measured when the robot was about 50% charged, so to make sure we don't end up in the kind of tickle charge when it is nearly full...

[cdondrup]

"tickle charge" hihi XD

[marc-hanheide]

@cdondrup also happy to amuse you.

@creuther are you then confident that the profiles shown result from a faulty charging unit? In other words, shall we be optimistic that a new charger will resolve the situation? Or should we consider further alternatives?

[creuther]

@marc-hanheide Somewhat confident but I agree that the behaviour observed is kind of odd. Waiting for an expert analysis of the data but from what I see, the robot experienced cell undervoltage quite a few times. It does not look like any of the battery cells dropped below the critical threshold of 2.0V (after which they are considered permanently damaged by the manufacturer of the cells), so they should be okay.

[marc-hanheide]

@creuther the charger has arrive, but we now urgently need the instructions on how to fit it.

[marc-hanheide]

So, new charger, old story: When initially starting the charge it started with about 5-6A charging, not the 8A we saw previously, but that was a "bug" anyway, as the charger should be limited to a total of 10A according to @creuther.

Then, we saw it slowly drop as temperature likely got up. After this initial drop, I recorded the whole profile overnight:

The X axis shows hours, with 0 being midnight local time. So, we can see that henry undocked as intended shortly after 9am when the charging finished, but from this plot I can also guess that @cdondrup probably started logging at 8:20, as that’s where the charging goes down to 0 again. In short, it seems that we didn't win anything.

At the beginning of this plot, /charger_status showed this:

---
header:
  seq: 63144
  stamp:
    secs: 1460400047
    nsecs: 169556000
  frame_id: ''
charging: True
empty: False
full: False
derating: True
charging_disabled: False
const_volt_mode: False
const_current_mode: False
internal_error_flag: False

The derating: True was suspicious and @creuther said it might indicate that the charger has reduced its power due to heat. This is to be confirmed.

[marc-hanheide]

So, we can also deduce that we loose about 10% per hour, and we gain (without logging, with it we seem to gain nothing!) max 5% per hour on charging.

[marc-hanheide]

Given our experience with both Werner and Linda I strongly advise the TSC deployment people to look at and consider this. We have seen little autonomous behaviour at aaf due to power problems. I suggest you do some tests with your robot under full load early to assess the charging profiles. @hawesie (at whoever is in charge)

[hawesie]

Oh poop, this is the first I've seen on this. @bfalacerda and you work with @creuther to see what happens with Betty? Would it help if we try to fit a cooling fan into the skirt somehow?

[Jailander]

@hawesie we are currently fitting a fan in Linda's lower skirt will send you a pic as soon as we get the part back.

[Jailander]

Also maybe its worth taking a look at Marc's Awesome launch action server to launch and stop components that are not necessary whilst charging.

[marc-hanheide]

The Problem is that the biggest drainer is logging for us. If we stop that any time we offer services at the charging station we have quite patchy data, but it seems we have to do that.

[agcohn]

and/or cooling holes drilled in skirt?

Tony

Prof A G Cohn, FREng, CEng, CITP, FAAAI, FECCAI, FAISB, FIET, FBCS School of Computing University of Leeds Leeds LS2 9JT 0113 3435482 www.comp.leeds.ac.uk/agc

-----Original Message----- From: Jaime Pulido Fentanes [mailto:notifications@github.com] Sent: 12 April 2016 12:37 To: strands-project/scitos_robot scitos_robot@noreply.github.com Subject: Re: [strands-project/scitos_robot] Charging slow on Werner (#72)

@hawesie https://github.com/hawesie we are currently fitting a fan in Linda's lower skirt will send you a pic as soon as we get the part back.

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/strands- project/scitos_robot/issues/72#issuecomment-208858194 https://github.com/notifications/beacon/AEGibIE0RzUIUbnfs4ABBpnjVj3IF mMaks5p24PEgaJpZM4HzJnu.gif

[marc-hanheide]

Yes, these options are all under consideration and being assessed. But we'd live advice from @creuther and metralabs.

[bfalacerda]

yeah, I've been kind of following what's going on. @cburbridge 's suggestion before leaving was to add a fan like this:

We'll do it next week when @maximeadjigble is back. I also think it might be a good idea to avoid using the second pc if possible, according to @creuther they are using quite a bit of power, and with both on we'll probably struggle to achieve full charges during night regardless of heating issues.

[marc-hanheide]

We considered taking the second one out, but the effects has been deemed minimal aus they use very little power when not under load (we have seen). Modern PCs are pretty good at power management, so having them there when they are not computing is almost as not having them, And hence, two PCs at 30% load end up as pretty much the same as one under 60% load. We have seen this also, when we don;t run anything, but the PCs idle, we get plenty of charge, so it is a load problem after all, and a heat problem.

[marc-hanheide]

The fan at that position is unlikely to cool the charger (which sits in the lower skirt above the wheel) effectively.

[marc-hanheide]

We may want to also add another heat sink to the charger, but again, best to hear advice from @creuther.

[Jailander]

Also I'm not sure you'll get much cooling from putting the fan there, unless, you have an escape on top of that skirt you will end up moving hot air only (which I admit still cools down but not as much), and not exchanging air with the outside of the robot.

[cdondrup]

So, we can see that henry undocked as intended shortly after 9am when the charging finished, but from this plot I can also guess that @cdondrup probably started logging at 8:20, as that’s where the charging goes down to 0 again.

I started the logging at around 8 while still on the charging station. So I guess you're right, that is what we see in the plot.

[bfalacerda]

The idea was to make the air flow up, there are some wholes when the skirt is on, and also on the neck and stuff

[marc-hanheide]

But @bfalacerda the charger is below the plate, so there won't be any circulation there.