Charging slow on Werner

[gestom]

Hi,

since yesterday, Werner's charging rate is very low. The charging current is typically below 2A and is fluctuating wildly. The 'charger_status' field internal_error_flag is True and 'const_current_mode' is fluctuating between True and False. Battery level was about 77% when I checked. We recorded the 'battery_status' for a few hours - the rosbag is available here: https://lcas.lincoln.ac.uk/owncloud/index.php/s/FbRBWMZnZQQjVcQ.

Can we do anything to figure out why is the charging slow ? normally it's about 10A.

Best,

Tom

[bfalacerda]

yeah, I see that. Just to confirm, these problems also happen with the skirt off?

[Jailander]

yes that is why we are drilling the hole on the lower skirt will go in a bit to get you a picture

[cdondrup]

The problems do not seem to happen with the skirt off, no.

[cdondrup]

OK, just to add to that. There still is an effect due to heat but the effect is smaller with the skirt off. So the charging is still reduce but not as much.

[marc-hanheide]

Position of the charger:

[marc-hanheide]

So, I guess if we improve the conduction of heat to the plate where the charger is mounted and then cool the air above it, it might have a good effect... Anyone every tried if the plate above the charger is warm/hot to the touch?

[bfalacerda]

ok, so if we manage to reduce the temperature inside the pc area when the skirt is on, we might not need to go to the bottom part? Anyway, if you guys find a solution this week we'll follow suit

[marc-hanheide]

It probably doesn't help that the GPU of the side PC is blowing its hot air right at the place where the charger sits...

[cdondrup]

Not using the GPU for anything currently. Can't be that hot...

[marc-hanheide]

Then, it might even have a cooling effect???

[marc-hanheide]

Anyway, can you guys check with Linda if the charger actually gets hot?

[Jailander]

Yes the plate gets hot, with and without the skirt I imagine the effect is reduced when the upper skirt is off because of the improved heat transfer. but the plate is always moderately warm

[Jailander]

I'll check in a second, we need a thermal camera ;)

[Jailander]

Ok Linda has a full battery right now so the charger is cold, I'll let her discharge for an hour and plug her and see after a while, however since the lower skirt is not here it won't be as warm as in other cases.

[marc-hanheide]

We have a thermal camera, ask Greg.

[Jailander]

haha I know, just realised we could have used it before I made the comment, but right now it doesn't make sense since we don't have the skirt. But I'll ask Greg for it when we have the part back.

[creuther]

I have been doing some asking around and had a look at Bob but it seems while I was doing this there were quite a few new posts. Anyway, I'll just continue with what I started typing before ;)

@marc-hanheide Unfortunately the life-percentage is not really that reliable, so it would be more helpful to see voltage plotted against current. What kind of services were running during the night, and what kind of power draw do you except the robot to have due to them? If the robot was still drawing around 6A then the numbers seen might make sense. Also, the charger per se should not need excessive cooling. The only time we have run into a temperature problem was with the HOBBIT project, which had a quite similar setup in terms of complexity and power consumption. The reasons there were heat build-up due to insufficient air circulation as well. So as @cdondrup also confirms that the problems do not happen with the skirt off, we have to improve the circulation / get rid of the heat build-up.

In terms of the derating flag, I have some pleasant news to deliver - there is yet another bug in the firmware at the moment that automatically sets the flag to true when the robot is charging. But heat will still most likely be the issue.

To see whether everything else in terms of charging is working correctly, I'd suggest again letting the robot charge with all computers shut down and then reading the current and voltage from the status LED display. It should read around -10A.

Bear in mind though that as soon the battery is full (voltage around 29V +-0.5V), the current will drop to 0 and then hover around small positive and negative currents.

But all in all it seems most likely that everything is in order and that the heat is the problem at the moment.

[Jailander]

@creuther in one of those charts @gestom made when we recorded the tape you could tell that the problem also happens with the skirt off just in a much lower scale, however, this could be happening on Linda because of the defective cell. Anyway solving the heat build up issue seems to improve charging.

[marc-hanheide]

Thanks @creuther, so given all this, have your MetraLabs guys any advice how to best get rid of the heat build up? What did you do in Hobbit? Taking the skirt off is no option. I agree that what we see now is mostly consistent, and when it was cool, we saw indeed that 10 A - current_drawn = current_charge.

So, the charger is functioning as it should, the firmware apparently isn't? Can we set the derated to false somehow?

[creuther]

@marc-hanheide I'm waiting for their response on it. The firmware bug is only a "display" issue, the charger isn't necessarily going into derating mode but when it is on the charging, the derating flag is being shown as set regardless of the internal state.

Apparently it is possible to see the actual power draw of the charger when on the charging station, but this data is not yet exposed in the charger status channel / topic (it is certainly the first I've heard of it). It is nevertheless present in one of the CAN bus PDOs and should be viewable with moderate effort. I am testing it on Bob now, I'll report back when I'm done.

[bfalacerda]

can someone point me to the scripts to do the nice plots, so we leave betty charging tonight?

[marc-hanheide]

I don't have a script. I used

1. rosrun topic_tools throle messages /battery_state 1 /battery_state_throttled to limit the data recorded to once per second
2. rostopic echo -p /battery_state_throttled > batt.csv used to record this into a CSV file directly
3. downloaded this file to my laptop
4. grep -v "%" batt.csv | tr "," "\t" > batt.dat to convert it into something gnuplot understands

5. run gnuplot with the following command:

   plot "bat.dat" using (($3-1460397600000000000)/3600000000000-4):5 axes x1y1 title "Current [A]", "bat.dat" using (($3-1460397600000000000)/3600000000000-4):6 axes x1y2 title "lifetime" with lines

   where 1460397600000000000 is the UNIX epoch multiplied by 10e9 to get the starting time of the plot... and 3600000000000 is how long an hour is in nano seconds ;-)

Feel free to cast into a script!

[creuther]

So I tested the PDO stuff on Bob and it seems to work, even though it not that user-friendly. You have to get exclusive access to the CAN bus (i.e. every other MIRA instance accessing the CAN bus has to be killed) and then opening miracenter e.g. like this: miracenter SCITOSConfigs:etc/SCITOS-mapping.xml --variables robot=SCITOS-A5,canType=MLCAN

Then open the "CAN Tools View" by going "Window -> Show view -> CAN Tools View". You should see something like this:

PDO 0x183 and 0x184 are the interesting ones. Bytes 3 and 2 will indicate the voltage in mV and bytes 5 and 4 will indicate the current in mA. 0x183 is the battery voltage and current, and 0x184 is the charging station (ACS) voltage and current. So for example for PDO 0x184 we get voltage = 89ef hex = 35.3 dec (least signifcant byte first or whatever, anyway the second one needs to go first).

To get the overall current provided by the charger, you do I_charger = (V_acs * I_acs / V_battery) * 0.95 And of course it holds that I_charger = I_draw + I_battery so we can see the power draw of the robot and its components as well.

A note on my part: It would be easier for these diagnostics to adapt the server_port parameter here to enable interprocess communication between MIRA frameworks. This way, we wouldn't have to kill anything to access MIRA stuff, we could just connect to the running instance by doing miracenter -k ip:port.

[bfalacerda]

So we got betty to use ~8A by loading all her processors, and then put her to charge with everything closed. Looks good I think:

[bfalacerda]

Just to clarify,battery starts dropping once @ferdianjovan started doing stuff with her

[marc-hanheide]

Wow, this looks nice... So, @creuther what is the difference between this newer robot (betty is the new one!) and the older versions? @bfalacerda can you do the same with bob, just to see if there is a difference?

[bfalacerda]

Bob has a broken charger thingy, so he only charges at 5A tops (we're getting a new one). I left him charging with a cable (we only have 1 charging station in Bham), and with skirt off, and this is what it looks like. It's less stable, but it also looks good I think. I'll try the same with the skirt on, but atm if we use more than 5A he'll be losing battery even when plugged in. That makes it harder to heat up the system... How should I do it?

[creuther]

@marc-hanheide Especially with a new charger, there should be no significant difference between the robots. As mentioned before, can you run another one of these plots with voltage instead of lifetime? Especially after changing the charger but also in general, the lifetime is (unfortunately) not the most reliable metric. In your last plot it could in theory be that the robot reached full charge at around 8am, though I know that it is unlikely. We would be able to see that by plotting the voltage, which would level out when the battery is fully charged (and then jittering as charging is turned on and off periodically to keep the robot at full charge but not overcharge it).

Also, to exclude any issues we should really do a test where the robot is charging from a non-full state with all computers and external peripherals turned off. The status LED will still show the voltage and current, and there we could see how well the robot charges without any kind of load.

[gestom]

Hi, since Betty had some problems with her charger at TSC, we left Linda's charger there and took Betty's to Lincoln. With Linda, the charging is working quite OK, but we still had the overheating problem. When charging, Linda's charging current initially started about 6A, but it dropped quickly to 3A, which makes charging rather slow. We went for a hardware solution with a 92 mm fan, see .

We tried three full charging cycles: without the vent, vent only and vent+fan. The results are attached as well.

So for us, the HW solution to prevent the overheat seems to work quite OK.

[marc-hanheide]

<3

[cdondrup]

Why is this assigned to me? Maybe you meant @creuther ?!

[marc-hanheide]

I assume so, yes.

[hawesie]

@gestom @marc-hanheide do you have instructions somewhere on how to do this or a description of what you did? I'd like to get Betty equipped with a fan before she goes back to TSC.

[gestom]

we (our technician) cut a circular vent in the bottom skirt (see the pic) and installed a 92mm 12V PC fan, which we connected to one of the side PCs

[marc-hanheide]

@kirumang this is what needs to be done for henry!

[gestom]

@kirumang I checked the logs from the last two days and Werner is charging only by 2 Amps. There are two differences to our robot: 1) the fan is 80mm instead of 92 2) the firmware was not probably updated. I will get a silent 92mm fan and update the firmware now. It seems that I will need to come here next week as well.

[marc-hanheide]

Boooh... Not again or still...