search

bluerobotics / BlueOS

The open source platform for ROV, USV, robotic system operation, development, and expansion.
https://blueos.cloud/docs/
Other
146 stars 72 forks source link

bug: High CPU Temperature observed on Heavy-config BlueROV2 with BlueOS installed on Navigator Flight Controller #2025

Open rajmis opened 1 year ago

rajmis commented 1 year ago

Bug description

Background: We recently performed a hardware upgrade on our Heavy BlueROV2, replacing the existing hardware with BR Navigator Flight Navigator, accompanied by an installation of BlueOS 1.0.0. Following the hardware upgrade, we updated BlueOS to version 1.1.0-beta.17 and have been utilizing this configuration since then.

Problem: In the course of regular usage, we have started observing an unexpected and concerning behavior related to the CPU temperature of the BlueROV2. Even when the vehicle is at idle (~20% CPU usage) and placed on a bench, the reported CPU temperature gradually rises to approximately 70°C and remains stable around this value. I even upgraded to 1.1.0-beta.28 but issue persists.

  1. Is the temperature rise expected?
  2. Is there any temperature limit by BlueOS that will result in throttling RPi4 and the communication with Navigator?

Steps to reproduce

  1. Take the Electronics Enclosure with Navigator Flight Controller, BlueOS can be either 1.1.0-beta.17 or 1.1.0-beta.28.
  2. Turn on using bench top power supply or BR battery pack.
  3. Wait for about 20 minutes, and CPU temperature will reach around 60 to 70 °C. This will be a gradual increase and not a sudden change.

Primary pain point(s)

No response

Prerequisites

patrickelectric commented 1 year ago

Hi @rajmis, thanks for raising the issue.

We have a major improvement on CPU usage on the new releases of BlueOS. I would recommend to connect your vehicle on the internet, perform an update to beta.28 or install a fresh image. If you do the update to beta.28 (not installing the image on the sdcard directly), go again on the BlueOS update page and perform a bootstrap update operation, the button should appear.

patrickelectric commented 1 year ago

Keep in mind that in a BlueROV2 you have an entire system running, with camera doing h264 video encoding, the enclosure will be closed and heat dissipation will be a major issue. The ROV should be on the water, with that the temperature will decrease. Avoid exposing the ROV on directly sunlight in field, protect it with a wet towel over the enclosure.

joaoantoniocardoso commented 1 year ago

Hi @rajmis, to complement @patrickelectric's answer while targetting your specific questions:

  1. Is the temperature rise expected?

Yes, both BlueROV configurations, when with the acrylic enclosure closed and out of the water, are expected to be around 70°C.

  1. Is there any temperature limit by BlueOS that will result in throttling RPi4 and the communication with Navigator?

AFAIK the Raspberry Pi 4 is rated to work up to 85°C, so it should thermal throttle when approaching that temperature, but from our experience, 70°C is fine. Also, we have been performing tests in hot waters (possibly above 25°C) and we haven't noticed any degradation in performance, although we don't have the data to check the CPU temperature during those tests.

Let us know if you experience any problems or have any further questions, thanks!

rajmis commented 1 year ago

Hello @patrickelectric and @joaoantoniocardoso, thank you for your comments and suggestions. I followed the advice and tried installing beta.28 using both a fresh SD card image and the BlueOS Firmware Update + Update Bootstrap method. Unfortunately, the issue with the CPU temperature persists despite these attempts.

Additionally, I have observed some frame drops when multicasting the video streams. After conducting further investigation, it appears that this issue is not related to the temperature. Please do let me know if this expected.

As a temporary solution, we have placed the Electronics Enclosure in an Aluminum casing, which has proven to be effective in dissipating heat.

Just a quick check, I'm using Ardusub Firmware 4.1.0 and want to understand if giving a target depth is still working. Are there any changes to type_mask or coordinate_frame? Didn't want to raise another issue until I'm sure of it.

joaoantoniocardoso commented 1 year ago

Hi @rajmis,

specifically about the video part, from my experience, dropping frames is likely to be a symptom of the network. If multiple clients need to receive the video at the same time, it's okay to configure the stream with multiple UDP, but it is important to keep in mind that each UDP might require peaks of 20Mbps bandwidth for a 30fps 1080p h264 stream, which will be challenging for more than two clients. If the idea is to have the flexibility to access the stream from different clients, not necessarily at the same time, or if you are using WebRTC (like in Cockpit), RTSP should have a better performance.

Just to share some data from my last test regarding the temperature topic: the CPU temperature indicated 70°C when the ROV was out of the water (clear sky, but the radiation was only 400W/m²), and 65°C when the ROV was just below the surface of the water. The air temperature was about 21°C, and the water was about 19°C.

What is the temperature situation with the aluminum case?

Just a quick check, I'm using Ardusub Firmware 4.1.0 and want to understand if giving a target depth is still working. Are there any changes to type_mask or coordinate_frame? Didn't want to raise another issue until I'm sure of it.

Maybe @patrickelectric or @Williangalvani could help you with this.

Thanks