Open apolak opened 4 months ago
Blyron
commented
4 months ago Hello! I have not been able to repro it. Which commands have you introduced in the Python client? have you enabled the recorder o maybe enabled de save images function for camera sensors? Is this prebuilt version or it is build from souce?
apolak
commented
4 months ago @Blyron This is the release version with additional maps installed using ImportAssets.sh. Both packages were downloaded using links from the release page. I'm using the original Python scripts with no modifications.
To reproduce the issue, it's enough to start the Python client while CarlaUE4 is minimized.
I now recall that I've never seen this issue before installing the additional maps package. Maybe this is related?
One thing to note is that to reproduce the issue, I don't need to load any of the additional maps or interact with the Python client. It's enough to simply start CarlaUE4 and the Python client with no command line options, then minimize the CarlaUE4 window.
I can reproduce this issue without installing the additional maps package too.
I'm using the pre-installed Python 3.8 version with a virtual environment created like below:
sudo apt-get install python3-venv
cd ./PythonAPI/examples
python3 -m venv ~/carla-venv
source ~/carla-venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
pip install carla==0.9.15My machine is an AWS-hosted g4dn.xlarge instance created from the AWS-provided Ubuntu Server 20 LTS x86 image to which I'm connecting using NICE DCV.
Blyron
commented
4 months ago Which Vulkan Drivers is using your machine?
apolak
commented
4 months ago @Blyron It's the NVIDIA GRID driver v550.54.17, see the output of vulkaninfo below. I tried NVIDIA Tesla T4 and NVIDIA L4 GPUs and was able to reproduce this issue on both.
Blyron
commented
4 months ago We have repro it. It seems is UE issue https://forums.unrealengine.com/t/when-minimized-i-have-a-memory-leak-in-packaged-game-when-only-using-vulcan-or-dx12-any-ideas/1429649 We are looking at it
Blyron
commented
4 months ago Which is the FPS of your client while server minimized?
apolak
commented
4 months ago Which is the FPS of your client while server minimized?
@Blyron On NVIDIA Tesla T4 I get 25-30 FPS, same when the window isn't minimized. On NVIDIA L4 I get 40-50 FPS in both cases. Same when I run the client in sync mode.
Blyron
commented
4 months ago Have you tried with -RenderOffScreen?
apolak
commented
4 months ago Have you tried with -RenderOffScreen?
@Blyron If I start CarlaUE4 if off-screen mode, after starting the Python client I can also see memory usage constantly increasing, but at a slower rate, see the screenshot below. I started the Python client around the 8:55 mark. At the 9:35 mark, the percentage of memory used has increased from 28% to 32%. This trend isn't visible if I start CarlaUE4 in on-screen mode (not minimized).
Blyron
commented
4 months ago Okay. We got it, we are able to repro it. But we do not think it would be quick fix. Seems to be related with Unreal itself. We need to dig into that
CARLA version: 0.9.15
Platform/OS: Ubuntu 20.04
Problem you have experienced: After minimizing the CarlaUE4 window, memory usage starts to increase indefinitely. It returns to normal as soon as the window is restored. See the screenshot below, where the previously minimized window has been restored near the 10 second mark.
What you expected to happen: Memory usage is normal when the CarlaUE4 window is minimized.
Steps to reproduce:
Start CarlaUE4 in on-screen mode:
Start the
manual_control.pyscript in another terminal:Minimize the CarlaUE4 window.
Monitor the memory usage in another terminal:
Other information:
no_rendering_mode.pyscript—possibly a camera sensor issue?