Capstone Simulator Latency

[wkhattak]

Facing severe latency/lag when using the simulator for the capstone project. This happens specifically when the camera is turned on. Also, when the generated waypoints in the waypoint-updater node are set to the default 200 (camera off), the same latency appears (seems kind of OK if the waypoints are reduced to 50). I have a Windows 7 machine with i7 / 32 GB of RAM and a GPU. I first used the simulator inside the VM (with 6 CPU allocated + around 12GB RAM). Then I moved on to a native install by converting the provided VM to a physical drive via Clonezilla. However, I don't see any decrease in latency even after moving to a native install. I contacted support & was told that there are no known issues but obviously there are. Apart from getting frustrated and wasting a lot of time, this is stopping me from completing the final project in a timely fashion. A quick search over Slack & the forum reveals that a number of students are facing the same issue. I request for a speedy resolution of this issue. Thanks

[aup1907]

I am having the same issue on two different machines

[mvirgo]

Capstone project has now been moved to a workspace to help isolate any particular issues - please re-open if you see the same issues arising within the workspace.

[MyCodeBits]

@mvirgo I have been using the workspace recently. Things stay good till the time camera is NOT ON. With camera being ON, Car barely manages to stay on track for 10 seconds and then goes off. So workspace does not seem to be the solution for the problem reported. Not sure if its dependent on combined workspace load across other workspaces in the cluster where its hosted.

BROWSER: Chrome

Video at 640x480 with Graphics being Simple.

• CAMERA ON at July 24th 6:50 PM PST - Car goes off track https://drive.google.com/file/d/1NrsOC8JU-d9X2XXB3pPnrqZMm-JCDIrX/view?usp=sharing

However, even with CAMERA OFF at 6:50 PM, car goes off track.

• CAMERA OFF at July 24th 6:50 PM PST - Car goes off track https://drive.google.com/open?id=1c9bLY9H7j_jPJpIFvrhqt5MDT-tN45Xu

  I tried ~ 2 hours later and now with same code base car stays in track with CAMERA OFF.

• CAMERA OFF at July 24th 8:58 PM PST - Car stays in track https://drive.google.com/open?id=18BykAoUnGduUNMfvxvoeVApSb4rdN8PS -- Most probably load by this time on cluster would have reduced (my conjecture).

I have provided timelines if someone wants to analyze cluster health based on how many workspaces were active.

CC @awbrown90

[mvirgo]

@MyCodeBits Can you post your internet connection speed as well as what browser you are using?

[MyCodeBits]

@MyCodeBits Can you post your internet connection speed as well as what browser you are using?

Hi @mvirgo. Thanks for quick response. Earlier videos/tests were using Chrome Browser in last comment.

Internet connection speed

Chrome on Host m/c (mac OS)

Firefox inside Workspace

Created another video of what the current state is at the workspace using Firefox on host (Mac OS) accessing Workspace at July 25th 3:00 PM PST

• Car without CAMERA ON goes off track. https://drive.google.com/open?id=1mSMFCYRNuoPm-sZUnOWjaLejragU5FU9

I would be happy if you want me to try any instrumented code for any reason.

Further please note that there are other people in my team who are facing the same issue who are geographically at different locations and one person has commented in Slack #general also for this after i posted for the same issue.

My understanding is that given the workspace is remote, if one has a decent enough internet connection, the connection speed in remote workspace only should be of consideration. Please comment if i am missing/over simplifying.

CC @SIakovlev

[jonam85]

On the observation for Workspace getting struck: till the manual operation is present, the terminal runs at nominal pace. when the manual checkbox is unchecked, then the program runs at a slower pace.

For each image topic received

With automatic mode (manual checkbox unchecked, camera checked) Actual Time Delta time from previous 1532625293.0018000000 0.4056799412 1532625293.5656900000 0.5638899803 1532625293.9795300000 0.4138400555 1532625294.5536900000 0.5741598606 1532625295.1094800000 0.5557899475 1532625295.5538500000 0.4443700314 1532625295.9139400000 0.3600900173 1532625296.2937000000 0.3797600269 1532625296.8977800000 0.6040799618 1532625297.3851800000 0.4874000549 1532625297.7796800000 0.3945000172 1532625298.2115300000 0.4318499565 1532625298.6865300000 0.4750001431 1532625299.1741400000 0.4876098633 1532625299.5141200000 0.3399801254 1532625300.0766600000 0.5625398159 1532625300.5452900000 0.4686300755 1532625300.8820800000 0.3367900848 1532625301.2525100000 0.3704299927 1532625301.6422000000 0.3896899223 1532625302.0498000000 0.4075999260 1532625302.4078500000 0.3580501080 1532625302.7898200000 0.3819699287 1532625303.1111800000 0.3213601112 1532625303.4052600000 0.2940800190 1532625303.9557500000 0.5504899025 1532625304.3985300000 0.4427800179 1532625304.7305400000 0.3320100307 1532625305.1275000000 0.3969600201 1532625305.5279200000 0.4004199505 1532625305.9520800000 0.4241600037 1532625306.3499500000 0.3978700638 1532625306.7005000000 0.3505499363 1532625307.1803400000 0.4798400402 1532625307.6294700000 0.4491300583 1532625308.0477500000 0.4182798862 1532625308.4741300000 0.4263799191 1532625308.8001100000 0.3259801865 1532625309.1900000000 0.3898899555 1532625309.6295500000 0.4395499229 1532625310.0781400000 0.4485900402 1532625310.3754400000 0.2972998619 1532625310.8115100000 0.4360702038 1532625311.2482000000 0.4366898537 1532625311.6494300000 0.4012300968 1532625311.9611100000 0.3116800785 1532625312.4449300000 0.4838199615 1532625312.8244400000 0.3795099258 1532625313.2389800000 0.4145400524 1532625313.6539400000 0.4149599075 1532625314.0785500000 0.4246101379 1532625314.7061800000 0.6276299953 1532625315.1100900000 0.4039099216 1532625315.3070800000 0.1969900131

With manual mode (manual checkbox checked, camera checked) 1532625315.3842100000 0.0771300793 1532625315.4616300000 0.0774199963 1532625315.5367000000 0.0750699043 1532625315.6145000000 0.0778000355 1532625315.6913000000 0.0767998695 1532625315.7682300000 0.0769300461 1532625315.8403500000 0.0721199512 1532625315.9141200000 0.0737700462 1532625315.9960000000 0.0818800926 1532625316.0744300000 0.0784299374 1532625316.1645600000 0.0901300907 1532625316.2793400000 0.1147799492 1532625316.3689000000 0.0895600319 1532625316.4614500000 0.0925500393 1532625316.5559000000 0.0944499969 1532625316.6602500000 0.1043498516 1532625316.7534200000 0.0931701660 1532625316.8396400000 0.0862197876 1532625316.9276100000 0.0879700184 1532625317.0184100000 0.0908000469 1532625317.1038900000 0.0854799747 1532625317.2116200000 0.1077301502 1532625317.2978300000 0.0862100124 1532625317.4037900000 0.1059598923 1532625317.4947600000 0.0909700394 1532625317.5928800000 0.0981199741 1532625317.6964200000 0.1035399437 1532625317.7957400000 0.0993199348 1532625317.8946900000 0.0989501476 1532625317.9857900000 0.0910999775 1532625318.0801800000 0.0943899155 1532625318.1683300000 0.0881500244 1532625318.2614400000 0.0931100845 1532625318.3467600000 0.0853199959 1532625318.4351000000 0.0883400440 1532625318.5383200000 0.1032199860 1532625318.6375100000 0.0991899967 1532625318.7277300000 0.0902199745 1532625318.8281400000 0.1004099846 1532625318.9242300000 0.0960900784 1532625319.0219600000 0.0977299213 1532625319.1298200000 0.1078600883 1532625319.2173200000 0.0874998569 1532625319.3214600000 0.1041400433 1532625319.4047700000 0.0833098888 1532625319.5123000000 0.1075301170 1532625319.6167000000 0.1043999195 1532625319.7118300000 0.0951299667 1532625319.8026800000 0.0908501148 1532625319.8751700000 0.0724899769 1532625319.9713200000 0.0961499214 1532625320.0650900000 0.0937700272 1532625320.1593000000 0.0942101479 1532625320.2509800000 0.0916798115 1532625320.3423000000 0.0913200378 1532625320.4300000000 0.0877001286 1532625320.5175900000 0.0875899792

1532625320.6145500000 0.0969600677 1532625320.7099100000 0.0953598022 1532625320.8370000000 0.1270899773

The top rows are when Manual is unselected and Camera is selected. Bottom rows are when both Manual and Camera selected. So the camera processing is not creating any problem. When manual is selected, the delay between two topics received by our program is about 0.5 seconds whereas in the other case it is <0.1 seconds.

Left side is the clock info, right column is the delta between previous to current clock.

[mvirgo]

Couple items for further clarification and testing on our end - 1 - How is the performance of the simulator within each browser? 2 - Are you running a detection model when you unselect Manual mode? That is when the code starts calling to a detection model (or attempting to). Depending on how you've coded this call, or how large the model is (if one is included), can have a big impact here.

[swapanshridhar]

@mvirgo

Couple items for further clarification and testing on our end - 1 - How is the performance of the simulator within each browser?

I did try with Safari and Chrome with not much changes. Performance was bad on both.

2 - Are you running a detection model when you unselect Manual mode? That is when the code starts calling to a detection model (or attempting to). Depending on how you've coded this call, or how large the model is (if one is included), can have a big impact here.

When I am testing my code, there is no detection code deployed. You can assume it to be walkthrough code getting tested, where car will move forward ignoring Traffic Lights.

CC @SIakovlev

[lukavia]

I have the same problem. Since I see the issue is closed, is there a solution to the problem or not?

[pkpp1233]

I have the same problem as well. In workspace and on my laptop.

[ivovandongen]

Performance is still very bad. To the point of being unusable. Any way to get better performance out of the simulator when the camera is turned on?

[Elgeweily]

Same issue here....I am not doing any TL classification yet, and the simulator is still very slow when I turn the camera on.

[papiot]

I am also facing the problem to the point that I cannot use the workspace to work on the final project. Any ideas/fixes?

[Elgeweily]

I solved it by switching to native Linux installation, I heard Docker also works pretty well, these are your 2 options, forget about the VM or the Workspace.

[papiot]

@Elgeweily Thanks, I will do the same then!

[loewner]

I have the same problems. Is there any other solution beside installing on native Linux? (I do not wanna install ros on my client ubuntu machine)

[lukavia]

I've never found one. At the end I did some optimizations and submitted on the blind

[Elgeweily]

@loewner try Docker then....this issue is not addressed properly by Udacity

[loewner]

@Elgeweily Thanks, Docker worked for me as well.

[deybvagm]

I have the same problem with the workspace, I tried the docker option locally but the car still goes off, although I don't have a GPU. My system info is:

OS: Ubuntu 18.04 Processor: intel core i7 @ 2.00 GHz RAM: 8 GB

My question is: is a GPU required in order to be able to run the project?

[johnadams2076]

I have the same problem.

[heikoschmidt1187]

Same here, lags with VM ROS and local simulator, but somehow at least for a part of the track usable if waypoint frequency is reduced to 10Hz. Workspace really bad, Docker only usable without Camera.

I'm using Arch Linux locally, so in theory I could run native ROS, but unfortunately Opencv3 in ROS doesn't work.

My machine is a Intel Core i7 Notebook with 16GiB RAM and an nvidia gtx 950m

[haritaparekh]

Any solution for the problem?

[marcin-sielski]

Hopefully this article can help https://medium.com/@marcin.sielski/capstone-project-for-self-driving-car-nanodegree-program-3c54c3c704bf

[drugescu]

This remains an issue to this date, our whole team is stuck for now. Really disappointed that Udacity did not even care enough to answer. So what if it's 1 in a 100?

Some HW combinations seems to trigger this, others which are slightly off, do not. I guess we will submit with "hacks" and hope Carla handles it fine.