lgsvl / simulator

A ROS/ROS2 Multi-robot Simulator for Autonomous Vehicles
Other
2.26k stars 777 forks source link

System Requirements to run AutowareAI Stack along with VSE- SVL simulator #1352

Closed marcusvinicius178 closed 3 years ago

marcusvinicius178 commented 3 years ago

Hi @EricBoiseLGSVL

This doubt is quite extense, and is a branch of our last discussion:

https://github.com/lgsvl/simulator/issues/1341#issuecomment-815247302.

Then decided to create another issue, as is related to other topic and I believe that can also help other students in the same condition as my.

Ok here go:

I am currently testing AutowareAI with SVL simulator. However I am facing some issues, as described here:

According to LG staff my notebook is not strong enough to run both SVL and Autoware AD stack simultaneously without having some failures in Modules.

According to the documentation: https://www.svlsimulator.com/docs/installation-guide/system-requirements/

I need to run both the simulator and the AD stack successfully the following requirements:

Requriments

From my Aspire VX 15 notebook I have the following configuration:

Cores:

Core_number+speed

Number of cores: 4 or 8 ?( I do not understand) Speed of 2.80 GHz right? What means capacity = 4005 MHz = 4 Ghz. I am confuse now the speed is 2.8 GHz or 4 GHz?

Memory:

Memory

Memory is around 4 Gb correct?

General System + Nvidia Graphic Card

System_configuration

Nvidia Card: Geforce GTX 1050 Ti

In a summarize comparing to the requirements of SVL documentation

1) A) I need 4 GHz Quad core CPU = 16 cores ?

N_of_cores

I believe I have 4 cores right or is it dual? B) Need 4 GHz CPU Reality: 2.8 GHz or 4 GHz ?

2) Need at least: GTX1080 with 8GB memory Reality: GTX 1050 Ti with 4 GB memory

3) GPU with at least 10 Gb of memory Reality: Gpu with 4 Gb of memory

As far as I understood and from experience trying the software the modules will fail. Then as solution, this following snippet of documentation suggest me to use a distributed system:

dDistributed_system

Actually it seems this strategy is even better than having a super powerful machine, as they say. However I have nerver used 2 notebooks or different PCs integrated, and I do not know how to set this: 1) Run SVL simulator on my machine (ok) 2) Run The AutowareAI stack on other machine and force to connect with this first machine running the simulator ( How to set this)

I am not an expert on computers, then after the next step I would like some advice from you, regarding what would be more feasible (in terms of efficiency and money). In my case is just for a research, I just need the modules working regular, not the best performance. Then What would you suggest:

1) Chosse the distributed systems option: Buy, rent or lend another notebook from a friend or store and try to run the AD Autoware Stack on ths this machine and simulator on my machine or the contrary? I ask this because I will probably get a less powerful machine than mine (because of price). And what would be the minimum specification for this machine to run the simulator or the AD stack?

2) Choose running run everything on the same machine: Buy a new Nvidia GTX 1080 (8 GB memory) and install on my actual notebook (retrieving the current Nvidia GTX 1050 Ti). Actually I do not know if this is possible because of size of this?

3) Rent a computer from an online computer company with the correct specifications? Does this service exist and is available, would you know to say? If yes I should opt to rent just 1 desktop with the full required system recommendation, or decide to rent just one notebook to run the SVL simulator and run the AutowareAI Ad stack on my machine?

I will be very grateful for any suggestion. Please let me know if exist other possibilities.

Thanks in advance.

EricBoiseLGSVL commented 3 years ago

@marcusvinicius178 This is a lengthy post. If this is your first time using git hub issues, you should have one clear concise issue with the application that you are trying to solve. This allows others that are searching to find the exact issue and quickly find the answer. This has many questions all in a very large post that is hard to parse. Since you are new to simulator and working with hardware I will try to help but it doesn't help others when we need to answer this much on one issue. Ok, strap in, here we go...

Example of how this issue should be: 1) Can simulator and autoware run on a laptop with these specs (include specs)? Example answer: I would point you to the documentation where we state the minimum specs (you don't need to post screen shots of the docs, just link the page if you need to explain) to run simulator and you can look up the specs needed to run Autoware. Then, you can make an informed decision on whether your machine is up to the task. Now, these are recommended specs, the simulator and autoware may run, be we can't trouble shoot crashes when it is so below spec to even run simulator. Use at your own risk.

Now on to the hardware questions. Nice details on exact hardware specs, awesome and thank you.

2) Your specs for your CPU is linked here If you are unsure about any of the terms or values, this is a good time to do some searching to find out. As for min spec for simulator, Quad means 4 and you have 4 cores, all good. We recommend 4ghz and you are at 2.8. This is not great, improving this would be good for better performance.

3) Ok GPU, this is the big one. We recommend a 1080 GTX 8gb at minimum to run simulator itself. The reason this is so high is that Unity's HD render pipeline needs lots of memory to render realistic lighting, physics, large texture maps, etc. It's similar to the concept of the latest games require really powerful GPU's to show off the best of the technology available. This is something we can't really change at the moment so running out of memory will crash the simulator. You can create very simple maps that won't use that much texture memory and it will work but again, once you are below the threshold it is difficult to support.

4) Autoware or Apollo AD stack software and Simulator together on the same machine. The documentation that you posted here clearly states we recommend a GPU memory of 10gb when running both on one machine. 8gb can work but again, you can have failures and crashes and it is difficult to support. We understand that this is really high and a barrier to test so we recommend one machine running linux that is dedicated to AD stack software and another windows machine running simulator. These are connected with a bridge on the same network with a gigabit switch or router. Here is a ROSbridge tutorial that may help explain the bridge and how it works. Then you can understand how simulator ROS bridge works for connecting to the machine running the Autoware.

5) Hardware advice. OK this is tough because of money/space/hardware available etc. For what you have described I would borrow or get another laptop to run Autoware so you can practice using the bridge. This should let you run simulator and Autoware together and see results. Then, if you are dedicated to continue, start shopping around for a desktop that exceeds minimum spec as best you can. It is always best to develop on a desktop. Once you have the best hardware you can get in the desktop, use the laptop to run Autoware and use the desktop for simulator development and testing.

6) Distributed. Not what you need. This is for sensor configurations that struggle to run on one high powered desktop, e.g. An ego sensor config that needs 4 128 beam lidars and multiple HD cameras. So then you can run the single simulation on multiple machines and the distributed feature will split the sensors to other machines to allow real-time performance.

I recommend you read through more documentation on Autoware.AI, ROS, Simulator and Unity and try the setup I detailed. You can do this, it just take practice and a ton of reading. Good luck.

marcusvinicius178 commented 3 years ago

@marcusvinicius178 This is a lengthy post. If this is your first time using git hub issues, you should have one clear concise issue with the application that you are trying to solve. This allows others that are searching to find the exact issue and quickly find the answer. This has many questions all in a very large post that is hard to parse. Since you are new to simulator and working with hardware I will try to help but it doesn't help others when we need to answer this much on one issue. Ok, strap in, here we go...

@EricBoiseLGSVL actually I have used github last year first time to a ROS1 project with self-driving cars in Gazebo. But just to commit codes, add or remove lines. For issues communication you are right, it is the first time. I apologize to put long texts, however I do this, because I was used to ROS Answers forums, where the users often view the doubt but rarely answer your question quickly or even reply you. Then I prefer to put all the doubts in a way that I will avoid the need of reply a lot of times because lack of information, once I considered that it would take too much time for someone answer me here. This is the reason to extense texts and redundant questions in similar issues. However I noticed that the support here is much better and quicker :) ! Then I will be more concise in next doubts. Thanks for the hints, I am learning...

Example of how this issue should be:

  1. Can simulator and autoware run on a laptop with these specs (include specs)? Example answer: I would point you to the documentation where we state the minimum specs (you don't need to post screen shots of the docs, just link the page if you need to explain) to run simulator and you can look up the specs needed to run Autoware. Then, you can make an informed decision on whether your machine is up to the task. Now, these are recommended specs, the simulator and autoware may run, be we can't trouble shoot crashes when it is so below spec to even run simulator. Use at your own risk.

Now on to the hardware questions. Nice details on exact hardware specs, awesome and thank you.

  1. Your specs for your CPU is linked here If you are unsure about any of the terms or values, this is a good time to do some searching to find out. As for min spec for simulator, Quad means 4 and you have 4 cores, all good. We recommend 4ghz and you are at 2.8. This is not great, improving this would be good for better performance.
  2. Ok GPU, this is the big one. We recommend a 1080 GTX 8gb at minimum to run simulator itself. The reason this is so high is that Unity's HD render pipeline needs lots of memory to render realistic lighting, physics, large texture maps, etc. It's similar to the concept of the latest games require really powerful GPU's to show off the best of the technology available. This is something we can't really change at the moment so running out of memory will crash the simulator. You can create very simple maps that won't use that much texture memory and it will work but again, once you are below the threshold it is difficult to support.
  3. Autoware or Apollo AD stack software and Simulator together on the same machine. The documentation that you posted here clearly states we recommend a GPU memory of 10gb when running both on one machine. 8gb can work but again, you can have failures and crashes and it is difficult to support. We understand that this is really high and a barrier to test so we recommend one machine running linux that is dedicated to AD stack software and another windows machine running simulator. These are connected with a bridge on the same network with a gigabit switch or router. Here is a ROSbridge tutorial that may help explain the bridge and how it works. Then you can understand how simulator ROS bridge works for connecting to the machine running the Autoware.

Ok I got the idea. But according to documentation the PythonAPI and VSE does not support windows yet. Then for my specific case that I need to set a custom scenario (with Python or VSE on windows) and integrate with AutowareAI stack ( in linux) this best performance would not open right? Just for random traffic work on windows.

In addition I will be very grateful, if you could identify why the simulator does not keep opened when I use VSE to create the custom scenario and closes in few seconds, as I displayed on youtube: VSE + Simulator (open and closes after few seconds) https://www.youtube.com/watch?v=crb7QoS8N3E

Would be a limitation of my laptop to render the scenario and keep the agents moving, a CPU/GPU overload? Or is it a current issue of the simulator? I ask because using PythonAPI the simulation maintained opened as much as I needed adding the following python code line at the end: sim.run(0) PythonAPI + Simulator (always opened) https://www.youtube.com/watch?v=KgB-t0tMxXk

Is there a "trick" to keep the simulation based on VSE "infinitely" opened?

  1. Hardware advice. OK this is tough because of money/space/hardware available etc. For what you have described I would borrow or get another laptop to run Autoware so you can practice using the bridge. This should let you run simulator and Autoware together and see results. Then, if you are dedicated to continue, start shopping around for a desktop that exceeds minimum spec as best you can. It is always best to develop on a desktop. Once you have the best hardware you can get in the desktop, use the laptop to run Autoware and use the desktop for simulator development and testing.

Regarding topic 4 above of your answer I have found a solution, but also another doubt:

A relative will borrow his desktop to use the SVL simulator, and I will run the Autoware AD stack in my laptop, as you suggested. However because of covid and distance I will use his desktop virtually with TeamViewer to run the simulator in his house and my laptop in my house. Is it possible to establish this laptop-desktop connection just with wirelless internet and ip connection or is it necessary to have a physical connection between his desktop and my laptop through network cables to run the simulator-AD stack? This issue is best described here to shrink space here and get more viewers: https://answers.ros.org/question/375845/connect-svl-simulator-desktop-to-autowareai-running-in-other-machine/

  1. Distributed. Not what you need. This is for sensor configurations that struggle to run on one high powered desktop, e.g. An ego sensor config that needs 4 128 beam lidars and multiple HD cameras. So then you can run the single simulation on multiple machines and the distributed feature will split the sensors to other machines to allow real-time performance.

Oh this is an amazing feature you provide. Congrats!

I recommend you read through more documentation on Autoware.AI, ROS, Simulator and Unity and try the setup I detailed. You can do this, it just take practice and a ton of reading. Good luck.

Yes you are right! Actually I read the documentation of SVL, did all the video lectures of AutowareAuto and AI. As well as read the PythonAPI and tested the quickstart scripts. However the documentation is never 100% functional. For example to make Python API scripts work with AutowareAI I needed to modify them and create a new folder with the modified scripts. For AutowareAuto they worked perfectly, but the behaviour of LGSVL simulator changed from SVL regarding these Python scripts. This is why I put questions. It is always when I get stuck in documentation. Not because I did not try to read them.

Anyway I got the idea of being more straight to the point. It will bring advantage for everyone :)

In a summarize to close this issue I would like first an assistance of these 2 doubts:

1) Fix VSE with AutowareAI integration on simulation (open the simulation and keep it opened) 2) Possibility to distant communication of SVL simulator (on desktop) to the AutowareAI stack (on laptop) without physical (cable) connection?

Thanks for the answers and the way you put the answers in clever topics. It helps a lot!!!

EricBoiseLGSVL commented 3 years ago

@marcusvinicius178 No worries, glad that is sorted. The information is great, we just want to limit issues to one topic to make it easier for others to parse.

Yes, VSE is not for windows yet. We hope to have this soon. Sorry for the confusion, I thought your screenshots were Linux. The issue you are seeing with the unexpected stop is a bug and we hope to have a fix soon.

Running over a network that is not local is not recommended. The lag would be great. It can be done though. No wireless. Dropped packets are not good for any AD stack but again it can be done :)

@KarByc has worked very hard on this feature, thanks

Overall we try to make sure all features of the simulator are working as intended but yes. some docs are not updated or bugs occur. That is why we leverage the community to help find these issues. It makes simulator better for everyone and helps us determine what features are next. I hope this helps answer the issue and let us know how the wireless works out. Sometimes you just have to develop with what you have at hand. Good luck

KarByc commented 3 years ago

Hello @marcusvinicius178 Thanks for using our latest version of the Simulator :)

Network: It is possible to run a distributed simulation using Wi-Fi in a local network, but a cable connection will give better results. Data delay while using Wi-Fi may lead to a simulation desynchronization and unexpected AutowareAI behavior. As Eric mentioned, using distributed simulation will help you if used sensors require too much performance from a single machine.

VSE: We are currently working on fixing the VSE runner, thanks for the report and the included video. Please track the following issue to get updated about this bug: https://github.com/lgsvl/simulator/issues/1340

marcusvinicius178 commented 3 years ago

Hi @EricBoiseLGSVL , @KarByc thanks for the reply.

No Eric I am not using windows. I am running on Linux, sorry for the misunderstanding. I would use windows in my relative's desktop(for the better performance reason you suggested, if worked with PythonApi) and Linux on my laptop. However we installed a new ssd in his desktop then I installed Linux there to run the simulator.

Regarding the wireless connection unfortunately failed. I have published a video on YouTube of it below. I tried a lot of different IP configuration to make it work. However on bridge info I get a kind of "Htts forbidden, error sending request, and url exception lnvalid string" errors. As you can check:

https://youtu.be/qkRPkCguY5c

I tried the following ones inside Bridge IP tab:

Any attempt worked successfuly to set up the bridge connection, unfortunately 😕. Should I try a different IP, add or subtract some info I have put on this Autopilot Bridge connection?

Just to make clear, I am using the simulator on a desktop in my relative's house, and the notebook with AutowareAI AD stack at my home. Is there a configuration setup missing that I must do, inside or outside the simulator, to make the connection work with wirelles? If yes please let me know how to setup properly and enable to run these frameworks with parallell computational power 🤩!

I would like to buy a desktop now to run everything locally with cable network connection, however the price here in Brazil has super increased, because people are mining Bitcoin. Currently It is pretty hard to find good GPU boards available on market 😅.

Thanks very much in advance

EricBoiseLGSVL commented 3 years ago

The bridge is the ip/port of the machine running the AD stack. You can check what addresses/ports are being used. Make sure no other applications are using them or a antivirus is blocking. We don't support wireless connection, so we are not sure how to debug your setup. Yes, miners and Covid shortages are making PC purchases difficult :(

lemketron commented 3 years ago

If I understand correctly, the simulator is running on a remote linux machine, and should be configured with your public IP address because it will need to connect with the autopilot bridge running on your linux laptop. However, you'll need to configure port-forwarded on your internet router to route from the 9090 port on the public (WAN) side to port 9090 on the private IP of your laptop.

In addition, if you're running python API scripts on your linux laptop, you'll need to use the public IP of the simulator host and the remote router will need to configure port-forwarding for port 8181 (python API port on simulator) from the public IP of the simulator network to the private IP of the simulator machine.

That's a lot of steps, and ultimately the latency may cause other issues (though if you use the clock sensor then the autopilot will run on simulator time and latency should be less of an issue). Still, while this might be interesting from a "can it be done" standpoint, it can also be quite a challenge to configure and make work... good luck!

marcusvinicius178 commented 3 years ago

Yes @lemketron you understood pretty correctly!!! But OMG 😳!!! it really seems quite challenging, even more for me, a mechanical engineer. I have not any TI deep experience, and therefore any idea how to setup a complex communication such as this you have written above.

What I understood from your message as a layman is: there is not trivial, functional natural private Ip connection Simulator (remote desktop) to AutowareAI (Linux notebook) as I tried previously ( in forwarded YouTube video), it is more complex than I imagined... Let's go to reality( my understanding): "I should at first enable the Public IP communication and afterwards enable the access to the Private IP through the Public IP for both machines (but different ports because of different functionalities use different ports). Later I need to fix the lag/delay in simulation using a clock sensor..." Is basically that?

What's this clock sensor? Is it provided by the simulator, ROS or must I setup also this externally as the IP connections?

Would you have in Hands please, any tutorial that could help me in this regard? If not from LG simulator or ROS, another one of anything that use a similar concept. Because I confess I did not catched the complete idea. Maybe with a YouTube video I can have a better visualization of similar steps to be done. If possible could you recommend a video tutorial to set up these "mixed Ips" configuration and network enabling, please? Perhaps a site, documentation is useful too, etc?

And finally if LG support wish to close the issue. You can do it, as we discussed here a lot... and found a possible solution for that.

Thanks very much again for all the support you have towarded to this issue!

EricBoiseLGSVL commented 3 years ago

Thanks @lemketron for the help. These links might help although I think some are too simple but it might help others. IP/Port UnityROSExample ROSC# Hope this helps. It seems very complex but once you have it setup, it doesn't seem so bad. You got this.

Yes, the clock sensor is in webUI but I don't this it supports Autoware.AI. @lemketron ?

No problem and happy coding :)

lemketron commented 3 years ago

Yes, the clock sensor is in webUI but I don't this it supports Autoware.AI. @lemketron ?

I believe the clock sensor works for Autoware.AI (@hadiTab?) as discussed in https://github.com/lgsvl/simulator/issues/653#issuecomment-595275100

Note that the clock sensor documentation for 2021.1 is here: https://www.svlsimulator.com//docs/simulation-content/sensors-list/#clock

marcusvinicius178 commented 3 years ago

Hi @lemketron I tried the connection we discussed about (desktop-notebook). The connection worked after lot of attempts to configure the gateways...

connection_worked

However unfortunately as you said, the syncrhonizations is too bad. And the simulator-Autoware.AI delay about 30 seconds to share the msgs. The localization module for example, localize the car in rviz after lot of time be activated.

Then I launched the mission and motion planning launch files, expecting the car would move in SVL simulator. But it didn't and the simulator closes, with socket disconnection error:

error

The last hope was trying to use the clock sensor. For that inside Autoware_AI docker image I issued the command:

rosparam set /use_sim_time true

It seems that the clock sensor works good with Autoware_AUto and LGSVL: https://github.com/lgsvl/simulator/issues/653#issuecomment-595275100

However if used for Autoware_AI and SVL simulator, the output is an error in AD stack side, specifically within rviz that fails to get LG environment data and does not generate the map:

AD_Stack_fails_with_clock

Then it is not possible to continue launching other AD stack modules.

My next step is set up a phisical cable network connection between the Desktop and my notebook. Do you believe Am I going to face problems with Gateway connections and delay?

Thanks. And here is my feedback video about this attempt: It is a bit long...need patience to see it. The success in connection happened in the end...so you can skip the beginning (setting up the gateways, etc).

https://youtu.be/ktZVZd8_0aM

EricBoiseLGSVL commented 3 years ago

@marcusvinicius178 You will have less delay and connection issues when connected via cable. Be sure to use a gigabit switch/router.

marcusvinicius178 commented 3 years ago

@marcusvinicius178 This is a lengthy post. If this is your first time using git hub issues, you should have one clear concise issue with the application that you are trying to solve. This allows others that are searching to find the exact issue and quickly find the answer. This has many questions all in a very large post that is hard to parse. Since you are new to simulator and working with hardware I will try to help but it doesn't help others when we need to answer this much on one issue. Ok, strap in, here we go...

Example of how this issue should be:

  1. Can simulator and autoware run on a laptop with these specs (include specs)? Example answer: I would point you to the documentation where we state the minimum specs (you don't need to post screen shots of the docs, just link the page if you need to explain) to run simulator and you can look up the specs needed to run Autoware. Then, you can make an informed decision on whether your machine is up to the task. Now, these are recommended specs, the simulator and autoware may run, be we can't trouble shoot crashes when it is so below spec to even run simulator. Use at your own risk.

Now on to the hardware questions. Nice details on exact hardware specs, awesome and thank you.

  1. Your specs for your CPU is linked here If you are unsure about any of the terms or values, this is a good time to do some searching to find out. As for min spec for simulator, Quad means 4 and you have 4 cores, all good. We recommend 4ghz and you are at 2.8. This is not great, improving this would be good for better performance.
  2. Ok GPU, this is the big one. We recommend a 1080 GTX 8gb at minimum to run simulator itself. The reason this is so high is that Unity's HD render pipeline needs lots of memory to render realistic lighting, physics, large texture maps, etc. It's similar to the concept of the latest games require really powerful GPU's to show off the best of the technology available. This is something we can't really change at the moment so running out of memory will crash the simulator. You can create very simple maps that won't use that much texture memory and it will work but again, once you are below the threshold it is difficult to support.
  3. Autoware or Apollo AD stack software and Simulator together on the same machine. The documentation that you posted here clearly states we recommend a GPU memory of 10gb when running both on one machine. 8gb can work but again, you can have failures and crashes and it is difficult to support. We understand that this is really high and a barrier to test so we recommend one machine running linux that is dedicated to AD stack software and another windows machine running simulator. These are connected with a bridge on the same network with a gigabit switch or router. Here is a ROSbridge tutorial that may help explain the bridge and how it works. Then you can understand how simulator ROS bridge works for connecting to the machine running the Autoware.

Hi again @EricBoiseLGSVL running the AD stack and the simulator in 2 different machines lead me to a too much delay... For this reason I wish to rent a powerful machine per day/week to run my simulations. I am still unsure about the minimum requirements (you have specified correctly and clear I know, however there a lot of combinations that does not fit exactly all the requirements, sometimes matches a little bit with less or more capacity, as you can check on these websites) https://www.leadergpu.com/?utm_source=google&utm_medium=cpc&utm_campaign=leadergpu|706109400&utm_content=tar|kwd-915384745830|aid|472799154889|gid|110218481206|dvc|c|reg|9074157|rin|9042583&utm_term=%2Bleadergpu&adposition=&matchtype=b&device=c&network=g&gclid=Cj0KCQjwytOEBhD5ARIsANnRjVj1S7HhIGgQsfHBgwMRlkjulc0ODj1Zx-3AZxUIW65FOKbqWt2SBncaAv13EALw_wcB

https://www.vps-mart.com/gpu-server?vps=onsale&gclid=Cj0KCQjwytOEBhD5ARIsANnRjVgs_kTuJh8KWeNjD03ygjOnZ-3gltvlV25KLndPbl7MfNWockR1AoIaAkvGEALw_wcB

There are some Nvidia-Cards different from GTX (Tesla for example), also some machines that overcomes the number of GPU memory required (> 10 Gb), however presents lower required CPU frequency (< 4GHz)... and so on...Could you please check the video below and suggest from it, or give a general list overview of a machine that I could rent that would satisfy the requirements to setup a simulation with Autoware.AI or Apollo with few Agents and perform just an evasive maneuver on BorregasAve?

https://youtu.be/6aoexvOjjLk I am unsure to fullfill the TABS:

image image image

Which cheapest configuration would work?

Woud this one be enough? Or other one from this site? image or some of these? image

Or configuration from another site you could suggest? Thanks vey much in advance.

  1. Hardware advice. OK this is tough because of money/space/hardware available etc. For what you have described I would borrow or get another laptop to run Autoware so you can practice using the bridge. This should let you run simulator and Autoware together and see results. Then, if you are dedicated to continue, start shopping around for a desktop that exceeds minimum spec as best you can. It is always best to develop on a desktop. Once you have the best hardware you can get in the desktop, use the laptop to run Autoware and use the desktop for simulator development and testing.
  2. Distributed. Not what you need. This is for sensor configurations that struggle to run on one high powered desktop, e.g. An ego sensor config that needs 4 128 beam lidars and multiple HD cameras. So then you can run the single simulation on multiple machines and the distributed feature will split the sensors to other machines to allow real-time performance.

I recommend you read through more documentation on Autoware.AI, ROS, Simulator and Unity and try the setup I detailed. You can do this, it just take practice and a ton of reading. Good luck.

EricBoiseLGSVL commented 3 years ago

Damn, with those prices, I would just build or buy one. ad stack dictates the gpu. If you look at apollo docs, certain versions support certain architectures. Not sure with autoware but apollo 5.0 only supports up to 1080. I would use apollo 6.0 GTX 2080ti 16 ram high frequency cpu ubuntu version is tricky. we have some users that must use 18.04 but others that use 20.04. You'll need to look at apollo forums on that. Borregas is a bit high texture memory, but with one EGO and a few npcs/peds you will be fine. Hope this helps

marcusvinicius178 commented 3 years ago

@EricBoiseLGSVL Thanks very much now it is much more clear to me. Yeah yeah the prices are sharp!

I tested on my relative's home my laptop with his Desktop on the same Router with computers physically side by side (very close) using wireless, and despite bridge connection works, the high delay made the SVL closes after some time, due failures.... But I still have one last shot! we will try this weekend a network cable connection to router. I hope both computers connected to router, with respective wires, be faster and better than wireless. If not I will need to rent one of these PCs, unfortunately. Have you already tested this, does it work.

I will take advantage of your attention and kind support, as usual 😀, and ask you if you know how to trigger the Motion Planners (Obstacle Avoiders) from Apollo 6.0. I realized the planning module just performs standard maneuvers, such as Lane Changes, stop, overtake. But not the ones I have available in Autoware (A*, RRT, Lattice, etc). I could not deviate from the static Car as you can check quickly here:

https://youtu.be/sQI9B8kVkW8

I don't know how deep you used Apollo but if you have seen or used some Obstacle Avoidance planner, please let me know how to, or where can I find these tutorials.

Thanks very much, and have a good weekend.

EricBoiseLGSVL commented 3 years ago

Sorry, not familiar with obstacle avoidance. Maybe @hadiTab or @lemketron? Good luck and let us know if you have issues.

marcusvinicius178 commented 3 years ago

Thanks, and about the wired network connection (Laptop-Desktop) have you already tried? Bridge Works? Better than wireless?

Sorry I have checked that @KarByc have already answered

marcusvinicius178 commented 3 years ago

Hi @EricBoiseLGSVL just to let you know the network wired connection (Laptop-Desktop) works! As you can check below

https://youtu.be/9keDL75MS8M

The wireless connection does not work. And the Simulator closes in few minutes with bridge disconnection errors... Howeve the Ego Vehicle does not avoid the obstacle. I still wonder how to properly use the motion planners and how to switch them. I am going to open a new issue, here, considering anyone answered me in Apollo issues: https://github.com/ApolloAuto/apollo/issues/13763

Maybe some user here can have already worked with this module.

marcusvinicius178 commented 3 years ago

@EricBoiseLGSVL just one last question on this regard for optimization purpose.

I had some issues in speed performance, when running Apollo and SVL on same machine. I have rent the P6000 machine on Paperspace. But There are 2 machines more powerful as possibility to rent: The P6000x2 and the RTX5000x2. Which one do you believe would guarantee me a more constant speed on simulator ( I mean the simulation om P6000 is kind of running slow, with some jerk/freezing). Do you believe one of these ones I described above could address a better result on this regard, which one SHOULD I RENT?

The specification of these machines are below:

Screenshot_2021-06-07-14-31-08-098_com google android gm

Screenshot_2021-06-07-14-28-00-939_com android chrome

I have already tested the V100 vs P6000. And V100 gives a slower simulation performance. To test one by one machine pdemand too much time, to build Apollo module, and money (10$ each rent for storage). That is why I wish to give the best shot now...

I did not find a complete description of RTX5000x2 just found an article that says the clock is frequency is better than P6000x2. I also do not know the meaning of x2 in front of P6000 and RTX They double which feature?

I ask because I do not know which parameter is related to the speed performance on simulation. Is clock, memory, GPU? Could you provide an advice (P6000x2 OR RTX5000x2)?

Thanks

EricBoiseLGSVL commented 3 years ago

I think P5000 is fine as long as apollo supports.

marcusvinicius178 commented 3 years ago

Really? P5000 has better performance than:

1-P6000 2- RTX5000x2 3- P6000x2

I thought the options (2 or 3 ) would be the better ones to enhance speed performance, not?

lemketron commented 3 years ago

I had some issues in speed performance, when running Apollo and SVL on same machine. I have rent the P6000 machine on Paperspace. But There are 2 machines more powerful as possibility to rent: The P6000x2 and the RTX5000x2. Which one do you believe would guarantee me a more constant speed on simulator ( I mean the simulation om P6000 is kind of running slow, with some jerk/freezing). Do you believe one of these ones I described above could address a better result on this regard, which one SHOULD I RENT?

Sounds like you've switched from Autoware to Apollo... Probably should leave the autoware thread closed and start a new one for Apollo thoughts so we don't mix Apollo issues into an Autoware post.

That said, you can probably also search for Apollo GPU or check these docs for latest Apollo as Apollo 6.0 (and Master) should support GTX or RTX architecture including Pascal, Turing, Volta, and Ampere as discussed in https://github.com/ApolloAuto/apollo/issues/12294.

marcusvinicius178 commented 3 years ago

You are right, my fault sorry, I mixed softwares...forgot about this detail 😅.

But my doubt is not linked to the accepted GPUs. But which one could run the simulation faster without face a freezing, delay...if RTX500x2 or P6000x2 ?

I do not understand if one or another would address better speed to the simulations. (I am using a complex scenario, SAN FRANCISCO city). Using Modular Perception or directly Perception module tab does not offer significantly changes...

lemketron commented 3 years ago

But which one could run the simulation faster without face a freezing, delay...if RTX500x2 or P6000x2 ?

Check out these links for information (I just searched google for "nvidia RTX500 vs P6000"): https://gagadget.com/en/graphics-cards/quadro-p6000-vs-quadro-rtx-5000 https://gpu.userbenchmark.com/Compare/Nvidia-Quadro-RTX-5000-vs-Nvidia-Quadro-P6000/m712800vsm187394

The Quadro P6000 is build with Pascal architecture, and Quadro RTX 5000 - with Turing. The Quadro P6000 has 8 GB more memory, than Quadro RTX 5000 Note: Quadro P6000 and Quadro RTX 5000 are only used in laptop graphics. They have lower GPU clock speed compared to the desktop variant, which results in lower power consumption, but also 10-30% lower gaming performance.

So the RTX 5000 is newer architecture, lower power, and appears to be at least slightly faster, though either one should probably run SVL Simulator just fine. You know these are both laptop GPUs, right?

marcusvinicius178 commented 3 years ago

Hi Steve no I didn't. I am beginner in computer science... I thought they were from Desktop. I just rent a machine on paperspace.com and the SVL runned on P6000 and on V100 but slowly....than I decided to try better ones (P6000x2 and RTX5000x2). However considering Apollo takes too time to build I decided to ask first (also to save money because each time I rent a new machine I am 15$ charged + 2$ hour) . Which website could I get Desktop GPUs for reasonable $ value? And which one do you recommend? I am mechanical engineer getting touch now with programming, docker, git, computer architectures.

Now I am really confused which machine to rent on paperspace.com I even know if their machines are Desktops or Laptops 😐.

Thanks very much for the amazing detailed answer! According to the link you provided RTX has greater FPS so the simulation will go faster. I hope. I will close the issue but feel free to suggest me a website and machine. I have already take a look on AWS cloud machines and other companies. But still have no idea of which one to rent.