flowdriveai / flowpilot

flow-pilot is an openpilot based driver assistance system that runs on linux, windows and android powered machines.
GNU General Public License v3.0
1.64k stars 236 forks source link

How to run on PC / Desktop #38

Open SrinivasuluCharupally opened 1 year ago

SrinivasuluCharupally commented 1 year ago

Hi, I am trying to setup it on PC / Desktop, Intel CPU + Ubuntu 20.04. I followed the install steps from here : https://github.com/flowdriveai/flowpilot/wiki/Installation#installing-on-desktop Then I followed "Running on Simulator" steps from here : https://github.com/flowdriveai/flowpilot/wiki/Running-On-Simulation#following-steps-will-help-you-get-carla-simulation---flowpilot-integration-up-and-running

I got confused with these following two steps... Could you help to clarigy

This means, total 3 terminals do i need to launch, one for "./CarlaUE4.sh -RenderOffScreen", one for "./launch_flowpilot.sh" and one for "python tool/sim/bridge.py"

Is that correct ? and, Do i need to have Camera or SmartPhone to run the application ?

MankaranSingh commented 1 year ago

Yes, you need to run them parallely in 3 seperate terminals.

SrinivasuluCharupally commented 1 year ago

Thanks Mankaran. One more thing.. This is what i am getting when try to run . I am running from SSH.

`(flowpilot) che7@che7-w30871:~/flowpilot$ ./CarlaUE4.sh -RenderOffScreen 4.26.2-0+++UE4+Release-4.26 522 0 Disabling core dumps. WARNING: lavapipe is not a conformant vulkan implementation, testing use only.

che7@che7-w30871:~/flowpilot$ ./launchflowpilot.sh logmessaged thermald statsd keyvald flowpilot pandad deleter flowinitd logmessaged thermald statsd keyvald flowpilot pandad deleter flowinitd logmessaged thermald statsd keyvald flowpilot pandad deleter flowinitd logmessaged thermald_ statsd keyvald flowpilot pandad deleter flowinitd accepted_terms False completed_training False ignition False

che7@che7-w30871:~/flowpilot$ python3 tools/sim/bridge.py Restarting bridge. Error: time-out of 5000ms while waiting for the simulator, make sure the simulator is ready and connected to 127.0.0.1:2000 Restarting bridge. Error: time-out of 5000ms while waiting for the simulator, make sure the simulator is ready and connected to 127.0.0.1:2000 Restarting bridge. Error: time-out of 5000ms while waiting for the simulator, make sure the simulator is ready and connected to 127.0.0.1:2000

`

RaghavPrabhakar66 commented 1 year ago
(flowpilot) che7@che7-w30871:~/flowpilot$ ./CarlaUE4.sh -RenderOffScreen
4.26.2-0+++UE4+Release-4.26 522 0
Disabling core dumps.
WARNING: lavapipe is not a conformant vulkan implementation, testing use only.

Is Carla running properly?

SrinivasuluCharupally commented 1 year ago

Not sure, doing it for the first time. i running it from SSH.

MankaranSingh commented 1 year ago

You will have to make sure carla is running. If you remove the RenderOffScreen arg, it should show the carla window which will confirm its up and running

SrinivasuluCharupally commented 1 year ago

Could you please confirm this error. this is OK or, Not ?

(flowpilot) che7@che7-w30871:~/flowpilot$ python scripts/dummy_params.py

:228: RuntimeWarning: compiletime version 3.8 of module 'common.params_pyx' does not match runtime version 3.9 :228: RuntimeWarning: compiletime version 3.8 of module 'common.transformations.transformations' does not match runtime version 3.9 done
MankaranSingh commented 1 year ago

I think you compiled (ran scons command) without entering the flowpilot env (pip env shell) Do pip env shell then scons in flowpilot root directory then this warning should go away

SrinivasuluCharupally commented 1 year ago

Got it, thanks Mankaran.

SrinivasuluCharupally commented 1 year ago

MankaranSingh - We tried the steps but that did not work for us. Attaching a screenRecording of what we did. Could tell us what was the issue ?

jFYI ... Prithiveen from our team tried the steps.

https://github.com/flowdriveai/flowpilot/assets/7656634/83bd1ae9-1baf-4457-a442-d073f2c09113

https://github.com/PrithiveenKumaarRamkumar

RaghavPrabhakar66 commented 1 year ago

@SrinivasuluCharupally Can you share the output of following command.

file tmp.avi
SrinivasuluCharupally commented 1 year ago

@RaghavPrabhakar66
che7@che7-w30871:~/flowpilot/selfdrive/assets$ file tmp tmp: RIFF (little-endian) data, AVI, 1164 x 874, 25.00 fps, video: Motion JPEG

PrithiveenKumaarRamkumar commented 1 year ago

@MankaranSingh @RaghavPrabhakar66 Able to successfully launch flowpilot app now :) bridge.py still renders a Segmentation fault (core dumped) error

https://github.com/flowdriveai/flowpilot/assets/99075098/f5949cfb-b580-4fb4-badc-8821c5c299f1

MankaranSingh commented 1 year ago

I'd recommed you clean the build using scons -c. I am not sure but maybe when you interchanged the python version (3.10) and ran scons that built some files that were not compatible.

SrinivasuluCharupally commented 1 year ago

@MankaranSingh , @RaghavPrabhakar66 - Could we have short call some time next week to look into the issues ? We are from Valeo team / org , trying to explore the option for different platform.

PrithiveenKumaarRamkumar commented 1 year ago

@MankaranSingh @RaghavPrabhakar66 Tired cleaning the build using scons -c and then building again using scons but bridge.py still renders a Segmentation fault (core dumped) error user journey: https://drive.google.com/file/d/1h7DjBOh8l7pfuaAKOrPIbo6Kdaa1f4Sl/view?usp=sharing

MankaranSingh commented 1 year ago

@PrithiveenKumaarRamkumar @SrinivasuluCharupally I am not able to reproduce the issue. It would be helpful if you exactly mention what lines are causing segmenation fault in the script.

MankaranSingh commented 1 year ago

@SrinivasuluCharupally the issue is related to the carla-py package provided in this repo. you can try uninstalling that and do pip install carla==0.9.12 and rerun this.

SrinivasuluCharupally commented 1 year ago

@MankaranSingh - the segfault occurs when try to import the carla from bridge.py script

Steps:

#pipenv shell
# scons
# python tools/sim/bridge.py
Segmentation fault

===> this seg fault occur from the line import carla in the tools/sim/bridge.py

SrinivasuluCharupally commented 1 year ago

BTW - We tried the above steps with pip install carla==0.9.12, pip3 install carla==0.9.12 and pip install -e carla-py. Result is same all the time, seg fault while importing carla into the bridge.py. I am sure some thing is missing in the README.

HiHeXi commented 1 year ago

I also encountered the same problem. Have you solved the problem? If it is resolved, I would like to ask for advice on a solution

@MankaranSingh @RaghavPrabhakar66 Able to successfully launch flowpilot app now :) bridge.py still renders a Segmentation fault (core dumped) error flowpilot_app_journey.mp4

I also encountered the same problem. Have you solved the problem? If it is resolved, I would like to ask for advice on a solution

MankaranSingh commented 1 year ago

You can try installing carla like this https://github.com/carla-simulator/carla/issues/1466#issuecomment-522797557

SrinivasuluCharupally commented 1 year ago

@MankaranSingh - could you confirm these steps are correct for each terminal. We are running every thing from the pipenv shell, check the below steps

Terminal#1

pipenv shell scons ./CarlaUE4.sh -RenderOffScreen

Terminal#2

pipenv shell launch_flowpilot.sh

Terminal#3

pipenv shell python tool/sim/bridge.py

MankaranSingh commented 1 year ago

Your steps are correct. you just need a working carla-python api. Please check this out https://github.com/carla-simulator/carla/issues/1466#issuecomment-522797557

HiHeXi commented 1 year ago

你的步骤是正确的。 你只需要一个工作carla-蟒蛇。 请看看这个 卡拉-模拟器/卡拉#1466(注释)

This method won't work

PrithiveenKumaarRamkumar commented 12 months ago

@MankaranSingh In https://github.com/carla-simulator/carla/issues/1466#issuecomment-522797557 it is recommended to install carla by extracting the .egg files, but the .egg files are for python3.7.

However, activating the flowpilot environment pipenv shell takes you to python3.9 environment

The Carla build documentation https://carla.readthedocs.io/en/latest/build_linux/ and discussions https://github.com/carla-simulator/carla/discussions/4519 says that the PythonAPI is supported till python3.8. I am now able to import carla successfully in python3.7 and python3.8

Using python3.8 I was able to generate the build using scons and run the other scripts successfully but bridge.py renders the following traceback:

Traceback (most recent call last):
  File "tools/sim/bridge.py", line 28, in <module>
    pm = messaging.PubMaster(['roadCameraState', 'roadCameraBuffer', 'wideRoadCameraState',
  File "/home/che7/flowpilot/cereal/messaging/__init__.py", line 257, in __init__
    self.sock[s] = pub_sock(s)
  File "/home/che7/flowpilot/cereal/messaging/__init__.py", line 47, in pub_sock
    sock.connect(context, endpoint)
  File "cereal/messaging/messaging_pyx.pyx", line 139, in cereal.messaging.messaging_pyx.PubSocket.connect
    raise MultiplePublishersError
cereal.messaging.messaging_pyx.MultiplePublishersError
MankaranSingh commented 12 months ago

@PrithiveenKumaarRamkumar for MultiplePublishersError, you'd need to comment out the ROAD_CAMERA_SOURCE and wide camera source. This will not start the video publishers and allow publihers from carla bridge script to take over

PrithiveenKumaarRamkumar commented 11 months ago

@MankaranSingh Now we are able to run Carla, launch the flowpilot app and run bridge.py from the python3.8 environment but we are not able to "take control of the wheel" in the flowpilot app. Are we missing something here? https://drive.google.com/file/d/1gLtdT1o6qeKijSyk_YwiPW62gROMnFJn/view?usp=sharing

MankaranSingh commented 11 months ago

https://github.com/flowdriveai/flowpilot/wiki/Running-On-Simulation

see the last step. The input keys would be taken in the bridge.py terminal

SrinivasuluCharupally commented 11 months ago

@MankaranSingh - We are able to control the Carla output with 1, 2 and W-A-D keys. Could you tell us how to render with our own video with the environment, we have a local traffic video, would like to see the flowdriver output for that. Let us know the details.

SrinivasuluCharupally commented 11 months ago

@MankaranSingh - could you guide us on this ?

MankaranSingh commented 11 months ago

The video can be specified in ROAD_CAMERA_SOURCE in launch script. By default, flowpilot desktop version run on a video only.

SrinivasuluCharupally commented 11 months ago

@MankaranSingh - do we have to launch all the three terminals with the "ROAD_CAMERA_SOURCE" enabled ? The reason is, we noticed following error with ROAD_CAMERA_SOURCE enabled , you suggested to disable it /comment out last time.

_Traceback (most recent call last): File "tools/sim/bridge.py", line 28, in pm = messaging.PubMaster(['roadCameraState', 'roadCameraBuffer', 'wideRoadCameraState', File "/home/che7/flowpilot/cereal/messaging/init.py", line 257, in init self.sock[s] = pub_sock(s) File "/home/che7/flowpilot/cereal/messaging/init.py", line 47, in pub_sock sock.connect(context, endpoint) File "cereal/messaging/messaging_pyx.pyx", line 139, in cereal.messaging.messaging_pyx.PubSocket.connect raise MultiplePublishersError cereal.messaging.messagingpyx.MultiplePublishersError

MankaranSingh commented 11 months ago

ROAD_CAMERA_SOURCE needs to be disabled to accept stream from carla simulation. do not run carla or or bridge.py if you want to run on video

On Sun, Oct 1, 2023 at 2:06 PM Srinivasulu Charupally < @.***> wrote:

@MankaranSingh https://github.com/MankaranSingh - do we have to launch all the three terminals with the "ROAD_CAMERA_SOURCE" enabled ? The reason is, we noticed following error with ROAD_CAMERA_SOURCE enabled , you suggested to disable it /comment out.

Traceback (most recent call last): File "tools/sim/bridge.py", line 28, in pm = messaging.PubMaster(['roadCameraState', 'roadCameraBuffer', 'wideRoadCameraState',

— Reply to this email directly, view it on GitHub https://github.com/flowdriveai/flowpilot/issues/38#issuecomment-1742006470, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKHLNLGCXAN7HSIWLOSQWCLX5ETRHANCNFSM6AAAAAA2YQM3PE . You are receiving this because you were mentioned.Message ID: @.***>

SrinivasuluCharupally commented 10 months ago

Hi @MankaranSingh - Sorry for the late repose, we were busy with other projects. Still we are not able to render the video.

script:

Attaching our "launch_flowpilot.sh" script.

steps:

pipenv shell scons ./launch_flowpilot.sh

output

attaching screen shot

Screenshot from 2023-10-30 11-39-41

launch_flowpilot.py.txt

PrithiveenKumaarRamkumar commented 10 months ago

Hi @MankaranSingh Do we have to make changes on these scripts for using user's input video? https://github.com/flowdriveai/flowpilot/blob/master/selfdrive/loggerd/video_process.py https://github.com/flowdriveai/flowpilot/blob/master/selfdrive/loggerd/config.py

MankaranSingh commented 10 months ago

The default launchfile when you download it freshly is taking in video input only. you do not need to run bridge.py. after launch just run scripts/dummy_publishers.py in seperate terminal

SrinivasuluCharupally commented 10 months ago

Thanks @MankaranSingh I am able to run it now. Attaching screen shots one more question : Would it display the bounding boxes on detected objects ?

Screenshot from 2023-11-01 17-49-14 Screenshot from 2023-11-01 17-50-10 Screenshot from 2023-11-01 17-50-15 Screenshot from 2023-11-01 17-50-53 ?

MankaranSingh commented 10 months ago

This is an end to end driving model. this dosent outputs bounding boxes

SrinivasuluCharupally commented 10 months ago

@MankaranSingh - I have downloaded a video from web, mp4 format, and tried to render the video under flowDrive, but seeing following error. You have any idea ?

Screenshot from 2023-11-02 09-19-08

Video download link : https://www.pexels.com/video/cars-traveling-on-expressway-5382495/

MankaranSingh commented 10 months ago

convert video to avi format.

SrinivasuluCharupally commented 10 months ago

@MankaranSingh - I tried but did not work. Does it really requires Calibration & CAN signals ? it shows "CalibrationError" & "canError".

piyushmz commented 4 months ago

Hey @MankaranSingh I followed the steps in here https://github.com/flowdriveai/flowpilot/wiki/Running-On-Simulation for running on desktop. I get flowpilot in infinite loop. ubuntu 20.04 nvidia-driver-535(proprietary tested) carla= 0.9.12 I've given ROAD_CAMERA_SOURCE="test.avi" and Iaunched flowpilot following is the o/p I get on after running.

[main] INFO org.nd4j.nativeblas.NativeOpsHolder - Number of threads used for linear algebra: 2
[main] INFO org.nd4j.linalg.cpu.nativecpu.CpuNDArrayFactory - Binary level Generic x86 optimization level AVX/AVX2
[main] INFO org.nd4j.nativeblas.Nd4jBlas - Number of threads used for OpenMP BLAS: 2
[main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Backend used: [CPU]; OS: [Linux]
[main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Cores: [24]; Memory: [7.8GB];
[main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Blas vendor: [OPENBLAS]
[main] INFO org.nd4j.linalg.cpu.nativecpu.CpuBackend - Backend build information:
 GCC: "7.5.0"
STD version: 201103L
DEFAULT_ENGINE: samediff::ENGINE_CPU
HAVE_FLATBUFFERS
HAVE_OPENBLAS
[ERROR:0] global /home/runner/work/opencv/opencv/opencv-4.5.1/modules/videoio/src/cap.cpp (142) open VIDEOIO(CV_IMAGES): raised OpenCV exception:

OpenCV(4.5.1) /home/runner/work/opencv/opencv/opencv-4.5.1/modules/videoio/src/cap_images.cpp:253: error: (-5:Bad argument) CAP_IMAGES: can't find starting number (in the name of file): /home/rp/Downloads/testhd.avi in function 'icvExtractPattern'

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by ai.flow.common.BufferUtils (file:/home/rp/flowpilot/common/build/libs/common-0.0.1.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of ai.flow.common.BufferUtils
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[main] INFO ai.flow.launcher.FlowInitd - Connected to the FlowInit daemon
[ERROR:0] global /home/runner/work/opencv/opencv/opencv-4.5.1/modules/videoio/src/cap.cpp (142) open VIDEOIO(CV_IMAGES): raised OpenCV exception:

OpenCV(4.5.1) /home/runner/work/opencv/opencv/opencv-4.5.1/modules/videoio/src/cap_images.cpp:253: error: (-5:Bad argument) CAP_IMAGES: can't find starting number (in the name of file): /home/rp/flowpilot/selfdrive/assets/videos/intro.avi in function 'icvExtractPattern'

Exception in thread "camerad:roadCameraState" CvException [org.opencv.core.CvException: cv::Exception: OpenCV(4.5.1) /home/runner/work/opencv/opencv/opencv-4.5.1/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'
]
        at org.opencv.imgproc.Imgproc.cvtColor_1(Native Method)
        at org.opencv.imgproc.Imgproc.cvtColor(Imgproc.java:6842)
        at ai.flow.sensor.camera.CameraManager.processFrame(CameraManager.java:112)
        at ai.flow.sensor.camera.CameraManager.run(CameraManager.java:165)
        at java.base/java.lang.Thread.run(Thread.java:829)

Is there any hardcoded path like /home/runner/* somewhere in flowpilot

If I comment out ROAD_CAMERA_SOURCE and start carla, flowpilot is always into infinite loop and nothing is detected.

MankaranSingh commented 4 months ago

@piyushmz what is test.avi ? where are you placing it ?

piyushmz commented 4 months ago

so I was testing with a downloaded mp4 file which I converted to avi and I placed the path in ROAD_CAMERA_SOURCE but it didn't work. The selfdrive/assets/tmp does work with flowpilot. As of bridging of carla with flowpilot I get segmentation fault.