search

autorope / donkeycar

Open source hardware and software platform to build a small scale self driving car.
http://www.donkeycar.com
MIT License
3.13k stars 1.29k forks source link

Protect LineFollower from None cam_img #961

Closed Ezward closed 2 years ago

Ezward commented 2 years ago

User anbello is getting an error trying to start the cv_control template.:

Traceback (most recent call last):
  File "/home/pi/projects/donkeycar/donkeycar/vehicle.py", line 151, in start
    self.update_parts()
  File "/home/pi/projects/donkeycar/donkeycar/vehicle.py", line 199, in update_parts
    outputs = p.run(*inputs)
  File "./manage.py", line 80, in run
    max_yellow, confidense, mask = self.get_i_color(cam_img)
  File "./manage.py", line 59, in get_i_color
    scan_line = cam_img[iSlice : iSlice + self.scan_height, :, :]
TypeError: 'NoneType' object is not subscriptable

My theory is that when the template first starts, the camera is not 'warmed' so it doesn't immediately return an image, which causes the LineFollower to fail. We should 1) warm the camera (wait until it starts providing frames) 2) protect LineFollower from a None image by just returning.

Discovered bug: cv_control.py will not run on a headless OS install because it tries to open a window with debug info using opencv. The fix is to make this debug window optional.

Tasks

Testing

Note: these tests require installation of 'imgaug' on the donkeycar, which is an optional install pip install imgaug

Ezward commented 2 years ago

It looks like the picam already handles it's own warming; perhaps it should wait until frames get returned. https://github.com/autorope/donkeycar/blob/3297771078ad7d9dd52f6497f55f3f831cd9f31b/donkeycar/parts/camera.py#L35

Ezward commented 2 years ago

Calling the run() method on the camera part does NOT set self.frame. It should probably set self.frame and return that if it does not get a frame from the camera. That's not the issue here, but probably still a good idea. Then we can call the run method until a non-empty frame is returned and then we know positively that the camera is warmed OR if we do this for 2 seconds and do not get a frame then we know the camera is not going to work. Then when enter the vehicle loop the camera part will already have one frame ready.

Ezward commented 2 years ago

https://github.com/autorope/donkeycar/tree/961-protect-from-empty-camera-image

Ezward commented 2 years ago

I get a related bug using the complete.py template from dev branch and a model that uses 'CROP' transformation. In that case it appears that imgaug get's a None image and becomes unhappy

INFO:donkeycar.vehicle:Starting vehicle at 20 Hz
/usr/lib/python3/dist-packages/picamera/encoders.py:544: PiCameraResolutionRounded: frame size rounded up from 160x120 to 160x128
  width, height, fwidth, fheight)))
Traceback (most recent call last):
  File "/home/pi/projects/autorope/donkeycar/donkeycar/vehicle.py", line 154, in start
    self.update_parts()
  File "/home/pi/projects/autorope/donkeycar/donkeycar/vehicle.py", line 202, in update_parts
    outputs = p.run(*inputs)
  File "/home/pi/projects/autorope/donkeycar/donkeycar/pipeline/augmentations.py", line 120, in run
    aug_img_arr = self.augmentations.augment_image(img_arr)
  File "/home/pi/env/lib/python3.7/site-packages/imgaug/augmenters/meta.py", line 766, in augment_image
    type(image).__name__),)
TypeError: %d format: a number is required, not str
INFO:donkeycar.vehicle:Shutting down vehicle and its parts...

If I remove 'CROP' transformation then the model runs without the error (of course it does not perform well) So that shows it is the imgaug that is erroring. If I run the fix branch https://github.com/autorope/donkeycar/tree/961-protect-from-empty-camera-image then the error goes away. That indicates the imgaug was crapping out on a None camera image. So fixing the camera parts to make sure they return non-none images fixes more than one kind of bug.

Ezward commented 2 years ago

I was able to repeat the cv_control bug using the current dev branch (commit a26bb7300e0c6ee348876a08491ee172c169203e) . I then switch the donkeycar project folder to the fix branch, https://github.com/autorope/donkeycar/tree/961-protect-from-empty-camera-image and the error went array. So this confirms that the issue was with a None image and the fix branch fixes that. I need to test with CISC camera and Web cam as well.

Ezward commented 2 years ago

Got the imgaug error on a Jetson Nano with CSIC camera config trying to run a model that uses 'CROP' configuration using the complete.py template and the dev branch; relevant configuration in myconfig.py:

CAMERA_TYPE = "CSIC"   # (PICAM|WEBCAM|CVCAM|CSIC|V4L|D435|MOCK|IMAGE_LIST)
IMAGE_W = 160
IMAGE_H = 120
IMAGE_DEPTH = 3         # default RGB=3, make 1 for mono

TRANSFORMATIONS = ['CROP']
ROI_CROP_TOP = 45               # the number of rows of pixels to ignore on the top of the image

python manage.py drive --type=tflite_linear --model=models/linear_crop_multiply.tflite results in 

INFO:donkeycar.vehicle:Starting vehicle at 20 Hz
Traceback (most recent call last):
  File "/home/ed/projects/donkeycar/donkeycar/vehicle.py", line 154, in start
    self.update_parts()
  File "/home/ed/projects/donkeycar/donkeycar/vehicle.py", line 202, in update_parts
    outputs = p.run(*inputs)
  File "/home/ed/projects/donkeycar/donkeycar/pipeline/augmentations.py", line 120, in run
    aug_img_arr = self.augmentations.augment_image(img_arr)
  File "/home/ed/env/lib/python3.6/site-packages/imgaug/augmenters/meta.py", line 766, in augment_image
    type(image).__name__),)
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ed/projects/donkeycar/donkeycar/parts/camera.py", line 174, in update
    self.init_camera()
  File "/home/ed/projects/donkeycar/donkeycar/parts/camera.py", line 169, in init_camera
    self.poll_camera()
  File "/home/ed/projects/donkeycar/donkeycar/parts/camera.py", line 181, in poll_camera
    self.frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
cv2.error: OpenCV(4.5.4) /tmp/pip-req-build-bbq4c33o/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

TypeError: %d format: a number is required, not str

INFO:donkeycar.vehicle:Shutting down vehicle and its parts...
stopping CSICamera
Ezward commented 2 years ago

Pull Request 967 addresses this issue and a little more.

Ezward commented 2 years ago

We are unable to test 'CROP' with CSIC camera on the Jetson Nano because of a bug in imgaug on the Jetson Nano; see Issue 970. We can verify that the CSIC camera passes the warm up phase and returns frames. So the 'CROP' code would work if imgaug worked and given we have confirmed 'CROP' works on RaspberryPi with PICAM and we have not actually changed any 'CROP' code, then we can presume that it would work on the Nano if imgaug is fixed.

Ezward commented 2 years ago

Merged https://github.com/autorope/donkeycar/pull/967