uci-uav-forge / uavf_2024

Flight software for Student Unmanned Aerial Systems 2024 competition
MIT License
3 stars 0 forks source link

Fix camera latency #168

Closed EricPedley closed 5 months ago

EricPedley commented 6 months ago

Summary

Switched from using Opencv to PyAV. This also let us get rid of all the dumbass threading code to keep the latest frame being read. I also removed an unused test and moved the test that we were using to a separate script instead of having it inside the siyi stream class.

Actual file changes are here: https://github.com/uci-uav-forge/siyi_sdk/commit/594bbfd033a693c3755b84ded444360994128c50 https://github.com/uci-uav-forge/siyi_sdk/commit/a3936907fd446b0add3569055b9a5c025180096c

Test Plan

I ran siyi_sdk/dev/stream_test.py and it works with unnoticeable latency.

https://github.com/uci-uav-forge/uavf_2024/assets/48658337/b5fb6a93-a5a4-47d1-9603-428592848a6a

Issues

Closes #166

Dat-Bois commented 6 months ago

Do an outdoor test if you can and place targets some distance away, I wanna see if there is any noticeable drop in resolution / increased compression.

But the fact u found something which worked off the bat makes me so happy, I might end up using it for my other stuff too.

Also can you try disconnecting the camera while running the stream code, we should see how it reacts since during comp the battery will disconnect during battery swaps.

EricPedley commented 6 months ago

For still images the quality seems about the same. However, with AV sometimes the frame has compression artifacts during the start of the stream but surprisingly they don't seem to come back when there's a lot of motion.

Compression artifacts: image

With opencv: image With av: image

EricPedley commented 6 months ago

Also I'm not realizing that by removing the threading, get_frame actually isn't returning the latest frame unless it's being called constantly so I'll have to put the thread to consume the buffer back in.

EricPedley commented 6 months ago

Changing the delay to 1/1000 seconds on the frame-grabbing thread seemed to fix the decoding artifacts. TBH I don't know it doesn't work without that, but I figured it would work because when I was just doing a simple loop, the cv.waitKey(1) was that little of a delay and it worked fine.

EricPedley commented 6 months ago

When the camera is physically disconnected from power, it stops working but doesn't crash the program, but reconnecting it doesn't fix it. I'll try to figure out a good way to handle this.

EricPedley commented 6 months ago

When the cam powers off, it hangs at next(self._stream), so I'm thinking we need a way to set a timeout on that and call disconnect / raise an exception if it times out, then force the user to call connect again.

EricPedley commented 6 months ago

Actually this error handling won't be that important or necessary because we can call disconnect() before landing and call connect() after takeoff, we just need to include that detail in the higher level Camera class instead of abstracting it away. But maybe we should have an exception raised when the camera is powered off while connected so we can log if that happens instead of freezing the program.