The web app often doesn't show a camera feed when returning to Bite Selection, if your phone falls asleep during Bite Selection, or if you put your phone on sleep (e.g., press the power button) during Bite Selection.
Further, sometimes even if it does render an image, it is a very old image (e.g., there is lots of latency).
Bite Selection overlays the mask on top of the live video feed for every mask (3 currently), hence maintaining 4 independent live video streams at once.
As documented in that post, latency/lag is a well-known issue with web_video_server because it guarantees delivery of every packet. The only way to avoid it is to move away from web_video_server.
My hunch about why it doesn't show the image is that either React or the browser caches some elements of the page layout and therefore do not attempt to reload the web_video_server URL. Or maybe it thinks the stream is still open, but web_video_server has closed the stream, Either way, changing the URL the web app is rendering should address this.
Although in theory web_video_server supports video compression (e.g., h264 / vp8), in practice I hadn't been able to get them to work. The types I was able to get to work, mjpeg and ros_compressed essentially stream individual images anyway, nixing the video compression benefit of web_video_server.
Hence, I decided to move away from web_video_server and towards sending and rendering individual CompressedImage messages in the app. This should not require any more bandwidth than our existing approach, and should address the latency and blank image.
However, based on the linked post and Vinitha's experience with the Stretch robot, WebRTC will likely have lower latency and lower bandwidth. This is particularly important, because when running the app with network and CPU throttling, the image doesn't render (nor does it get feedback). Hence, we should look into whether WebRTC will address that, and consider switching to WebRTC that for our web app.
Since ada_feeding#141, SegmentFromPoint now returns the RGB image, so the web app can overlay that with the mask, instead of the whole video feed.
[x] Recreate old errors: Without the code in this branch (i.e., using the old web app code), do the below steps and verify that you see some or all of the following errors: video feed not rendering (i.e., showing up as white), video feed rendering but frozen, video feed rendering but laggy.
[x] Do a complete nominal run-through of a bite and then return to bite selection. Do this at least twice. You should see one or more of the above issues.
[x] On the bite selection screen, let your phone go to sleep. After 5-10 secs, re-open your phone to the browser. You should see one or more of the above issues.
[x] On the bite selection screen, put your phone to sleep (e.g., pushing the power button). Wait, reopen, and look for errors.
[x] On the btie selection screen, go back to your app's home screen (e.g., move the app into the background). Wait, reopen, and look for errors.
[x] Pull the code in this branch
[x] Redo the above steps. Verify that there are no errors.
[x] Real
[x] Recreate old errors: Without the code in this branch (i.e., using the old web app code), do the below steps and verify that you see some or all of the following errors: video feed not rendering (i.e., showing up as white), video feed rendering but frozen, video feed rendering but laggy.
[x] Do a complete nominal run-through of a bite and then return to bite selection. Do this at least twice. You should see one or more of the above issues.
[x] On the bite selection screen, let your phone go to sleep. After 5-10 secs, re-open your phone to the browser. You should see one or more of the above issues.
[x] On the bite selection screen, put your phone to sleep (e.g., pushing the power button). Wait, reopen, and look for errors.
[x] On the btie selection screen, go back to your app's home screen (e.g., move the app into the background). Wait, reopen, and look for errors.
[ ] Pull the code in this branch
[ ] Redo the 4 above checks on both wifi networks. Verify that there are no errors.
[ ] Phone is on Lovelace-2.4G
[ ] Phone is on Lovelace-5G
Before creating a pull request
[x] Format React code with npm run format
[x] Format Python code by running python3 -m black . in the top-level of this repository
[x] Thoroughly test your code's functionality, including unintended uses.
[N/A] Fully test the responsiveness of the feature as documented in the Responsiveness Testing Guidelines. If you deviate from those guidelines, document above why you deviated and what you did instead.
[N/A] Consider the user flow between states that this feature introduces, consider different situations that might occur for the user, and ensure that there is no way for the user to get stuck in a loop.
Before merging a pull request
[ ] Squash all your commits into one (or Squash and Merge)
Description
Issues
Design Considerations
web_video_serverbecause it guarantees delivery of every packet. The only way to avoid it is to move away fromweb_video_server.web_video_serverURL. Or maybe it thinks the stream is still open, butweb_video_serverhas closed the stream, Either way, changing the URL the web app is rendering should address this.web_video_serversupports video compression (e.g., h264 / vp8), in practice I hadn't been able to get them to work. The types I was able to get to work,mjpegandros_compressedessentially stream individual images anyway, nixing the video compression benefit ofweb_video_server.web_video_serverand towards sending and rendering individualCompressedImagemessages in the app. This should not require any more bandwidth than our existing approach, and should address the latency and blank image.ada_feeding#141, SegmentFromPoint now returns the RGB image, so the web app can overlay that with the mask, instead of the whole video feed.Solution
See 1.iv above.
Testing
Setup
Sim via EC2:
ros2 launch feeding_web_app_ros2_test feeding_web_app_dummy_nodes_launch.xmlros2 run ada_feeding_perception republisher --ros-args --params-file src/ada_feeding/ada_feeding_perception/config/republisher.yamlcd src/feeding_web_interface/feedingwebapp/; npm startReal:
lovelace. (Note:pymoveit2has a new main branch,amaln/scale_collision_meshes)Tests
Lovelace-2.4GLovelace-5GBefore creating a pull request
npm run formatpython3 -m black .in the top-level of this repositoryBefore merging a pull request
Squash and Merge)