Here's a small openpilot bug clip:
https://github.com/commaai/openpilot/assets/5363/97a6c767-9b67-4206-8ba7-b4030f08a8cd
Capture and develop clips of openpilot from comma.ai's Comma Connect.
The clipper can produce clips of:
All clip options have a configurable target file size option as platforms like Discord limit file upload sizes.
The clipper is deployed on Replicate:
https://replicate.com/nelsonjchen/op-replay-clipper
Replicate is an ultra-low-cost pay-as-you-go compute platform for running software jobs. Replicate is a great way to run this clipper as it's fast, easy to use, and you don't need to install anything on your computer or even deploy anything yourself. Just enter in the required information into the form, and Replicate will generate a clip. Expect to pay about ~$0.01 per clip but not even need to put in any payment details until you've reached a generously large level of usage.
[!WARNING] comma devices should not be used as primary dashcams for numerous reasons!
They are still great as a backup dashcam and for other purposes though.
We assume you've already paired your device and have access to the device with your comma connect account.
Now I need to select the portion of the route I want to clip. Here's a video of what that UI looks like
Video:
https://github.com/commaai/openpilot/assets/5363/504665de-9222-4e6b-b090-c26cdcc7137a
uploaded
. You may need to wait and your device may need to be on for a while for all files to upload.
route
, paste the URL you copied in the previous step.
Run
.Once done, you can download the clip. If you want, turn off "Public access" on the route after you're done.
Here's a generated clip with the wide
rendering type with no UI:
https://github.com/commaai/openpilot/assets/5363/8bd91642-51ff-4de9-87d2-31e770c64542
Demonstration of speed or longitudinal behavior of openpilot with model-based longitudinal is nearly impossible or hard without this clipper. This video is of a good model based long behavior at highway speeds.
https://user-images.githubusercontent.com/5363/202886008-82cfbf02-d19a-4482-ab7a-59f96c802dd1.mp4
Cars can have bugs themselves. Here's my 2020 Corolla Hatchback phantomly braking on metal strips in stop and go traffic probably from the radar. Perhaps a future openpilot that doesn't depend on radar might be the one sanity checking the radar instead of the other way around currently. And another example of that in Portland.
https://user-images.githubusercontent.com/5363/219708673-4673f4ff-9b47-4c57-9be3-65f3ea703f3f.mp4
https://github.com/nelsonjchen/op-replay-clipper/assets/5363/1e59844b-46f8-4289-bea9-511db2718549
This is a video of a bug report where openpilot's lateral handling lost the lane.
https://user-images.githubusercontent.com/5363/205901777-53fd18f9-2ab5-400b-92f5-45daf3a34fbd.mp4
Lane cutting?
https://github.com/nelsonjchen/op-replay-clipper/assets/5363/d0ab3365-b5ef-4e05-84ee-370b88e8af02
Nav-assisted follow the road instead of taking the side road.
https://github.com/nelsonjchen/op-replay-clipper/assets/5363/8f970c76-21d1-4209-b0e1-3eb6989feea8
Copying the car in front to get around someone waiting for the left turn
https://github.com/nelsonjchen/op-replay-clipper/assets/5363/9f845b8d-e4aa-4ab3-8785-8d09b83c9d8b
Learn how to bookmark, preserve, and flag interesting points on a drive/route.
Preservation saves the last couple segments from being deleted on your device as well.
With the car on, within a minute after an incident when it is safe to do so:
[!TIP] If you find it a hassle to reach out and touch the device or it is too inconvenient, try installing a custom macropad like the 🦾 comma three Faux-Touch keyboard!
There is a JWT Token input field. This is for users who do not wish to set a route to be "Public access". There is a major catch though. The JWT Token is valid for 90 days and is irrevocable in any way. Password changes from SSO account logins like in Comma Connect will not invalidate the token. Addtionally, it is not granular, meaning it will give access to all routes for the user if leaked.
If you share a JWT Token with anyone, they will be able to access all your routes for 90 days with no possibility of revocation from you. This is why it's not recommended to use this feature unless you know what you're doing compared to the "Public access" method which is much easier to revoke access to.
Tokens can be obtained from visiting https://jwt.comma.ai/ and logging in with the same comma connect account type. Tokens should be about 181 characters or longer.
After you run something, just use your browser to "Duplicate" the tab, change the settings for the next thing, and press Run. Replicate will queue up jobs and if necessary, even scale up to run multiple jobs in parallel. Very cool!
360 videos are cool but sometimes you want a normal video pointing at a specific direction or directions from that data.
https://github.com/user-attachments/assets/08b51cee-f357-4afc-87f2-4c4d0f6aedba
With 360 videos, it is possible to reframe the 360 video so it is a non-360 video to a normal video pointing at a specific direction.
The best current way to do this is to use a 360 video editor like Insta360 Studio to reframe the video to a normal video. Simply load the 360 video into the editor and reframe the video to the desired direction.
There may be alternative software that'll do it and I will take pull requests to add them to this README, but this is the best way I know how to do it and it is free.
The 360 Forward Upon Wide rendering option scales input videos and renders the final result in a much higher 8K resolution to assist reframing with a high resolution forward video. The normal 360 option just glues the videos together.
If wanting to use 360 Forward Upon Wide, test with the non-360 Forward Upon Wide option and start with a value for forwardUponWideH
around "1.9" to dial in that value in first for that overlay before using the 360 Forward Upon Wide option to save money and time.
The real MVP is @deanlee for the replay tool in the openpilot project. The level of effort to develop the replay tool is far beyond this script. The script is just a wrapper around the replay tool to make it easy to use for clipping videos.
https://github.com/commaai/openpilot/blame/master/tools/replay/main.cc
A lot of the FFmpeg commands is based off of @ntegan1's research and documentation including a small disclosure of some but not all details by @incognitojam when @incognitojam was at comma.
https://discord.com/channels/469524606043160576/819046761287909446/1068406169317675078
@morrislee provided original data suitable to try to reverse engineer 360 clips.