search

nelsonjchen / op-replay-clipper

📽 Capture and develop clips of openpilot. UI optional. Already deployed on Replicate.com for YOUR immediate use!
https://replicate.com/nelsonjchen/op-replay-clipper
MIT License
88 stars 13 forks source link
github-codespaces openpilot replicate

readme

📽 openpilot Replay Clipper

Here's a small openpilot bug clip:

https://github.com/commaai/openpilot/assets/5363/97a6c767-9b67-4206-8ba7-b4030f08a8cd

Capture and develop clips of openpilot from comma.ai's Comma Connect.

The clipper can produce clips of:

All clip options have a configurable target file size option as platforms like Discord limit file upload sizes.

The clipper is deployed on Replicate:

https://replicate.com/nelsonjchen/op-replay-clipper

Replicate is an ultra-low-cost pay-as-you-go compute platform for running software jobs. Replicate is a great way to run this clipper as it's fast, easy to use, and you don't need to install anything on your computer or even deploy anything yourself. Just enter in the required information into the form, and Replicate will generate a clip. Expect to pay about ~$0.01 per clip but not even need to put in any payment details until you've reached a generously large level of usage.

[!WARNING] comma devices should not be used as primary dashcams for numerous reasons!

They are still great as a backup dashcam and for other purposes though.

Terminology

Requirements

Non-Requirements

Quick Usage

We assume you've already paired your device and have access to the device with your comma connect account.

  1. Visit comma connect and select a route.
  2. Scrub to the time you want to clip.
    • In this example, I've scrubbed to a time where I want to make a small clip of behind this cool car.
    • image

  3. Now I need to select the portion of the route I want to clip. Here's a video of what that UI looks like

    • See how I drag and select a portion.
    • You can see me make a mistake but pressing the left arrow (←) in the top-left corner lets me re-expand and try to trim again.
    • The clipper has a maximum length of 5 minutes. Try to select a portion that's less than that. Try to aim for 20 seconds to a minute though as everybody else has short attention spans.

    • Video:

      https://github.com/commaai/openpilot/assets/5363/504665de-9222-4e6b-b090-c26cdcc7137a

  4. Once satisified with the selected portion, prepare the route and files for rendering.
    • Make sure all files are uploaded. Select "Upload All" under the "Files" dropdown if you haven't already and make sure it says uploaded. You may need to wait and your device may need to be on for a while for all files to upload.
      • The clipper only works with high-resolution files and needs all files that are part of the clip to be uploaded.
      • image
    • Make sure the route has "Public access" under "More info" turned on. You can set this to off after you're done with clip making.
      • image
  5. Copy the URL in the address bar of your browser to your clipboard. This is not the segment ID underneath the More Info button. In the case above, I've copied an old URL of "https://connect.comma.ai/fe18f736cb0d7813/1698203405863/1698203460702" to my clipboard. Note: comma has changed the URL format since this step was originally written. New URLs are like "https://connect.comma.ai/fe18f736cb0d7813/000001bb--4c0c0efba9/21/90".
    • When you were adjusting the selected portion of the route in a previous step, it was changing those last two numbers in the browser address bar URL which is the start time and end time respectively.
    • "Share This Route" button if it is present will work too. Choose "copy to clipboard" or similar.
  6. Visit https://replicate.com/nelsonjchen/op-replay-clipper
  7. Under route, paste the URL you copied in the previous step.
    • image
  8. Tweak any settings you like.
  9. Press Run.
  10. Wait for the clip to render. It may take a few minutes.

  11. Once done, you can download the clip. If you want, turn off "Public access" on the route after you're done.

Gallery

Demonstration of speed or longitudinal behavior of openpilot with model-based longitudinal is nearly impossible or hard without this clipper. This video is of a good model based long behavior at highway speeds.

https://user-images.githubusercontent.com/5363/202886008-82cfbf02-d19a-4482-ab7a-59f96c802dd1.mp4

Cars can have bugs themselves. Here's my 2020 Corolla Hatchback phantomly braking on metal strips in stop and go traffic probably from the radar. Perhaps a future openpilot that doesn't depend on radar might be the one sanity checking the radar instead of the other way around currently. And another example of that in Portland.

https://user-images.githubusercontent.com/5363/219708673-4673f4ff-9b47-4c57-9be3-65f3ea703f3f.mp4

https://github.com/nelsonjchen/op-replay-clipper/assets/5363/1e59844b-46f8-4289-bea9-511db2718549

This is a video of a bug report where openpilot's lateral handling lost the lane.

https://user-images.githubusercontent.com/5363/205901777-53fd18f9-2ab5-400b-92f5-45daf3a34fbd.mp4

Lane cutting?

https://github.com/nelsonjchen/op-replay-clipper/assets/5363/d0ab3365-b5ef-4e05-84ee-370b88e8af02

Nav-assisted follow the road instead of taking the side road.

https://github.com/nelsonjchen/op-replay-clipper/assets/5363/8f970c76-21d1-4209-b0e1-3eb6989feea8

Copying the car in front to get around someone waiting for the left turn

https://github.com/nelsonjchen/op-replay-clipper/assets/5363/9f845b8d-e4aa-4ab3-8785-8d09b83c9d8b

Limitations

Usage Tips

Bookmark/Preserve

Learn how to bookmark, preserve, and flag interesting points on a drive/route.

Preservation saves the last couple segments from being deleted on your device as well.

With the car on, within a minute after an incident when it is safe to do so:

  1. Tap the screen to reveal a bookmark flag button in the bottom left if it isn't there already.
    • button_flag
  2. Tap that icon.
  3. This will result in small slivers of yellow in the timeline you can quickly hone in on.
    • flagged
  4. You should also set the route to preserve under More Info while you're working on it. Non-comma Prime users need to heed this especially since while files aren't deleted on the device, visiblity in and through comma connect sunsets after 3 days.
  5. With regards to the clipper usage, during the process in which you are honing in on the start and end boundaries of the clip, your upper bound of the clip will nearly all the time be at that yellow so your first or early drags to hone down should basically top out there and be very generous with the start time before the yellow.

[!TIP] If you find it a hassle to reach out and touch the device or it is too inconvenient, try installing a custom macropad like the 🦾 comma three Faux-Touch keyboard!

touchkey keyboard demo

Advanced Usage

JWT Token Input

There is a JWT Token input field. This is for users who do not wish to set a route to be "Public access". There is a major catch though. The JWT Token is valid for 90 days and is irrevocable in any way. Password changes from SSO account logins like in Comma Connect will not invalidate the token. Addtionally, it is not granular, meaning it will give access to all routes for the user if leaked.

If you share a JWT Token with anyone, they will be able to access all your routes for 90 days with no possibility of revocation from you. This is why it's not recommended to use this feature unless you know what you're doing compared to the "Public access" method which is much easier to revoke access to.

Tokens can be obtained from visiting https://jwt.comma.ai/ and logging in with the same comma connect account type. Tokens should be about 181 characters or longer.

Replicate can queue up jobs to run in parallel

After you run something, just use your browser to "Duplicate" the tab, change the settings for the next thing, and press Run. Replicate will queue up jobs and if necessary, even scale up to run multiple jobs in parallel. Very cool!

Reframing 360 and 360 Forward Upon Wide to a normal video

360 videos are cool but sometimes you want a normal video pointing at a specific direction or directions from that data.

https://github.com/user-attachments/assets/08b51cee-f357-4afc-87f2-4c4d0f6aedba

With 360 videos, it is possible to reframe the 360 video so it is a non-360 video to a normal video pointing at a specific direction.

The best current way to do this is to use a 360 video editor like Insta360 Studio to reframe the video to a normal video. Simply load the 360 video into the editor and reframe the video to the desired direction.

insta360

There may be alternative software that'll do it and I will take pull requests to add them to this README, but this is the best way I know how to do it and it is free.

The 360 Forward Upon Wide rendering option scales input videos and renders the final result in a much higher 8K resolution to assist reframing with a high resolution forward video. The normal 360 option just glues the videos together.

If wanting to use 360 Forward Upon Wide, test with the non-360 Forward Upon Wide option and start with a value for forwardUponWideH around "1.9" to dial in that value in first for that overlay before using the 360 Forward Upon Wide option to save money and time.

Credits

UI

The real MVP is @deanlee for the replay tool in the openpilot project. The level of effort to develop the replay tool is far beyond this script. The script is just a wrapper around the replay tool to make it easy to use for clipping videos.

https://github.com/commaai/openpilot/blame/master/tools/replay/main.cc

Video-only

A lot of the FFmpeg commands is based off of @ntegan1's research and documentation including a small disclosure of some but not all details by @incognitojam when @incognitojam was at comma.

https://discord.com/channels/469524606043160576/819046761287909446/1068406169317675078

@morrislee provided original data suitable to try to reverse engineer 360 clips.