initial plan and ideas

[Armandpl]

• [x] #5
  • use the bicycle model
  • or learn a model using deep learning and data from the actual car requires localization
• [x] #6
  • use a render engine/game engine. I think blender is a good choice here
  • gen script should log its settings to wandb?
  • and/or execute the trajectories on an actual track with the car requires localization + executing the trajectories
  • make sure to record extrinsic and intrinsinc cam parameters and match those to our hardware. for synthetic data not a big deal since we can probably re-generate it
  • make sure to add noise to the trajectories or to sim. or else the neural net wont be able to recover when outside the ideal trajectory.
  • see e2e lateral planning
  • think about if we want the model to output anything for longitudinal?
  • think about augmentation. e.g diffusion, pix2pix? speed blur? lighting? warp to change pitch/yaw?

• 7

  • match frequency at which we sample images with the control frequency we're shooting for
  • think about what "trajectory means" e.g if we go faster or sample differently the points are going to be closer or further appart. Does the distance between points encode speed? basically think about how to handle the longitudinal aspect?
  • look into camera calibration. maybe we want to calibrate. start without it imo
  • or maybe we can avoid calibrating since we generate synthetic data and can match the intrinsic and extrinsic params
  • maybe also predict pose from the NN! might be a bit difficult and we can start without it see comment below re: state estimator
  • https://kornia.readthedocs.io/en/v0.1.4/geometry.camera.perspective.html use that to project traj on images for viz

• 2

  • get global shutter cam
  • stay on jetson nano? make sure we can install dependencies? python3.7?
  • make sure our IMU works, maybe get a second one
  • rework the way the electronics fit on the car
  • make sure the cam doesn't see the car? or we'll have to mask it to the nn? or add it to the synth data?
  • re-check the reduction ratio. I feel the car doesn't accelerate/brake fast enough on the new chassis

• 9

  • add a messaging system, possibly cereal
  • have daemons running to:
  • fetch data from sensors (IMU, motor encoder, camera)
  • run model inference
  • log data to disk using mcap
  • execute trajectories
  • estimate state (kalman rednose)
  • actually could we use openpilot? and save a bunch of engineering
• validate state estimator need localization
  • drive manually on the track to get sensor data + ground truth localization
    • from this dataset tune a kalman filter to estimate system state
• validate speed estimator?
  • can we validate it against the motor speed?
  • estimated speeds along each axis should sum to motor speed * reduction ratio?
• learn mpc
  • try executing the trajectories by using external localization
  • good time to collect an image test set for the trajectory preds
    • maybe introduce voluntary noise (maybe in the same way we introduce noise for the synthetic data?) to get trajectories outside the optimal ones
• test the whole stack in simulation
  • e.g carla or smaller sims (check comma ai bounty)

[Armandpl]

Simpler version

To make it easy to build a v1 we could go for predicting throttle + steering from synth dataset. This reduces complexity a lot as we wouldn't need a tracking system, a messaging system, executing MPC, multiple threads and daemons, a state estimator etc... we could just run the model in a python loop and call it a day. We know it can work and it would be a good first step to validate synth -> real (we know it works since we know people who have done it)

We could also try to install openpilot on the jetson nano and gauge how easy/hard it is to mod. I suspect it might be doable! I think we would also need to install ubuntu 20, and we could try this image

[Armandpl]

Re: state estinator + MPC:

We have a few sensors on the car: a camera, an IMU (accelerometer, gyroscope and magnetometer) and an encoder on the motor. Estimating state (position + velocity) with the IMU only is annoying because small errors are going to accumulate and the state estimation is going to drift.
But I think we only need the velocity estimation along each axis to run the MPC to execute the predicted trajectory. The predicted trajectory is relative to the car so we could just estimate the car position to be 0,0,0. We actually don't need the real position to compare the Model (model as in MPC) output to the target trajectory. Also, the measured motor speed is absolute, it doesn't drift so we should be able to estimate the speed without drift