CoRL ToDos

[alexBeuth]

TODO:

After rebuttal

• [x] License (MIT) and Max Planck Society

Alex

• [x] Read papers GP-SSM

  • [x] learn a dynamics model, use the model for exploration strategy, maybe update more than one points
  • [x] identify structure, use this structure to make some assumptions on the safety function
  • [x] main point of the paper

• [x] general rewrite of GP approximation (3.2)

• [x] redo figures

• [x] 4-dimensional system

• [x] Redo 2-dimensional experiments

• [x] IAV affiliation

• [x] Matthias comments

• [x] Mention noise in discussion

• [ ] Read new papers of reviewers

Steve

• [x] proof extension, based on Bellman update equations
• [x] acknowledgments
• [x] rewrite conclusions
  • [x] Paper by Kirchner (ETH)
  • [x] State-dependent uncertainty
  • [x] replacing old samples ("closeness")
  • [x] mixing in dynamics
• [x] General improvements based on reviews

prep for rebuttal

• [x] Alex get working example with the 4-D spaceship model: 02.08
  • [x] Test convergence
  • [x] Test existing 2d examples
• [x] Steve implement spaceship model with 2-D action space: 29.07
• [x] Alex get working example with the 5-D spaceship model: Not needed anymore?
• [x] Alex handle the last comment from Matthias

prep for Sept 7

• [x] Add commentary in conclusions: determinitic dynamics assumption is theoretically not required, though we have not investigated this and expect that practical complications of interest will arise.
• [x] Split into modules
  • [x] models and viability Steve
  • [x] GP learning Alex (also merge in submission branch)
• [x] label submission version of CoRL. Add in LaTeX files of the paper
• [ ] Obtain better graphs
  • [x] figures... we are not always converging to a safe subset
  • [ ] With multiple trajectories on the parameters, and get a nice convergence
  • [ ] Other types of graphs? In suppl. material? Comparison with Random Search, convergence/iterations, and failure rate. Alex
  • [ ] Comparison with cost-function not doing this
• [ ] Clean up code
  • [ ] remove viability computations for warm-start in estimate_measure. Q_V, Q_M etc. should be calculated by the user outside, and then passed to the learning class. Classes implemented in measure should not depend on viability
  • [x] data going into the sampler class... what does this contain? It shouldn't require any ground-truth data...
  • [ ] string together trajectories low priority
  • [ ] test function to run a bunch of trials with uniform random sampling {Steve, Alex}
  • [x] 3D example look up
  • [x] 5D example look up
  • [ ] Acrobot example? low

• [ ] Rewrite

  • [x] Point out notation Steve

  • [x] Point out examples is in the suppl. code

  • [x] Better colormaps Alex Use hatching for ground-truth, color for learned stuff

• [x] Appendix, with descriptions of additional examples
  • [x] convergence proof in appendix See rebuttal

---

Deadline

• [ ] Train GP hyperparameters with failures and infeasable points

• [x] Rewrite to be able to include different models

• [x] Arbitrary dynamics 2d

• [x] Q-Feas?

• [ ] Arbitrary dynamics more-d

• [x] states undiscrete

• [x] plots

• [x] clean up code for submission

  • [x] all examples of figures used in paper
  • [ ] bonus RL within the safe set

• [x] intro to GPs in 3 sentences

• [x] re-iterate on related work

• [ ] do the extra models

[sheim]

I've made a hovership model and a spaceship model. The hovership is basically the toy example but with continuous state-action space. The spaceship is similar, but has 2D state space (vertical and horizontal), and 1D action space (vertical thrusters). It is therefore underactuated. There is wind which alternates right and left depending on the height. You can also think of this like the Google Loom project.

The Acrobot would be a nice 5D Q-space example, but will require a bit more programming (it is sort of done, but I'm pretty sure some things need to be taken care of still...). Perhaps an easier high-D example is the space-ship, but with inertia (so you have positions and velocities).