Errors occur when running simulate_mpc.py

[ZhaiMen-Hub]

We are trying to do some further work based on your paper Low-Level Control of a Quadrotor With Deep Model-Based Reinforcement Learning Hence, we wish to reproduce your experiment about implementing neural network and random MPC on quadrotor crazy_file, but some errors occur when we run the script simulate_mpc.py as below. python learn/simulate_mpc.py robot=iono How can we deal with these errors? And I also wonder, which commit is the perfect version of your work in that paper? So our reproduction will be much faster. Thanks.

Error output for running simulate_mpc.py in branch master

$ python learn/simulate_mpc.py robot=iono
[2022-04-06 22:13:13,215][__main__][INFO] - ============= Configuration =============
[2022-04-06 22:13:13,222][__main__][INFO] - Config:
bo:
  iterations: 10
  sim: false
checkpoint_file: trial_{}.dat
data: false
dimension: all
env:
  params:
    dt: 6
    du: 4
    dx: 6
    flight_mode: basic
    name: iono
experiment:
  num_roll: 20
  random: 5
  repeat: 1
  seeds: 1
load:
  data_labels:
  - pwm1
  - pwm2
  - pwm3
  - pwm4
  - ax
  - ay
  - az
  - wx
  - wy
  - wz
  - pitch
  - roll
  - yaw
  dir: false
  fname: /Users/nato/Documents/Berkeley/Research/Codebases/dynamics-learn/_logged_data_autonomous/iono/video-setup/data_video.txt
  type: .csv
metric:
  minimize: false
  name: Square
model:
  class: learn.models.GeneralNN
  params:
    datahandler:
      class: learn.utils.nn.ModelDataHandler
      params:
        X:
          U:
            class: sklearn.preprocessing.MinMaxScaler
            params:
              feature_range:
              - -1.0
              - 1.0
          class: sklearn.preprocessing.StandardScaler
          dX:
            class: sklearn.preprocessing.MinMaxScaler
            params:
              feature_range:
              - -1.0
              - 1.0
          sine_expand: []
    delta_state_targets:
    - pitch
    - roll
    - yaw
    dt: 6
    du: 4
    dx: 6
    extra_inputs: null
    flight_mode: basic
    history: 1
    ignore_in:
    - omega
    - vbat
    name: iono
    optimizer:
      batch: 18
      epochs: 17
      lr: 0.0025
      lr_schedule:
      - 30
      - 6
      preprocess: true
      split: 0.8
      verbose: false
    training:
      activ: learn.utils.nn.Swish
      cluster: 0
      dropout: 0
      dt: ${env.params.dt}
      du: ${env.params.du}
      dx: ${env.params.dx}
      ensemble: false
      hid_depth: 2
      hid_width: 250
      mode: delta
      plot_loss: true
      probl: true
      split: false
    true_state_targets:
    - linax
    - linay
    - linaz
mpc:
  params:
    N: 20000
    T: 5
    hold: true
    mode: random
pid:
  params:
    dt: 0.02
    equil:
    - 1500
    - 1500
    - 1500
    - 1500
    int_max: 1000
    living_cost: 0.1
    max_pwm: 3000
    max_values:
    - 500
    - 25
    - 50
    min_pwm: 0
    min_values:
    - 1
    - 0.01
    - 0.1
    mode: INTEG
    pitch_mult:
    - 1
    - 1
    - -1
    - -1
    pry:
    - 0
    - 1
    - 2
    roll_mult:
    - 1
    - -1
    - -1
    - 1
    terminal_cost: 1
policy:
  mode: mpc
  params:
    period: 1
random_seed: 1
robot: iono
save: true

[2022-04-06 22:13:13,222][__main__][INFO] - =========================================
Traceback (most recent call last):
  File "learn/simulate_mpc.py", line 239, in <module>
    sys.exit(mpc())
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/hydra/main.py", line 25, in decorated_main
    strict,
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/hydra/_internal/utils.py", line 48, in run_hydra
    hydra.run(overrides=args.overrides)
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 111, in run
    job_subdir_key=None,
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/hydra/plugins/common/utils.py", line 91, in run_job
    ret.return_value = task_function(task_cfg)
  File "learn/simulate_mpc.py", line 43, in mpc
    env = gym.make(env_name)
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/gym/envs/registration.py", line 142, in make
    return registry.make(id, **kwargs)
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/gym/envs/registration.py", line 86, in make
    spec = self.spec(path)
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/gym/envs/registration.py", line 115, in spec
    raise error.Error('Attempted to look up malformed environment ID: {}. (Currently all IDs must be of the form {}.)'.format(id.encode('utf-8'), env_id_re.pattern))
gym.error.Error: Attempted to look up malformed environment ID: b'iono'. (Currently all IDs must be of the form ^(?:[\w:-]+\/)?([\w:.-]+)-v(\d+)$.)
```bash

[ZhaiMen-Hub]

I attempt to solve this problem via check out to an early commit but fail. early commit: 29c93daa 2020-01-31 "improving visualization of training"

Error output:

$ python learn/simulate_mpc.py robot=iono
[2022-04-06 22:19:56,420][__main__][INFO] - ============= Configuration =============
[2022-04-06 22:19:56,425][__main__][INFO] - Config:
bo:
  iterations: 15
checkpoint_file: trial_{}.dat
control:
  mode: INTEG
  params:
    dt: 0.01
    equil:
    - 16383
    - 16383
    - 16383
    - 16383
    max_pwm: 65535
    min_pwm: 0
    period: 10
  pid:
    params:
      max_values:
      - 10000
      - 1000
      - 1000
      min_values:
      - 0
      - 0
      - 0
dimension: all
experiment:
  num_r: 15
  r_len: 2500
  seeds: 10
load:
  data_labels:
  - pwm1
  - pwm2
  - pwm3
  - pwm4
  - ax
  - ay
  - az
  - wx
  - wy
  - wz
  - pitch
  - roll
  - yaw
  dir: false
  fname: /Users/nol/Documents/code-bases/data/iono/ex1_test.txt
  type: .csv
model:
  class: learn.models.GeneralNN
  params:
    datahandler:
      class: learn.utils.nn.ModelDataHandler
      params:
        X:
          U:
            class: sklearn.preprocessing.MinMaxScaler
            params:
              feature_range:
              - -1.0
              - 1.0
          class: sklearn.preprocessing.StandardScaler
          dX:
            class: sklearn.preprocessing.MinMaxScaler
            params:
              feature_range:
              - -1.0
              - 1.0
          sine_expand: []
    delta_state_targets:
    - pitch
    - roll
    - yaw
    dt: 6
    du: 4
    dx: 6
    extra_inputs: null
    flight_mode: basic
    history: 2
    ignore_in:
    - omega
    - vbat
    name: iono
    optimizer:
      batch: 18
      epochs: 3
      lr: 0.0025
      lr_schedule:
      - 30
      - 6
      preprocess: true
      split: 0.8
      verbose: false
    training:
      activ: learn.utils.nn.Swish
      dropout: 0
      dt: ${env.params.dt}
      du: ${env.params.du}
      dx: ${env.params.dx}
      ensemble: false
      hid_depth: 2
      hid_width: 250
      mode: delta
      plot_loss: true
      probl: true
      split: false
    true_state_targets:
    - linax
    - linay
    - linaz
policy:
  mode: mpc
  params:
    N: 20000
    T: 5
    hold: true
    period: 10
random_seed: 1
robot: iono
save: true

[2022-04-06 22:19:56,425][__main__][INFO] - =========================================
[2022-04-06 22:19:56,430][__main__][INFO] - Random Seed: 0
[2022-04-06 22:19:56,782][learn.trainer][INFO] - Training Model
successful init?
Traceback (most recent call last):
  File "learn/simulate_mpc.py", line 138, in <module>
    sys.exit(mpc())
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/hydra/main.py", line 25, in decorated_main
    strict,
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/hydra/_internal/utils.py", line 48, in run_hydra
    hydra.run(overrides=args.overrides)
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 111, in run
    job_subdir_key=None,
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/hydra/plugins/common/utils.py", line 91, in run_job
    ret.return_value = task_function(task_cfg)
  File "learn/simulate_mpc.py", line 50, in mpc
    model, train_log = train_model(X, U, dX, cfg.model)
  File "/home/jc/JC/px4/dynamicslearn/learn/trainer.py", line 105, in train_model
    acctest, acctrain = model.train_cust((X, U, dX), model_cfg.params)
  File "/home/jc/JC/px4/dynamicslearn/learn/models/model_general_nn.py", line 275, in train_cust
    dataset)  # trainLoader, testLoader)
  File "/home/jc/JC/px4/dynamicslearn/learn/models/model_general_nn.py", line 330, in _optimize
    output = self.forward(input)  # compute the output
  File "/home/jc/JC/px4/dynamicslearn/learn/models/model_general_nn.py", line 128, in forward
    x = self.features(x)
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/torch/nn/functional.py", line 1370, in linear
    ret = torch.addmm(bias, input, weight.t())
RuntimeError: size mismatch, m1: [18 x 10], m2: [20 x 250] at /opt/conda/conda-bld/pytorch_1579022030672/work/aten/src/TH/generic/THTensorMath.cpp:136

[ZhaiMen-Hub]

We are trying to do some further work based on your paper Low-Level Control of a Quadrotor With Deep Model-Based Reinforcement Learning Hence, we wish to reproduce your experiment about implementing neural network and random MPC on quadrotor crazy_file, but some errors occur when we run the script simulate_mpc.py as below. python learn/simulate_mpc.py robot=iono How can we deal with these errors? And I also wonder, which commit is the perfect version of your work in that paper? So our reproduction will be much faster. Thanks.

Error output for running simulate_mpc.py in branch master

$ python learn/simulate_mpc.py robot=iono
[2022-04-06 22:13:13,215][__main__][INFO] - ============= Configuration =============
[2022-04-06 22:13:13,222][__main__][INFO] - Config:
bo:
  iterations: 10
  sim: false
checkpoint_file: trial_{}.dat
data: false
dimension: all
env:
  params:
    dt: 6
    du: 4
    dx: 6
    flight_mode: basic
    name: iono
experiment:
  num_roll: 20
  random: 5
  repeat: 1
  seeds: 1
load:
  data_labels:
  - pwm1
  - pwm2
  - pwm3
  - pwm4
  - ax
  - ay
  - az
  - wx
  - wy
  - wz
  - pitch
  - roll
  - yaw
  dir: false
  fname: /Users/nato/Documents/Berkeley/Research/Codebases/dynamics-learn/_logged_data_autonomous/iono/video-setup/data_video.txt
  type: .csv
metric:
  minimize: false
  name: Square
model:
  class: learn.models.GeneralNN
  params:
    datahandler:
      class: learn.utils.nn.ModelDataHandler
      params:
        X:
          U:
            class: sklearn.preprocessing.MinMaxScaler
            params:
              feature_range:
              - -1.0
              - 1.0
          class: sklearn.preprocessing.StandardScaler
          dX:
            class: sklearn.preprocessing.MinMaxScaler
            params:
              feature_range:
              - -1.0
              - 1.0
          sine_expand: []
    delta_state_targets:
    - pitch
    - roll
    - yaw
    dt: 6
    du: 4
    dx: 6
    extra_inputs: null
    flight_mode: basic
    history: 1
    ignore_in:
    - omega
    - vbat
    name: iono
    optimizer:
      batch: 18
      epochs: 17
      lr: 0.0025
      lr_schedule:
      - 30
      - 6
      preprocess: true
      split: 0.8
      verbose: false
    training:
      activ: learn.utils.nn.Swish
      cluster: 0
      dropout: 0
      dt: ${env.params.dt}
      du: ${env.params.du}
      dx: ${env.params.dx}
      ensemble: false
      hid_depth: 2
      hid_width: 250
      mode: delta
      plot_loss: true
      probl: true
      split: false
    true_state_targets:
    - linax
    - linay
    - linaz
mpc:
  params:
    N: 20000
    T: 5
    hold: true
    mode: random
pid:
  params:
    dt: 0.02
    equil:
    - 1500
    - 1500
    - 1500
    - 1500
    int_max: 1000
    living_cost: 0.1
    max_pwm: 3000
    max_values:
    - 500
    - 25
    - 50
    min_pwm: 0
    min_values:
    - 1
    - 0.01
    - 0.1
    mode: INTEG
    pitch_mult:
    - 1
    - 1
    - -1
    - -1
    pry:
    - 0
    - 1
    - 2
    roll_mult:
    - 1
    - -1
    - -1
    - 1
    terminal_cost: 1
policy:
  mode: mpc
  params:
    period: 1
random_seed: 1
robot: iono
save: true

[2022-04-06 22:13:13,222][__main__][INFO] - =========================================
Traceback (most recent call last):
  File "learn/simulate_mpc.py", line 239, in <module>
    sys.exit(mpc())
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/hydra/main.py", line 25, in decorated_main
    strict,
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/hydra/_internal/utils.py", line 48, in run_hydra
    hydra.run(overrides=args.overrides)
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 111, in run
    job_subdir_key=None,
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/hydra/plugins/common/utils.py", line 91, in run_job
    ret.return_value = task_function(task_cfg)
  File "learn/simulate_mpc.py", line 43, in mpc
    env = gym.make(env_name)
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/gym/envs/registration.py", line 142, in make
    return registry.make(id, **kwargs)
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/gym/envs/registration.py", line 86, in make
    spec = self.spec(path)
  File "/home/jc/application/anaconda3/envs/ml/lib/python3.7/site-packages/gym/envs/registration.py", line 115, in spec
    raise error.Error('Attempted to look up malformed environment ID: {}. (Currently all IDs must be of the form {}.)'.format(id.encode('utf-8'), env_id_re.pattern))
gym.error.Error: Attempted to look up malformed environment ID: b'iono'. (Currently all IDs must be of the form ^(?:[\w:-]+\/)?([\w:.-]+)-v(\d+)$.)
```bash

It seems that the environment for simulation can not be made but I am not sure.

[natolambert]

Hi, these are two different things. First simulate_mpc.py should be one of the easier ones to get running.

The first bug, seems to be caused by copy-pasting text in a weird way. It's complaining about the env id? Did you build your env from the requirements? This is all quite old so newer versions of tools could cause bugs.

raise error.Error('Attempted to look up malformed environment ID: {}. (Currently all IDs must be of the form {}.)'.format(id.encode('utf-8'), env_id_re.pattern)) gym.error.Error: Attempted to look up malformed environment ID: b'iono'. (Currently all IDs must be of the form ^(?:[\w:-]+\/)?([\w:.-]+)-v(\d+)$.)

With regards to the shapes, I have seen this before. Sorry it is still persisting. The iono environment is harder to work with because it uses real world data. I would try starting with iono_sim. (iono itself is not supported in support_mpc).

I would recommend you look at our newer, much better supported code: https://github.com/facebookresearch/mbrl-lib

Then, writing quadrotor firmware to map to that (happy to answer questions on that).