Open jc-bao opened 1 year ago
possible issues:
multi-dynamic model-free RL: (not very important)
:star: Inference-based adaptation module:
Generalizability given model mismatch (OOD) -> introduce another optimization method
Feedback given imprecise estimation -> introduce a method to identify the problem
TODO
Research question: how to enable a model-free method to achieve better performance given complex dynamics?
Expected result: better tracking performance in residue dynamics.
assumption: the robot learn a robust policy which is agnoistic to different residue dynamics.
description: first learn without period disturbance, then add it back
results: the same (11cm error)
conclusion: curriculum cannot solve the problem. The main issue still comes from the suboptimal strategy, given partially random dynamics.
:warning: Deprecated due to the case might not match the real-world case. (too much human-engineering dynamics. )
:eye_speech_bubble: Hindsight: wrong trail. Our research question is to achieve high-performance adaptive control, not better model-free performance.
Question: how to enable adaptor to generalize to unseen scenarios?
assumption:
description
adaptor
-> consider adaptation module during the training
compressor
compressor
(0.0, 1.0, 10.0) -> instead of mimicking the expert module, we can also update the policy
Results
Origin (L2 loss to z)
Train(policy=expert, ) Adapt(policy=adaptor, ) Perfect model
Method | Expert | Adapt begin | Adapt end |
---|---|---|---|
Baseline | 0.0444 | 0.0518 | 0.0395 |
ARC0 | 0.0444 | 1.3167 | 0.3180 |
ARC1 | 0.0444 | 0.7872 | 0.1423 |
ARC10 | 0.0444 | 0.0770 | 0.0363 |
ARC50 | 0.0444 | 0.0572 | 0.0384 |
TUC-1 | 0.1663 | 0.1810 | 0.1635 |
TUC-3 | 0.1229 | 0.0784 | 0.0961 |
Imperfect model
Method | Expert | Adapt begin | Adapt end |
---|---|---|---|
Baseline | 0.5054 | 0.1674 | 0.2007 |
Baseline-Adapt | 0.0379 | 0.0435 | 0.0416 |
TUC-3 | 0.5338 | 0.2005 | 0.2476 |
TUC-3-Adapt | 0.0453 | 0.0514 | 0.0470 |
:eye_speech_bubble: Hindsight: wrong trail. Dive into detail before identifying the true research question.
Question: under what circumstances can we observe a significant performance drop for the RMA algorithm?
$f(v,w) = x^T M x + C$, where $x=[v, w], v \in R^3, w \sim \mathcal{U}(-1,1) \in R^{d_w} $ $C \sim \mathcal{U}(-1,1) \in R^3, M \sim \mathcal{U}(-1,1) \in R^{3 \times 4 \times 4}$
Last 10 steps average tracking error. | $d_w=2$ Expert | $d_w=2$ RMA before adaptation | $d_w=2$ RMA after adaptation | Vanilla(Robust) |
---|---|---|---|---|
0.027 | 0.065 | 0.029 | 0.147 | |
$d_w=1$ Expert | $d_w=1$ RMA before adaptation | $d_w=1$ RMA after adaptation | ||
0.063 | 0.101 | 0.077 | ||
$d_w=1$ C-4 Expert | $d_w=1$ C-4 RMA before adaptation | $d_w=1$ C-4 RMA after adaptation | ||
0.067 | 0.097 | 0.074 | ||
$d_w=0$ Expert | $d_w=0$ RMA before adaptation | $d_w=0$ RMA after adaptation | ||
0.008 | 0.028 | 0.007 |
*C-4: use MLP to compress all parameters to a 4-dimensional embedding.
[128, 128] Mlp initialized with nn.init.orthogonal_(m.weight, gain=1), nn.init.uniform_(m.bias, -0.2, 0.2).
$d_w=1$ Expert | $d_w=1$ RMA before adaptation | $d_w=1$ RMA after adaptation | Vanilla(Robust) |
---|---|---|---|
0.0141 | 0.0294 | 0.0161 | 0.0354 |
$d_w=1$ C-4 Expert | $d_w=1$ C-4 RMA before adaptation | $d_w=1$ C-4 RMA after adaptation | ![]() |
0.0169 | 0.0183 | 0.0164 | ![]() |
low sensitivity to u
$d_w=0$ Expert | $d_w=0$ RMA before adaptation | $d_w=0$ RMA after adaptation | Vanilla(Robust) |
---|---|---|---|
0.0189 | 0.0215 | 0.0183 | 0.0640 |
$d_w=1$ Expert | $d_w=1$ RMA before adaptation | $d_w=1$ RMA after adaptation | |
0.0386 | 0.0422 | 0.0274 | |
$d_w=1$ C-4 Expert | $d_w=1$ C-4 RMA before adaptation | $d_w=1$ C-4 RMA after adaptation | ![]() |
0.0103 | 0.0120 | 0.0092 | ![]() |
high sensitivity to u (by using non-zero bias mlp. )
$d_w=0$ Expert | $d_w=0$ RMA before adaptation | $d_w=0$ RMA after adaptation | Vanilla(Robust) |
---|---|---|---|
0.0287 | 0.0340 | 0.0296 | 0.0586 |
$d_w=1$ Expert | $d_w=1$ RMA before adaptation | $d_w=1$ RMA after adaptation | |
0.0687 | 0.0449 | 0.0331 | |
$d_w=1$ C-4 Expert | $d_w=1$ C-4 RMA before adaptation | $d_w=1$ C-4 RMA after adaptation | ![]() |
0.0367 | 0.0331 | 0.0269 | ![]() |
high sensitivity + force scale *2
$d_w=1$ Expert | $d_w=1$ RMA before adaptation | $d_w=1$ RMA after adaptation | Vanilla |
---|---|---|---|
0.0687 | 0.0449 | 0.0331 | |
$d_w=1$ C-4 Expert | $d_w=1$ C-4 RMA before adaptation | $d_w=1$ C-4 RMA after adaptation | ![]() |
0.0367 | 0.0331 | 0.0269 | ![]() |
Conclusion:
Interesting observation: the network can handle the unobservable parameters pretty well.
OOD Expert | OOD RMA before adaptation | OOD RMA after adaptation | Plot |
---|---|---|---|
0.2074 | 0.4166 | 0.2110 | ![]() |
w/o OOD Expert | w/o OOD RMA before adaptation | w/o OOD RMA after adaptation | Plot |
0.0687 | 0.0449 | 0.0331 | ![]() |
C-4 OOD Expert | C-4 OOD RMA before adaptation | C-4 OOD RMA after adaptation | Plot |
0.3596 | 0.3225 | 0.3516 | |
C-4 w/o OOD Expert | C-4 w/o OOD RMA before adaptation | C-4 w/o OOD RMA after adaptation | Plot |
0.0629 | 0.0482 | 0.0403 |
w/o OOD | OOD | |
---|---|---|
mass-decay | ![]() |
![]() |
decay-param | ![]() |
![]() |
param-mass | ![]() |
![]() |
Updated results
Policy | Expert | Before adaptation | After adaptation |
---|---|---|---|
Baseline | 0.2287 | nan | nan |
Baseline-OOD(full dynamic param) | 0.2567 | nan | nan |
Baseline-OOD | 0.3688 | nan | nan |
RMA | 0.0945 | 0.1784 | 0.1455 |
RMA-OOD(full dynamic param) | 0.1448 | 0.2841 | 0.2186 |
RMA-OOD | 0.2119 | 0.3287 | 0.2714 |
Policy | Expert | Before adaptation | After adaptation |
---|---|---|---|
No OOD | 0.1183 | 0.2247 | 0.1317 |
intra OOD full | 0.1883 | 0.3352 | 0.2461 |
extra OOD full | 0.2446 | 0.4018 | 0.3160 |
only intra res dyn | 0.1008 | 0.2032 | 0.1161 |
only extra res dyn | 0.1134 | 0.2205 | 0.1313 |
50% | 70% | 100% |
---|---|---|
![]() |
![]() |
![]() |
Center | Left | Out | Full |
---|---|---|---|
![]() |
![]() |
![]() |
![]() |
Test case: compress 2d/3d/4d residue dynamic parameters with environment encoder to a 2d embedding.
2d | 3d | 4d |
---|---|---|
![]() |
dim3=0 ![]() |
dim3,4=[0,0] ![]() |
dim3=0.5![]() |
dim3,4=[1,0]![]() |
|
dim3=1.0![]() |
dim3,4=[1,1]![]() |
Conclusion: In OOD parameters, the encoder space is still continuous.
2 residue dynamic parameters
none OOD | OOD | OOD |
---|---|---|
![]() |
![]() |
![]() |
No compressor visualization.
3 disturbance values
visualize disturbance mapping
mass | disturb | decay |
---|---|---|
![image-20230209191434616](/Users/reedpan/Library/Application Support/typora-user-images/image-20230209191434616.png) | ![image-20230209191535401](/Users/reedpan/Library/Application Support/typora-user-images/image-20230209191535401.png) | ![]() |
resdyn & force scale | ⚠️force scale | ⚠️resdyn |
![]() |
![]() |
![]() |
mass | disturb | decay |
---|---|---|
![]() |
![]() |
![image-20230209192350901](/Users/reedpan/Library/Application Support/typora-user-images/image-20230209192350901.png) |
resdyn & force scale | force scale | resdyn |
![]() |
![]() |
![]() |
None OOD mass_max | OOD mass_max | None OOD disturb max | OOD disturb max |
---|---|---|---|
![]() |
![]() |
![]() |
![]() |
None OOD decay_max | OOD decay_max | none OOD res param | OOD res param |
![]() |
![]() |
![]() |
![]() |
none OOD force scale | OOD force scale | noneOOD all max | OOD all max |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
check with other kind of parameters.
1 | 2 | 3 | 4 | 5 |
---|---|---|---|---|
![image-20230204213708462](/Users/reedpan/Library/Application Support/typora-user-images/image-20230204213708462.png) | ![image-20230204213758802](/Users/reedpan/Library/Application Support/typora-user-images/image-20230204213758802.png) | ![]() |
![]() |
![]() |
PCA analysis with higher dimensional parameters
OOD performance evaluation
OOD | ![]() |
---|---|
none OOD | ![]() |
Training set | left boundary(OOD) | center parameter | right boundary(OOD) |
---|---|---|---|
100% | ![]() |
![]() |
![]() |
50% | ![]() |
![]() |
![]() |
$f(v,u,w)$
$d_w=1$ Expert | $d_w=1$ RMA before adaptation | $d_w=1$ RMA after adaptation | Vanilla(Robust) |
---|---|---|---|
0.7379 | 0.8989 | 0.6864 | 0.4900 |
$d_w=1$ C-4 Expert | $d_w=1$ C-4 RMA before adaptation | $d_w=1$ C-4 RMA after adaptation | ![]() |
0.6545 | 0.5992 | 0.5941 | ![]() |
$f(v,u)$
$d_w=1$ Expert | $d_w=1$ RMA before adaptation | $d_w=1$ RMA after adaptation | Vanilla(Robust) |
---|---|---|---|
0.0685 | 0.0808 | 0.1392 | 0.0872 |
$d_w=1$ C-4 Expert | $d_w=1$ C-4 RMA before adaptation | $d_w=1$ C-4 RMA after adaptation | |
0.0659 | 0.0900 | 0.1254 | ![]() |
Force Scale =[3, 3, 3]
Training with fitted model (32 trajectory, mean error=0.29) Fail to stablize.
$d_w=1$ Expert | $d_w=1$ RMA before adaptation | $d_w=1$ RMA after adaptation |
---|---|---|
0.4646 | 0.3908 | 0.2465 |
Training with fitted model (128 trajectory, mean error=0.07)
$d_w=1$ Expert | $d_w=1$ RMA before adaptation | $d_w=1$ RMA after adaptation |
---|---|---|
0.6057 | 0.5747 | 0.5860 |
Training with fitted model (512 trajectory, mean error=0.01)
$d_w=1$ Expert | $d_w=1$ RMA before adaptation | $d_w=1$ RMA after adaptation |
---|---|---|
0.1654 | 0.5242 | 0.3003 |
$d_w=1$ Expert | $d_w=1$ RMA before adaptation | $d_w=1$ RMA after adaptation |
---|---|---|
0.0718 | 0.0588 | 0.0355 |
Force Scale =[2, 4, 2]
Training with fitted model (32 trajectory, mean error=0.327) Fail to stablize.
$d_w=1$ Expert | $d_w=1$ RMA before adaptation | $d_w=1$ RMA after adaptation |
---|---|---|
1.4253 | 0.5395 | 0.7555 |
Training with fitted model (64 trajectory, mean error=0.102)
$d_w=1$ Expert | $d_w=1$ RMA before adaptation | $d_w=1$ RMA after adaptation |
---|---|---|
0.1978 | 0.1346 | 0.1117 |
Training with fitted model (128 trajectory, mean error=0.033)
$d_w=1$ Expert | $d_w=1$ RMA before adaptation | $d_w=1$ RMA after adaptation |
---|---|---|
0.3220 | 0.2422 | 0.1854 |
$d_w=1$ Expert | $d_w=1$ RMA before adaptation | $d_w=1$ RMA after adaptation |
---|---|---|
0.0781 | 0.0703 | 0.0447 |
$d_w=1$ Expert | $d_w=1$ RMA before adaptation | $d_w=1$ RMA after adaptation |
---|---|---|
0.1910 | 0.1951 | 0.1660 |
This issue is used to keep a record of the research emphasis and engineering effort of each research period.