Reward function investigation

[cathywu]

For env Walker2d-v1.

Note: local rllab/envs/mujoco/walker2d_env.py is not used. Can / should we use this instead?

From /Users/cathywu/anaconda/envs/rllabcathywu/lib/python3.5/site-packages/gym/envs/mujoco/walker2d.py:

    def _step(self, a):
        posbefore = self.model.data.qpos[0, 0]
        self.do_simulation(a, self.frame_skip)
        posafter, height, ang = self.model.data.qpos[0:3, 0]
        alive_bonus = 1.0
        reward = ((posafter - posbefore) / self.dt)
        reward += alive_bonus
        reward -= 1e-3 * np.square(a).sum()
        done = not (height > 0.8 and height < 2.0 and
                    ang > -1.0 and ang < 1.0)
        ob = self._get_obs()
        return ob, reward, done, {}

[cathywu]

Reward breakdown.

Forward velocity and control cost, respectively, for a rollout:

WALKER REWARD -1.70888205672 0.00349287494029
WALKER REWARD -2.24264778979 0.00406637826887
WALKER REWARD 0.239668921115 0.0031597688981
WALKER REWARD -1.7946215543 0.00233756125308
WALKER REWARD -0.706939956168 0.00412793893491
WALKER REWARD -2.0468429637 0.00160338001116
WALKER REWARD -2.21090924768 0.00397572871957
WALKER REWARD 0.158582021222 0.00538539460773
WALKER REWARD -1.49437104333 0.00546762098538
WALKER REWARD -0.316479628388 0.00533141348255
WALKER REWARD -2.40651862829 0.00332056525721
WALKER REWARD -2.17037999883 0.00482165212478
WALKER REWARD -0.114150796091 0.00500031938954
WALKER REWARD -1.12290235527 0.00353117472414
WALKER REWARD -0.174284686267 0.00292828321774
WALKER REWARD -2.82416212557 0.00246844048656
WALKER REWARD -1.78211423986 0.00308213585685
WALKER REWARD -0.181363059536 0.00275328447942
WALKER REWARD -1.00613522046 0.00353784658408
WALKER REWARD -0.466331562591 0.00280941097622
WALKER REWARD -2.89035855254 0.00518700043491
WALKER REWARD 0.0444120158547 0.00397691469244
WALKER REWARD 0.060050966701 0.00161058345773
WALKER REWARD -0.863874350227 0.00328282420279
WALKER REWARD -0.433738807914 0.00304136248745
WALKER REWARD -2.60143571753 0.00528476614385
WALKER REWARD 0.0671975808117 0.00256471234064
WALKER REWARD -0.095946457088 0.00381505231045
WALKER REWARD -0.349761078457 0.00369007489384
WALKER REWARD -0.251296872034 0.00265532502926
WALKER REWARD -2.05224092744 0.00341616899534
WALKER REWARD -0.0253756939726 0.000759476584009
WALKER REWARD -0.360203657884 0.00508144739193
WALKER REWARD -0.321214054066 0.00368500434698
WALKER REWARD -0.42701643223 0.00443938018878
WALKER REWARD -0.196406006408 0.00302370957538
WALKER REWARD -0.526399264229 0.00367924631945
WALKER REWARD -0.172312394111 0.00278451360305
WALKER REWARD -0.585437377086 0.00409687468365
WALKER REWARD -0.746573300348 0.00257054625982
WALKER REWARD -0.36508200541 0.00216167354731
WALKER REWARD -0.76918736614 0.00071401231535
WALKER REWARD 0.0559271195715 0.00424509037579
WALKER REWARD -0.711191248118 0.00319689806032
WALKER REWARD -0.957649267652 0.00265467015319
WALKER REWARD -0.121831152582 0.00265861034003
WALKER REWARD -0.786299326341 0.00410819119919
WALKER REWARD -0.0835074794887 0.00407344513701
WALKER REWARD -0.807134180334 0.00192822797363
WALKER REWARD -1.24002057414 0.00231718933305
WALKER REWARD 0.0269867506823 0.00352373950717
WALKER REWARD -0.807545945259 0.0038275923573
WALKER REWARD -0.193623786143 0.00249574012971
WALKER REWARD -0.753523962949 0.00374155958014
WALKER REWARD -1.78596221014 0.0043496936593
WALKER REWARD -0.0648593253145 0.00233118456052
WALKER REWARD -0.794976934516 0.00415497962707
WALKER REWARD -0.408364343692 0.00356392735493
WALKER REWARD -0.675683225067 0.00463408659011
WALKER REWARD -2.09167004026 0.00348407230784
WALKER REWARD -0.197438720418 0.00197000007541
WALKER REWARD -0.925102655412 0.00224687607646
WALKER REWARD -0.925236683599 0.00372264949208
WALKER REWARD -0.0850897493888 0.00132732366277
WALKER REWARD -2.0932942998 0.00290434056495
WALKER REWARD -0.470442525652 0.00351177819417
WALKER REWARD -0.982028820482 0.00220295381935
WALKER REWARD -1.70941598116 0.00325128503419
WALKER REWARD -0.300933806239 0.00319604152423
WALKER REWARD -1.9358312277 0.00263322095017
WALKER REWARD -0.233906126883 0.00304310265408
WALKER REWARD -1.18220516789 0.0024095766346
WALKER REWARD -1.69717955454 0.00281613422248
WALKER REWARD -0.33434566294 0.00260166838666
WALKER REWARD -1.72114242294 0.00510897982412
WALKER REWARD 0.179309650926 0.0037736574812
WALKER REWARD -1.20165334006 0.00152700330526
WALKER REWARD -1.34260542601 0.00209775271901
WALKER REWARD -0.521025869926 0.00319384306553
WALKER REWARD -1.466662008 0.00306260544121
WALKER REWARD -0.177022318185 0.00340304871368
WALKER REWARD -1.26385307666 0.00237578182629
WALKER REWARD -1.32754452915 0.00160652651743
WALKER REWARD -0.927347086207 0.000883984262531
WALKER REWARD -0.91011718291 0.00343509253897
WALKER REWARD -0.49203757013 0.00144183283336
WALKER REWARD -1.50787953554 0.00352403196432
WALKER REWARD -1.1209409517 0.00381196600684
WALKER REWARD -1.28548303023 0.00508393818643
WALKER REWARD -0.204899464593 0.00257858627126
WALKER REWARD -0.4927787761 0.00414649586304
WALKER REWARD -1.3407838667 0.00417097201244
WALKER REWARD -1.17198866264 0.00352603779857
WALKER REWARD -1.52369417287 0.00277584315128
WALKER REWARD 0.00624860491281 0.00313805969989
WALKER REWARD -0.510848354076 0.00504425996156
WALKER REWARD -0.987638656164 0.00430251695738
WALKER REWARD -1.51938367148 0.00247172421594
WALKER REWARD -1.30802624162 0.00285253055695
WALKER REWARD 0.589738937873 0.00313105130246
WALKER REWARD -0.657297182634 0.00151826843948
WALKER REWARD -0.48402549626 0.00319539135726
WALKER REWARD -1.23433309816 0.0036702885226
WALKER REWARD -1.35839111826 0.00367891411114
WALKER REWARD 0.25564694528 0.00342673944643
WALKER REWARD -0.921586062931 0.00380552275929
WALKER REWARD -0.303610370015 0.00310323890766
WALKER REWARD -0.736560912068 0.00310684115548
WALKER REWARD -1.74924786543 0.00322248522162
WALKER REWARD -0.181538915818 0.00250335209842
WALKER REWARD -0.65924174574 0.00328540539875
WALKER REWARD -0.784511453961 0.00217464814067
WALKER REWARD -0.599996723409 0.00476867933117
WALKER REWARD -1.78972433126 0.00400840200187
WALKER REWARD -0.508952848469 0.00276951023348
WALKER REWARD -0.244345518151 0.00351364747836
WALKER REWARD -1.23784914046 0.00356154838513
WALKER REWARD -0.375435826966 0.00245228373294
WALKER REWARD -0.824482680393 0.00276956184023
WALKER REWARD -0.843926630051 0.00153603524552
WALKER REWARD -0.133433142288 0.00258420454876
WALKER REWARD -1.41321935645 0.00475865249995
WALKER REWARD -0.841978953436 0.00406108383743
WALKER REWARD -0.24125427455 0.00369400126769
WALKER REWARD -1.03661008014 0.00335217466792
WALKER REWARD -0.288135084427 0.00431744015993
WALKER REWARD -1.3342948846 0.0030777770525
WALKER REWARD -0.710355067504 0.00428750462653
WALKER REWARD -0.89142339949 0.00313566847498
WALKER REWARD -0.993294748514 0.00234142518286
WALKER REWARD -0.646806877161 0.0031156635982
WALKER REWARD -1.21817504438 0.00485888234676
WALKER REWARD -0.44489291479 0.00359204780805
WALKER REWARD -1.32801681801 0.00197316184467
WALKER REWARD -0.76365523071 0.00459273020528
WALKER REWARD -0.881432665223 0.00492550770771
WALKER REWARD -1.31605932145 0.00186894072794
WALKER REWARD -0.592852836919 0.00427980129962
WALKER REWARD -1.64071386405 0.0032038492222
WALKER REWARD -0.611102572201 0.00437174853874
WALKER REWARD -1.33419475451 0.00300271937071
WALKER REWARD -1.62373504732 0.00347962035476
WALKER REWARD -0.762904478879 0.00270553340108
WALKER REWARD -1.75157516898 0.00347226770867
WALKER REWARD -0.427806392943 0.00466292984045
WALKER REWARD -1.82265916703 0.00336756441822
WALKER REWARD -1.96847227259 0.00419450448025
WALKER REWARD -0.521781510934 0.000882061072327
WALKER REWARD -1.58031896195 0.00510344370259
WALKER REWARD -0.625691501342 0.00233371164175
WALKER REWARD -2.14239026839 0.00118109531038
WALKER REWARD -2.06071158835 0.00395463402721
WALKER REWARD -0.244780824668 0.00399800653305
WALKER REWARD -1.25123082485 0.00393221973404
WALKER REWARD -1.04871065695 0.00431788550749
WALKER REWARD -2.30977669202 0.0027082288043
WALKER REWARD -0.239655912131 0.00466896947187
WALKER REWARD -0.156064372929 0.00381287189149
WALKER REWARD -0.906563182753 0.00214259612141
WALKER REWARD -0.94279624975 0.000954721371745
WALKER REWARD -2.68943066519 0.00472419572585
WALKER REWARD -0.841171861419 0.00273687388916
WALKER REWARD 0.0627696620353 0.0032402888898
WALKER REWARD -0.729580043256 0.00335515342068
WALKER REWARD -0.804486078892 0.00206790800068
WALKER REWARD -2.68856381243 0.00338595910208
WALKER REWARD -1.42196657281 0.00439784846078
WALKER REWARD -0.159266388295 0.00282222482561
WALKER REWARD -0.88851370467 0.00345772480735
WALKER REWARD -0.743364417825 0.00280227148104
WALKER REWARD -1.92588350416 0.0029870672714
WALKER REWARD -1.8075768851 0.00367013418985
WALKER REWARD -0.464292981835 0.00257839990602
WALKER REWARD -0.879936764558 0.00113354263207
WALKER REWARD -0.542190593634 0.00577668300963
WALKER REWARD -0.264001439494 0.00477181850879
WALKER REWARD -1.99594098926 0.00300132069357
WALKER REWARD -0.250862297204 0.00268934626622
WALKER REWARD -0.702540739408 0.0047062298877
WALKER REWARD -0.534870267636 0.00441924813989
WALKER REWARD -0.876013253382 0.00377745811849
WALKER REWARD -1.58769334845 0.00240851029059
WALKER REWARD -0.301301180477 0.00318386625138
WALKER REWARD -0.622875179717 0.00387510506
WALKER REWARD -0.732934714573 0.00188324948074
WALKER REWARD -0.920372724671 0.00410436378526
WALKER REWARD -1.00952915012 0.00361482071103
WALKER REWARD -0.98909721177 0.00207720513555
WALKER REWARD -0.750522874386 0.00337568963712
WALKER REWARD -0.793593823204 0.00500161731355
WALKER REWARD -0.82279488792 0.00369739259215
WALKER REWARD -0.974280065302 0.00406493065491
WALKER REWARD -1.44666303771 0.00182332661811
WALKER REWARD -0.998067937636 0.00302188372075
WALKER REWARD -0.901108513233 0.00398931391992
WALKER REWARD -1.10482047491 0.00205332272302
WALKER REWARD -1.06691095689 0.00287682114616
WALKER REWARD -1.70519260363 0.00322740144949
WALKER REWARD -1.17230589068 0.00342112906494
WALKER REWARD -0.650033968877 0.00398829863875
WALKER REWARD -0.862446363722 0.00375878292677
WALKER REWARD -1.14672049352 0.00376960738714
WALKER REWARD -2.13351859114 0.00385563211145
WALKER REWARD -1.06096370846 0.00362622956445
WALKER REWARD -0.125355848519 0.00325297536904
WALKER REWARD -0.471260565212 0.0041065859142
WALKER REWARD -1.14743810141 0.00277158514651
WALKER REWARD -2.38957777545 0.00379972460712
WALKER REWARD -1.08870169828 0.00330668692229
WALKER REWARD 0.164640704243 0.00279273882633
WALKER REWARD -0.440679511656 0.00344681077032
WALKER REWARD -1.20508135387 0.00341053723841
WALKER REWARD -2.201841274 0.00342753711801
WALKER REWARD -1.07482944755 0.00122434384319
WALKER REWARD 0.509232042304 0.00403939365081
WALKER REWARD -0.496205137397 0.00307289268418
WALKER REWARD -1.14096180273 0.00519888561616
WALKER REWARD -0.190556905147 0.00326145334079
WALKER REWARD -1.02005006994 0.0048415097841
WALKER REWARD 0.814924943111 0.00301492017961
WALKER REWARD -0.475538257743 0.00343347339719
WALKER REWARD -0.829341377753 0.00353731068364
WALKER REWARD -0.596043674276 0.00127059821805
WALKER REWARD -0.870003451583 0.00251774598521
WALKER REWARD 1.0149773064 0.00324976361239

Conclusion: control cost is negligible.