search

pytorch / rl

A modular, primitive-first, python-first PyTorch library for Reinforcement Learning.
https://pytorch.org/rl
MIT License
2.24k stars 296 forks source link

[Feature] ActionDiscretizer #2247

Closed vmoens closed 3 months ago

vmoens commented 3 months ago

A transform to discretize continuous actions

pytorch-bot[bot] commented 3 months ago

:link: Helpful Links

:test_tube: See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2247

Note: Links to docs will display an error until the docs builds have been completed.

:x: 3 New Failures, 2 Unrelated Failures

As of commit 7d1f59a89bf549c8c265d8bb9fb8ff64ee7d3e47 with merge base 559b729c30a20aeea67a9ed6981a31e2512936f6 (image):

NEW FAILURES - The following jobs have failed:

* [Habitat Tests on Linux / tests (3.9, 12.1) / linux-job](https://hud.pytorch.org/pr/pytorch/rl/2247#26701387442) ([gh](https://github.com/pytorch/rl/actions/runs/9678184939/job/26701387442)) `RuntimeError: Command docker exec -t f4d8c8b7178988ddb84c883ffc038f4c30c8e8ccf8b66cb3463a3bba45b2aeb4 /exec failed with exit code 139` * [Unit-tests on Linux / tests-optdeps (3.10, 12.1) / linux-job](https://hud.pytorch.org/pr/pytorch/rl/2247#26701391310) ([gh](https://github.com/pytorch/rl/actions/runs/9678184931/job/26701391310)) `RuntimeError: Command docker exec -t 2136a7755baf241bee60c1437494ef049b58fe013a497892fc767193d1206406 /exec failed with exit code 1` * [Unit-tests on Windows / unittests-cpu / windows-job](https://hud.pytorch.org/pr/pytorch/rl/2247#26701391986) ([gh](https://github.com/pytorch/rl/actions/runs/9678184937/job/26701391986)) `The process 'C:\Program Files\Git\cmd\git.exe' failed with exit code 128`

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

* [Libs Tests on Linux / unittests-gym (3.9, 12.1) / linux-job](https://hud.pytorch.org/pr/pytorch/rl/2247#26701393583) ([gh](https://github.com/pytorch/rl/actions/runs/9678184936/job/26701393583)) ([trunk failure](https://hud.pytorch.org/pytorch/rl/commit/559b729c30a20aeea67a9ed6981a31e2512936f6#26692655578)) `AttributeError: module 'torch' has no attribute 'uint16'` * [Unit-tests on Linux / tests-olddeps (3.8, 11.6) / linux-job](https://hud.pytorch.org/pr/pytorch/rl/2247#26701390939) ([gh](https://github.com/pytorch/rl/actions/runs/9678184931/job/26701390939)) ([trunk failure](https://hud.pytorch.org/pytorch/rl/commit/559b729c30a20aeea67a9ed6981a31e2512936f6#26692654497)) `AttributeError: module 'torch' has no attribute 'uint16'`

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions[bot] commented 3 months ago

$\color{#D29922}\textsf{\Large\⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 91. Improved: $\large\color{#35bf28}5$. Worsened: $\large\color{#d91a1a}3$.

Expand to view detailed results | Name | Max | Mean | Ops | Ops on Repo `HEAD` | Change | | ----------------------------------------------------------------------------------------- | --------- | --------- | --------------- | ------------------ | ----------------------------------- | | test_single | 0.1266s | 61.5613ms | 16.2440 Ops/s | 18.1520 Ops/s | $\textbf{\color{#d91a1a}-10.51\\%}$ | | test_sync | 33.0505ms | 31.6148ms | 31.6308 Ops/s | 32.6234 Ops/s | $\color{#d91a1a}-3.04\\%$ | | test_async | 50.8451ms | 28.7078ms | 34.8338 Ops/s | 32.4775 Ops/s | $\textbf{\color{#35bf28}+7.25\\%}$ | | test_simple | 0.4665s | 0.4004s | 2.4978 Ops/s | 2.6644 Ops/s | $\textbf{\color{#d91a1a}-6.25\\%}$ | | test_transformed | 0.5499s | 0.5480s | 1.8247 Ops/s | 1.8962 Ops/s | $\color{#d91a1a}-3.77\\%$ | | test_serial | 1.3825s | 1.3043s | 0.7667 Ops/s | 0.7889 Ops/s | $\color{#d91a1a}-2.82\\%$ | | test_parallel | 1.1664s | 1.0854s | 0.9213 Ops/s | 0.9084 Ops/s | $\color{#35bf28}+1.43\\%$ | | test_step_mdp_speed[True-True-True-True-True] | 0.1886ms | 22.9394μs | 43.5931 KOps/s | 44.7550 KOps/s | $\color{#d91a1a}-2.60\\%$ | | test_step_mdp_speed[True-True-True-True-False] | 55.8940μs | 13.3419μs | 74.9519 KOps/s | 76.1123 KOps/s | $\color{#d91a1a}-1.52\\%$ | | test_step_mdp_speed[True-True-True-False-True] | 36.2080μs | 13.3679μs | 74.8059 KOps/s | 76.1782 KOps/s | $\color{#d91a1a}-1.80\\%$ | | test_step_mdp_speed[True-True-True-False-False] | 39.1430μs | 7.8282μs | 127.7440 KOps/s | 130.9870 KOps/s | $\color{#d91a1a}-2.48\\%$ | | test_step_mdp_speed[True-True-False-True-True] | 58.4290μs | 24.3990μs | 40.9853 KOps/s | 42.3968 KOps/s | $\color{#d91a1a}-3.33\\%$ | | test_step_mdp_speed[True-True-False-True-False] | 44.4530μs | 14.5925μs | 68.5282 KOps/s | 70.0589 KOps/s | $\color{#d91a1a}-2.18\\%$ | | test_step_mdp_speed[True-True-False-False-True] | 50.1940μs | 14.7301μs | 67.8882 KOps/s | 70.2517 KOps/s | $\color{#d91a1a}-3.36\\%$ | | test_step_mdp_speed[True-True-False-False-False] | 0.1362ms | 9.1220μs | 109.6254 KOps/s | 113.4260 KOps/s | $\color{#d91a1a}-3.35\\%$ | | test_step_mdp_speed[True-False-True-True-True] | 0.1001ms | 25.8641μs | 38.6637 KOps/s | 40.1080 KOps/s | $\color{#d91a1a}-3.60\\%$ | | test_step_mdp_speed[True-False-True-True-False] | 43.1810μs | 16.0250μs | 62.4026 KOps/s | 63.9817 KOps/s | $\color{#d91a1a}-2.47\\%$ | | test_step_mdp_speed[True-False-True-False-True] | 41.2470μs | 14.6265μs | 68.3690 KOps/s | 69.4321 KOps/s | $\color{#d91a1a}-1.53\\%$ | | test_step_mdp_speed[True-False-True-False-False] | 45.5360μs | 9.1466μs | 109.3308 KOps/s | 112.1924 KOps/s | $\color{#d91a1a}-2.55\\%$ | | test_step_mdp_speed[True-False-False-True-True] | 71.3730μs | 26.8687μs | 37.2180 KOps/s | 38.0611 KOps/s | $\color{#d91a1a}-2.22\\%$ | | test_step_mdp_speed[True-False-False-True-False] | 50.7660μs | 17.2930μs | 57.8267 KOps/s | 59.0806 KOps/s | $\color{#d91a1a}-2.12\\%$ | | test_step_mdp_speed[True-False-False-False-True] | 60.0520μs | 15.7952μs | 63.3103 KOps/s | 63.7552 KOps/s | $\color{#d91a1a}-0.70\\%$ | | test_step_mdp_speed[True-False-False-False-False] | 52.2870μs | 10.4120μs | 96.0427 KOps/s | 98.4872 KOps/s | $\color{#d91a1a}-2.48\\%$ | | test_step_mdp_speed[False-True-True-True-True] | 63.7590μs | 25.7057μs | 38.9018 KOps/s | 40.0203 KOps/s | $\color{#d91a1a}-2.79\\%$ | | test_step_mdp_speed[False-True-True-True-False] | 51.4760μs | 15.9731μs | 62.6054 KOps/s | 64.3892 KOps/s | $\color{#d91a1a}-2.77\\%$ | | test_step_mdp_speed[False-True-True-False-True] | 66.8750μs | 17.1457μs | 58.3236 KOps/s | 59.5527 KOps/s | $\color{#d91a1a}-2.06\\%$ | | test_step_mdp_speed[False-True-True-False-False] | 51.8070μs | 10.3492μs | 96.6255 KOps/s | 98.3898 KOps/s | $\color{#d91a1a}-1.79\\%$ | | test_step_mdp_speed[False-True-False-True-True] | 57.2670μs | 27.1422μs | 36.8431 KOps/s | 37.9438 KOps/s | $\color{#d91a1a}-2.90\\%$ | | test_step_mdp_speed[False-True-False-True-False] | 54.4010μs | 17.2947μs | 57.8211 KOps/s | 60.2533 KOps/s | $\color{#d91a1a}-4.04\\%$ | | test_step_mdp_speed[False-True-False-False-True] | 44.1030μs | 18.6385μs | 53.6523 KOps/s | 55.4921 KOps/s | $\color{#d91a1a}-3.32\\%$ | | test_step_mdp_speed[False-True-False-False-False] | 43.8120μs | 11.6180μs | 86.0732 KOps/s | 88.4028 KOps/s | $\color{#d91a1a}-2.64\\%$ | | test_step_mdp_speed[False-False-True-True-True] | 68.3080μs | 28.9064μs | 34.5944 KOps/s | 36.0731 KOps/s | $\color{#d91a1a}-4.10\\%$ | | test_step_mdp_speed[False-False-True-True-False] | 54.4430μs | 18.8058μs | 53.1750 KOps/s | 55.5079 KOps/s | $\color{#d91a1a}-4.20\\%$ | | test_step_mdp_speed[False-False-True-False-True] | 54.4220μs | 18.4403μs | 54.2289 KOps/s | 55.8974 KOps/s | $\color{#d91a1a}-2.98\\%$ | | test_step_mdp_speed[False-False-True-False-False] | 33.8730μs | 11.6314μs | 85.9739 KOps/s | 89.4122 KOps/s | $\color{#d91a1a}-3.85\\%$ | | test_step_mdp_speed[False-False-False-True-True] | 62.3860μs | 30.1412μs | 33.1772 KOps/s | 33.7774 KOps/s | $\color{#d91a1a}-1.78\\%$ | | test_step_mdp_speed[False-False-False-True-False] | 51.5460μs | 19.9137μs | 50.2166 KOps/s | 52.3194 KOps/s | $\color{#d91a1a}-4.02\\%$ | | test_step_mdp_speed[False-False-False-False-True] | 61.8350μs | 19.4341μs | 51.4559 KOps/s | 52.2595 KOps/s | $\color{#d91a1a}-1.54\\%$ | | test_step_mdp_speed[False-False-False-False-False] | 54.2810μs | 12.7564μs | 78.3921 KOps/s | 80.2525 KOps/s | $\color{#d91a1a}-2.32\\%$ | | test_values[generalized_advantage_estimate-True-True] | 10.5926ms | 9.8855ms | 101.1584 Ops/s | 98.5011 Ops/s | $\color{#35bf28}+2.70\\%$ | | test_values[vec_generalized_advantage_estimate-True-True] | 38.8379ms | 35.4999ms | 28.1691 Ops/s | 27.9497 Ops/s | $\color{#35bf28}+0.78\\%$ | | test_values[td0_return_estimate-False-False] | 0.2170ms | 0.1725ms | 5.7958 KOps/s | 6.0907 KOps/s | $\color{#d91a1a}-4.84\\%$ | | test_values[td1_return_estimate-False-False] | 26.8993ms | 24.4183ms | 40.9530 Ops/s | 39.0308 Ops/s | $\color{#35bf28}+4.92\\%$ | | test_values[vec_td1_return_estimate-False-False] | 36.7626ms | 35.5567ms | 28.1241 Ops/s | 28.1282 Ops/s | $\color{#d91a1a}-0.01\\%$ | | test_values[td_lambda_return_estimate-True-False] | 38.0457ms | 35.2097ms | 28.4013 Ops/s | 27.2288 Ops/s | $\color{#35bf28}+4.31\\%$ | | test_values[vec_td_lambda_return_estimate-True-False] | 37.2839ms | 35.6552ms | 28.0464 Ops/s | 28.0422 Ops/s | $\color{#35bf28}+0.02\\%$ | | test_gae_speed[generalized_advantage_estimate-False-1-512] | 8.7065ms | 8.5605ms | 116.8162 Ops/s | 113.7573 Ops/s | $\color{#35bf28}+2.69\\%$ | | test_gae_speed[vec_generalized_advantage_estimate-True-1-512] | 2.9502ms | 1.9527ms | 512.1235 Ops/s | 526.2565 Ops/s | $\color{#d91a1a}-2.69\\%$ | | test_gae_speed[vec_generalized_advantage_estimate-False-1-512] | 0.5803ms | 0.3586ms | 2.7883 KOps/s | 2.7476 KOps/s | $\color{#35bf28}+1.48\\%$ | | test_gae_speed[vec_generalized_advantage_estimate-True-32-512] | 57.9110ms | 47.6184ms | 21.0003 Ops/s | 23.7257 Ops/s | $\textbf{\color{#d91a1a}-11.49\\%}$ | | test_gae_speed[vec_generalized_advantage_estimate-False-32-512] | 3.6812ms | 3.0364ms | 329.3335 Ops/s | 327.8175 Ops/s | $\color{#35bf28}+0.46\\%$ | | test_dqn_speed | 1.8166ms | 1.3779ms | 725.7167 Ops/s | 747.7392 Ops/s | $\color{#d91a1a}-2.95\\%$ | | test_ddpg_speed | 3.6153ms | 2.9127ms | 343.3204 Ops/s | 353.3411 Ops/s | $\color{#d91a1a}-2.84\\%$ | | test_sac_speed | 10.3630ms | 8.6838ms | 115.1566 Ops/s | 115.7527 Ops/s | $\color{#d91a1a}-0.51\\%$ | | test_redq_speed | 15.4812ms | 13.6360ms | 73.3354 Ops/s | 72.3165 Ops/s | $\color{#35bf28}+1.41\\%$ | | test_redq_deprec_speed | 14.4547ms | 13.3983ms | 74.6365 Ops/s | 69.1275 Ops/s | $\textbf{\color{#35bf28}+7.97\\%}$ | | test_td3_speed | 17.2934ms | 8.5938ms | 116.3631 Ops/s | 115.3169 Ops/s | $\color{#35bf28}+0.91\\%$ | | test_cql_speed | 39.4249ms | 37.8786ms | 26.4001 Ops/s | 27.0482 Ops/s | $\color{#d91a1a}-2.40\\%$ | | test_a2c_speed | 8.4272ms | 7.8171ms | 127.9244 Ops/s | 130.6356 Ops/s | $\color{#d91a1a}-2.08\\%$ | | test_ppo_speed | 9.5193ms | 8.1085ms | 123.3279 Ops/s | 126.0400 Ops/s | $\color{#d91a1a}-2.15\\%$ | | test_reinforce_speed | 8.9045ms | 7.0286ms | 142.2753 Ops/s | 144.5820 Ops/s | $\color{#d91a1a}-1.60\\%$ | | test_iql_speed | 34.9621ms | 33.5940ms | 29.7672 Ops/s | 29.7233 Ops/s | $\color{#35bf28}+0.15\\%$ | | test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] | 5.1733ms | 3.5074ms | 285.1100 Ops/s | 271.4297 Ops/s | $\textbf{\color{#35bf28}+5.04\\%}$ | | test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] | 0.7498ms | 0.4945ms | 2.0224 KOps/s | 2.0059 KOps/s | $\color{#35bf28}+0.82\\%$ | | test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] | 3.4665ms | 0.4775ms | 2.0942 KOps/s | 2.1174 KOps/s | $\color{#d91a1a}-1.10\\%$ | | test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] | 3.8897ms | 3.5218ms | 283.9444 Ops/s | 274.6002 Ops/s | $\color{#35bf28}+3.40\\%$ | | test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] | 0.5924ms | 0.4821ms | 2.0744 KOps/s | 1.9846 KOps/s | $\color{#35bf28}+4.52\\%$ | | test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] | 3.5906ms | 0.4634ms | 2.1578 KOps/s | 2.0839 KOps/s | $\color{#35bf28}+3.55\\%$ | | test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] | 2.2678ms | 1.7205ms | 581.2312 Ops/s | 568.4210 Ops/s | $\color{#35bf28}+2.25\\%$ | | test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] | 2.6190ms | 1.6448ms | 607.9807 Ops/s | 598.3857 Ops/s | $\color{#35bf28}+1.60\\%$ | | test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] | 5.4386ms | 3.6799ms | 271.7476 Ops/s | 255.9807 Ops/s | $\textbf{\color{#35bf28}+6.16\\%}$ | | test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] | 1.0200ms | 0.6336ms | 1.5782 KOps/s | 1.3411 KOps/s | $\textbf{\color{#35bf28}+17.68\\%}$ | | test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] | 3.7930ms | 0.6088ms | 1.6426 KOps/s | 1.6381 KOps/s | $\color{#35bf28}+0.27\\%$ | | test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] | 4.3938ms | 3.5486ms | 281.8046 Ops/s | 271.6443 Ops/s | $\color{#35bf28}+3.74\\%$ | | test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] | 0.7335ms | 0.4951ms | 2.0198 KOps/s | 1.9961 KOps/s | $\color{#35bf28}+1.19\\%$ | | test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] | 3.5638ms | 0.4739ms | 2.1101 KOps/s | 2.0811 KOps/s | $\color{#35bf28}+1.39\\%$ | | test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] | 5.5892ms | 3.5333ms | 283.0244 Ops/s | 276.5984 Ops/s | $\color{#35bf28}+2.32\\%$ | | test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] | 1.3302ms | 0.4966ms | 2.0136 KOps/s | 2.0451 KOps/s | $\color{#d91a1a}-1.54\\%$ | | test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] | 0.7464ms | 0.4820ms | 2.0748 KOps/s | 2.0891 KOps/s | $\color{#d91a1a}-0.68\\%$ | | test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] | 5.6607ms | 3.7029ms | 270.0577 Ops/s | 262.6546 Ops/s | $\color{#35bf28}+2.82\\%$ | | test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] | 1.0700ms | 0.6320ms | 1.5824 KOps/s | 1.5224 KOps/s | $\color{#35bf28}+3.94\\%$ | | test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] | 3.7117ms | 0.6119ms | 1.6343 KOps/s | 1.6424 KOps/s | $\color{#d91a1a}-0.49\\%$ | | test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] | 0.1238s | 6.2220ms | 160.7192 Ops/s | 157.0405 Ops/s | $\color{#35bf28}+2.34\\%$ | | test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] | 0.1175s | 14.7284ms | 67.8962 Ops/s | 65.6634 Ops/s | $\color{#35bf28}+3.40\\%$ | | test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] | 1.7369ms | 1.0657ms | 938.3737 Ops/s | 943.3332 Ops/s | $\color{#d91a1a}-0.53\\%$ | | test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] | 0.1121s | 6.0360ms | 165.6731 Ops/s | 169.3542 Ops/s | $\color{#d91a1a}-2.17\\%$ | | test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] | 15.3019ms | 12.6191ms | 79.2451 Ops/s | 78.1249 Ops/s | $\color{#35bf28}+1.43\\%$ | | test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] | 1.7682ms | 1.0762ms | 929.2217 Ops/s | 953.5917 Ops/s | $\color{#d91a1a}-2.56\\%$ | | test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] | 0.1096s | 6.1517ms | 162.5555 Ops/s | 163.8100 Ops/s | $\color{#d91a1a}-0.77\\%$ | | test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] | 16.1603ms | 12.8612ms | 77.7531 Ops/s | 76.8876 Ops/s | $\color{#35bf28}+1.13\\%$ | | test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] | 1.7787ms | 1.2272ms | 814.8538 Ops/s | 836.4138 Ops/s | $\color{#d91a1a}-2.58\\%$ |
github-actions[bot] commented 3 months ago

$\color{#D29922}\textsf{\Large\⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 94. Improved: $\large\color{#35bf28}5$. Worsened: $\large\color{#d91a1a}7$.

Expand to view detailed results | Name | Max | Mean | Ops | Ops on Repo `HEAD` | Change | | ----------------------------------------------------------------------------------------- | --------- | --------- | -------------- | ------------------ | ----------------------------------- | | test_single | 0.1758s | 0.1240s | 8.0677 Ops/s | 8.4137 Ops/s | $\color{#d91a1a}-4.11\\%$ | | test_sync | 0.1040s | 0.1015s | 9.8534 Ops/s | 10.0329 Ops/s | $\color{#d91a1a}-1.79\\%$ | | test_async | 0.1974s | 98.1861ms | 10.1847 Ops/s | 10.7519 Ops/s | $\textbf{\color{#d91a1a}-5.28\\%}$ | | test_single_pixels | 0.1280s | 0.1276s | 7.8353 Ops/s | 7.7460 Ops/s | $\color{#35bf28}+1.15\\%$ | | test_sync_pixels | 85.4459ms | 82.6895ms | 12.0934 Ops/s | 12.2055 Ops/s | $\color{#d91a1a}-0.92\\%$ | | test_async_pixels | 0.1657s | 69.9887ms | 14.2880 Ops/s | 14.2790 Ops/s | $\color{#35bf28}+0.06\\%$ | | test_simple | 0.8799s | 0.8141s | 1.2284 Ops/s | 1.2138 Ops/s | $\color{#35bf28}+1.20\\%$ | | test_transformed | 1.1365s | 1.0714s | 0.9334 Ops/s | 0.9274 Ops/s | $\color{#35bf28}+0.64\\%$ | | test_serial | 2.5323s | 2.4706s | 0.4048 Ops/s | 0.3945 Ops/s | $\color{#35bf28}+2.60\\%$ | | test_parallel | 2.4252s | 2.3765s | 0.4208 Ops/s | 0.4210 Ops/s | $\color{#d91a1a}-0.04\\%$ | | test_step_mdp_speed[True-True-True-True-True] | 82.1510μs | 33.7607μs | 29.6202 KOps/s | 29.0803 KOps/s | $\color{#35bf28}+1.86\\%$ | | test_step_mdp_speed[True-True-True-True-False] | 43.1500μs | 19.3216μs | 51.7555 KOps/s | 49.9619 KOps/s | $\color{#35bf28}+3.59\\%$ | | test_step_mdp_speed[True-True-True-False-True] | 51.8110μs | 19.1402μs | 52.2461 KOps/s | 51.1312 KOps/s | $\color{#35bf28}+2.18\\%$ | | test_step_mdp_speed[True-True-True-False-False] | 31.9400μs | 11.0007μs | 90.9033 KOps/s | 87.4927 KOps/s | $\color{#35bf28}+3.90\\%$ | | test_step_mdp_speed[True-True-False-True-True] | 76.3510μs | 35.6943μs | 28.0157 KOps/s | 27.7596 KOps/s | $\color{#35bf28}+0.92\\%$ | | test_step_mdp_speed[True-True-False-True-False] | 49.1400μs | 21.4137μs | 46.6991 KOps/s | 46.2948 KOps/s | $\color{#35bf28}+0.87\\%$ | | test_step_mdp_speed[True-True-False-False-True] | 43.1410μs | 20.6093μs | 48.5218 KOps/s | 46.9056 KOps/s | $\color{#35bf28}+3.45\\%$ | | test_step_mdp_speed[True-True-False-False-False] | 35.4200μs | 12.9347μs | 77.3115 KOps/s | 75.8126 KOps/s | $\color{#35bf28}+1.98\\%$ | | test_step_mdp_speed[True-False-True-True-True] | 69.8310μs | 37.4825μs | 26.6791 KOps/s | 26.2402 KOps/s | $\color{#35bf28}+1.67\\%$ | | test_step_mdp_speed[True-False-True-True-False] | 45.0410μs | 23.0117μs | 43.4562 KOps/s | 42.5360 KOps/s | $\color{#35bf28}+2.16\\%$ | | test_step_mdp_speed[True-False-True-False-True] | 44.1910μs | 20.5625μs | 48.6323 KOps/s | 47.1819 KOps/s | $\color{#35bf28}+3.07\\%$ | | test_step_mdp_speed[True-False-True-False-False] | 30.4710μs | 12.6717μs | 78.9160 KOps/s | 75.8660 KOps/s | $\color{#35bf28}+4.02\\%$ | | test_step_mdp_speed[True-False-False-True-True] | 72.2420μs | 38.4221μs | 26.0267 KOps/s | 24.8828 KOps/s | $\color{#35bf28}+4.60\\%$ | | test_step_mdp_speed[True-False-False-True-False] | 52.6110μs | 24.4690μs | 40.8681 KOps/s | 39.5797 KOps/s | $\color{#35bf28}+3.26\\%$ | | test_step_mdp_speed[True-False-False-False-True] | 40.9400μs | 22.4086μs | 44.6256 KOps/s | 43.8990 KOps/s | $\color{#35bf28}+1.66\\%$ | | test_step_mdp_speed[True-False-False-False-False] | 31.9610μs | 14.5058μs | 68.9377 KOps/s | 66.5261 KOps/s | $\color{#35bf28}+3.62\\%$ | | test_step_mdp_speed[False-True-True-True-True] | 73.0520μs | 36.5873μs | 27.3319 KOps/s | 26.1287 KOps/s | $\color{#35bf28}+4.61\\%$ | | test_step_mdp_speed[False-True-True-True-False] | 47.8310μs | 22.6848μs | 44.0823 KOps/s | 42.2950 KOps/s | $\color{#35bf28}+4.23\\%$ | | test_step_mdp_speed[False-True-True-False-True] | 47.2510μs | 24.0451μs | 41.5885 KOps/s | 38.9899 KOps/s | $\textbf{\color{#35bf28}+6.67\\%}$ | | test_step_mdp_speed[False-True-True-False-False] | 38.3700μs | 14.5006μs | 68.9626 KOps/s | 67.0282 KOps/s | $\color{#35bf28}+2.89\\%$ | | test_step_mdp_speed[False-True-False-True-True] | 62.1510μs | 38.3930μs | 26.0464 KOps/s | 25.1094 KOps/s | $\color{#35bf28}+3.73\\%$ | | test_step_mdp_speed[False-True-False-True-False] | 45.7010μs | 24.5306μs | 40.7655 KOps/s | 39.5725 KOps/s | $\color{#35bf28}+3.01\\%$ | | test_step_mdp_speed[False-True-False-False-True] | 46.6510μs | 25.7191μs | 38.8816 KOps/s | 36.6559 KOps/s | $\textbf{\color{#35bf28}+6.07\\%}$ | | test_step_mdp_speed[False-True-False-False-False] | 35.3100μs | 16.1091μs | 62.0767 KOps/s | 59.0599 KOps/s | $\textbf{\color{#35bf28}+5.11\\%}$ | | test_step_mdp_speed[False-False-True-True-True] | 0.1047ms | 40.2335μs | 24.8549 KOps/s | 23.8178 KOps/s | $\color{#35bf28}+4.35\\%$ | | test_step_mdp_speed[False-False-True-True-False] | 49.7510μs | 26.5165μs | 37.7124 KOps/s | 36.6622 KOps/s | $\color{#35bf28}+2.86\\%$ | | test_step_mdp_speed[False-False-True-False-True] | 44.4110μs | 25.8604μs | 38.6691 KOps/s | 36.7559 KOps/s | $\textbf{\color{#35bf28}+5.21\\%}$ | | test_step_mdp_speed[False-False-True-False-False] | 33.5410μs | 16.2721μs | 61.4547 KOps/s | 59.9336 KOps/s | $\color{#35bf28}+2.54\\%$ | | test_step_mdp_speed[False-False-False-True-True] | 57.4210μs | 43.6468μs | 22.9112 KOps/s | 22.4892 KOps/s | $\color{#35bf28}+1.88\\%$ | | test_step_mdp_speed[False-False-False-True-False] | 54.5310μs | 28.8318μs | 34.6840 KOps/s | 33.9260 KOps/s | $\color{#35bf28}+2.23\\%$ | | test_step_mdp_speed[False-False-False-False-True] | 47.4800μs | 27.9499μs | 35.7783 KOps/s | 34.3878 KOps/s | $\color{#35bf28}+4.04\\%$ | | test_step_mdp_speed[False-False-False-False-False] | 43.3100μs | 18.0696μs | 55.3415 KOps/s | 53.7972 KOps/s | $\color{#35bf28}+2.87\\%$ | | test_values[generalized_advantage_estimate-True-True] | 25.4239ms | 24.8336ms | 40.2680 Ops/s | 39.0341 Ops/s | $\color{#35bf28}+3.16\\%$ | | test_values[vec_generalized_advantage_estimate-True-True] | 98.1473ms | 2.8686ms | 348.6058 Ops/s | 364.6287 Ops/s | $\color{#d91a1a}-4.39\\%$ | | test_values[td0_return_estimate-False-False] | 88.5520μs | 64.9586μs | 15.3944 KOps/s | 14.7226 KOps/s | $\color{#35bf28}+4.56\\%$ | | test_values[td1_return_estimate-False-False] | 55.5598ms | 55.2061ms | 18.1140 Ops/s | 17.3897 Ops/s | $\color{#35bf28}+4.16\\%$ | | test_values[vec_td1_return_estimate-False-False] | 1.4085ms | 1.0905ms | 917.0038 Ops/s | 903.5277 Ops/s | $\color{#35bf28}+1.49\\%$ | | test_values[td_lambda_return_estimate-True-False] | 88.1784ms | 87.6707ms | 11.4063 Ops/s | 11.0202 Ops/s | $\color{#35bf28}+3.50\\%$ | | test_values[vec_td_lambda_return_estimate-True-False] | 1.2832ms | 1.0849ms | 921.7181 Ops/s | 908.7841 Ops/s | $\color{#35bf28}+1.42\\%$ | | test_gae_speed[generalized_advantage_estimate-False-1-512] | 25.8566ms | 25.1157ms | 39.8157 Ops/s | 38.4376 Ops/s | $\color{#35bf28}+3.59\\%$ | | test_gae_speed[vec_generalized_advantage_estimate-True-1-512] | 0.9407ms | 0.7236ms | 1.3820 KOps/s | 1.3413 KOps/s | $\color{#35bf28}+3.04\\%$ | | test_gae_speed[vec_generalized_advantage_estimate-False-1-512] | 0.7749ms | 0.6739ms | 1.4839 KOps/s | 1.4498 KOps/s | $\color{#35bf28}+2.36\\%$ | | test_gae_speed[vec_generalized_advantage_estimate-True-32-512] | 1.5725ms | 1.4733ms | 678.7479 Ops/s | 670.1310 Ops/s | $\color{#35bf28}+1.29\\%$ | | test_gae_speed[vec_generalized_advantage_estimate-False-32-512] | 0.7510ms | 0.6916ms | 1.4458 KOps/s | 1.4144 KOps/s | $\color{#35bf28}+2.22\\%$ | | test_dqn_speed | 1.8633ms | 1.4462ms | 691.4726 Ops/s | 687.7041 Ops/s | $\color{#35bf28}+0.55\\%$ | | test_ddpg_speed | 3.3261ms | 2.9575ms | 338.1189 Ops/s | 340.9231 Ops/s | $\color{#d91a1a}-0.82\\%$ | | test_sac_speed | 8.7033ms | 8.4601ms | 118.2018 Ops/s | 117.9184 Ops/s | $\color{#35bf28}+0.24\\%$ | | test_redq_speed | 11.9840ms | 10.7348ms | 93.1553 Ops/s | 93.7516 Ops/s | $\color{#d91a1a}-0.64\\%$ | | test_redq_deprec_speed | 0.1074s | 12.8048ms | 78.0956 Ops/s | 84.9666 Ops/s | $\textbf{\color{#d91a1a}-8.09\\%}$ | | test_td3_speed | 8.6319ms | 8.4592ms | 118.2148 Ops/s | 117.5651 Ops/s | $\color{#35bf28}+0.55\\%$ | | test_cql_speed | 27.5997ms | 26.0775ms | 38.3472 Ops/s | 39.1078 Ops/s | $\color{#d91a1a}-1.94\\%$ | | test_a2c_speed | 6.3574ms | 5.7441ms | 174.0926 Ops/s | 183.5519 Ops/s | $\textbf{\color{#d91a1a}-5.15\\%}$ | | test_ppo_speed | 6.2087ms | 6.0504ms | 165.2796 Ops/s | 172.4185 Ops/s | $\color{#d91a1a}-4.14\\%$ | | test_reinforce_speed | 4.8948ms | 4.7069ms | 212.4540 Ops/s | 225.3499 Ops/s | $\textbf{\color{#d91a1a}-5.72\\%}$ | | test_iql_speed | 20.2239ms | 19.6421ms | 50.9111 Ops/s | 52.3704 Ops/s | $\color{#d91a1a}-2.79\\%$ | | test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] | 4.7957ms | 4.6079ms | 217.0198 Ops/s | 213.7629 Ops/s | $\color{#35bf28}+1.52\\%$ | | test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] | 1.0897ms | 0.5929ms | 1.6865 KOps/s | 1.6859 KOps/s | $\color{#35bf28}+0.04\\%$ | | test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] | 0.7279ms | 0.5693ms | 1.7566 KOps/s | 1.7561 KOps/s | $\color{#35bf28}+0.03\\%$ | | test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] | 4.8833ms | 4.5900ms | 217.8665 Ops/s | 215.9378 Ops/s | $\color{#35bf28}+0.89\\%$ | | test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] | 0.6915ms | 0.5832ms | 1.7148 KOps/s | 1.6976 KOps/s | $\color{#35bf28}+1.01\\%$ | | test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] | 4.4577ms | 0.5686ms | 1.7587 KOps/s | 1.7737 KOps/s | $\color{#d91a1a}-0.84\\%$ | | test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] | 2.2282ms | 2.0983ms | 476.5795 Ops/s | 464.5635 Ops/s | $\color{#35bf28}+2.59\\%$ | | test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] | 5.7826ms | 2.0091ms | 497.7327 Ops/s | 489.2582 Ops/s | $\color{#35bf28}+1.73\\%$ | | test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] | 4.9380ms | 4.7237ms | 211.6974 Ops/s | 209.2221 Ops/s | $\color{#35bf28}+1.18\\%$ | | test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] | 0.8815ms | 0.7404ms | 1.3506 KOps/s | 1.3451 KOps/s | $\color{#35bf28}+0.41\\%$ | | test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] | 4.3652ms | 0.7210ms | 1.3869 KOps/s | 1.3916 KOps/s | $\color{#d91a1a}-0.33\\%$ | | test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] | 4.7686ms | 4.6201ms | 216.4458 Ops/s | 214.5844 Ops/s | $\color{#35bf28}+0.87\\%$ | | test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] | 1.3837ms | 0.5940ms | 1.6835 KOps/s | 1.6824 KOps/s | $\color{#35bf28}+0.06\\%$ | | test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] | 0.7850ms | 0.5683ms | 1.7597 KOps/s | 1.7318 KOps/s | $\color{#35bf28}+1.61\\%$ | | test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] | 4.8303ms | 4.5680ms | 218.9148 Ops/s | 216.3615 Ops/s | $\color{#35bf28}+1.18\\%$ | | test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] | 0.6840ms | 0.5878ms | 1.7013 KOps/s | 1.6890 KOps/s | $\color{#35bf28}+0.72\\%$ | | test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] | 4.5098ms | 0.5699ms | 1.7548 KOps/s | 1.7714 KOps/s | $\color{#d91a1a}-0.93\\%$ | | test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] | 4.9621ms | 4.7961ms | 208.5042 Ops/s | 206.5652 Ops/s | $\color{#35bf28}+0.94\\%$ | | test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] | 1.5008ms | 0.7449ms | 1.3425 KOps/s | 1.3447 KOps/s | $\color{#d91a1a}-0.16\\%$ | | test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] | 0.8852ms | 0.7121ms | 1.4043 KOps/s | 1.3749 KOps/s | $\color{#35bf28}+2.14\\%$ | | test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] | 0.1326s | 7.4774ms | 133.7355 Ops/s | 134.7580 Ops/s | $\color{#d91a1a}-0.76\\%$ | | test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] | 17.9905ms | 15.6521ms | 63.8891 Ops/s | 61.6285 Ops/s | $\color{#35bf28}+3.67\\%$ | | test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] | 1.3296ms | 1.2544ms | 797.1904 Ops/s | 771.8241 Ops/s | $\color{#35bf28}+3.29\\%$ | | test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] | 0.1238s | 7.3600ms | 135.8695 Ops/s | 100.1389 Ops/s | $\textbf{\color{#35bf28}+35.68\\%}$ | | test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] | 0.1353s | 18.0997ms | 55.2495 Ops/s | 61.3321 Ops/s | $\textbf{\color{#d91a1a}-9.92\\%}$ | | test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] | 7.5800ms | 1.3654ms | 732.3675 Ops/s | 775.5420 Ops/s | $\textbf{\color{#d91a1a}-5.57\\%}$ | | test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] | 0.1236s | 7.4911ms | 133.4922 Ops/s | 130.8427 Ops/s | $\color{#35bf28}+2.02\\%$ | | test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] | 18.2513ms | 15.8402ms | 63.1305 Ops/s | 60.6559 Ops/s | $\color{#35bf28}+4.08\\%$ | | test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] | 7.1780ms | 1.5630ms | 639.7798 Ops/s | 682.0215 Ops/s | $\textbf{\color{#d91a1a}-6.19\\%}$ |