search

pytorch / rl

A modular, primitive-first, python-first PyTorch library for Reinforcement Learning.
https://pytorch.org/rl
MIT License
2.01k stars 269 forks source link

[BugFix] Fix slice sampler end computation at the cursor place #2225

Closed vmoens closed 4 weeks ago

vmoens commented 4 weeks ago

Enables this code to run without error in all cases:

rb = ReplayBuffer(
storage=LazyTensorStorage(100, ndim=2), 
sampler=SliceSampler(strict_length=True, num_slices=1), batch_size=8, 
transform=Compose(
    UnsqueezeTransform(-2, in_keys=["step_count"], out_keys=["obs_cat0"]),
    CatFrames(N=4, dim=-2, in_keys=["obs_cat0"], done_key="done"),
    UnsqueezeTransform(-2, in_keys=["step_count"], out_keys=["obs_cat1"]),
    CatFrames(N=5, dim=-2, in_keys=["obs_cat1"], done_key="done")
)
)
env = SerialEnv(2, lambda: GymEnv("CartPole-v1", device=None).append_transform(StepCounter()))
r = env.rollout(120, break_when_any_done=False)
rb.extend(r)
print(rb._storage._last_cursor)
s, info = rb.sample(return_info=True)
assert (s["obs_cat0"].squeeze()[4:]!=0).all(), info
pytorch-bot[bot] commented 4 weeks ago

:link: Helpful Links

:test_tube: See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2225

Note: Links to docs will display an error until the docs builds have been completed.

:x: 2 New Failures, 2 Unrelated Failures

As of commit eaf9ef2e07e2f1efd5bbe5ad60d2d149d38fcac8 with merge base f613eefb0f44b6a9ccd79f82703bb309aa165009 (image):

NEW FAILURES - The following jobs have failed:

* [Habitat Tests on Linux / tests (3.9, 12.1) / linux-job](https://hud.pytorch.org/pr/pytorch/rl/2225#26116939244) ([gh](https://github.com/pytorch/rl/actions/runs/9479108948/job/26116939244)) `RuntimeError: Command docker exec -t 73c3cc79a31e98d22d3221e48e992112663e7f617e4ad277dfd47fb403483370 /exec failed with exit code 139` * [Unit-tests on Linux / tests-optdeps (3.10, 12.1) / linux-job](https://hud.pytorch.org/pr/pytorch/rl/2225#26116944715) ([gh](https://github.com/pytorch/rl/actions/runs/9479108961/job/26116944715)) `RuntimeError: Command docker exec -t 96a13c9c5168a380717beb5af9d6dcd6dad22a73357729ef09a4149b880cf236 /exec failed with exit code 1`

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

* [Libs Tests on Linux / unittests-gym (3.9, 12.1) / linux-job](https://hud.pytorch.org/pr/pytorch/rl/2225#26116952191) ([gh](https://github.com/pytorch/rl/actions/runs/9479108958/job/26116952191)) ([trunk failure](https://hud.pytorch.org/pytorch/rl/commit/f613eefb0f44b6a9ccd79f82703bb309aa165009#26116899246)) `##[error]The operation was canceled.` * [Unit-tests on Linux / tests-olddeps (3.8, 11.6) / linux-job](https://hud.pytorch.org/pr/pytorch/rl/2225#26116944129) ([gh](https://github.com/pytorch/rl/actions/runs/9479108961/job/26116944129)) ([trunk failure](https://hud.pytorch.org/pytorch/rl/commit/f613eefb0f44b6a9ccd79f82703bb309aa165009#26116897983)) `##[error]The operation was canceled.`

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions[bot] commented 4 weeks ago

$\color{#D29922}\textsf{\Large\⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 91. Improved: $\large\color{#35bf28}10$. Worsened: $\large\color{#d91a1a}3$.

Expand to view detailed results | Name | Max | Mean | Ops | Ops on Repo `HEAD` | Change | | ----------------------------------------------------------------------------------------- | --------- | --------- | --------------- | ------------------ | ----------------------------------- | | test_single | 0.1100s | 58.6732ms | 17.0436 Ops/s | 17.4875 Ops/s | $\color{#d91a1a}-2.54\\%$ | | test_sync | 31.9678ms | 30.9995ms | 32.2586 Ops/s | 30.4264 Ops/s | $\textbf{\color{#35bf28}+6.02\\%}$ | | test_async | 57.8898ms | 28.6807ms | 34.8666 Ops/s | 34.5737 Ops/s | $\color{#35bf28}+0.85\\%$ | | test_simple | 0.4656s | 0.4043s | 2.4732 Ops/s | 2.6120 Ops/s | $\textbf{\color{#d91a1a}-5.31\\%}$ | | test_transformed | 0.6037s | 0.5497s | 1.8193 Ops/s | 1.8543 Ops/s | $\color{#d91a1a}-1.89\\%$ | | test_serial | 1.3154s | 1.2581s | 0.7948 Ops/s | 0.7681 Ops/s | $\color{#35bf28}+3.48\\%$ | | test_parallel | 1.1371s | 1.0604s | 0.9430 Ops/s | 0.9375 Ops/s | $\color{#35bf28}+0.59\\%$ | | test_step_mdp_speed[True-True-True-True-True] | 0.1796ms | 21.9155μs | 45.6299 KOps/s | 46.3847 KOps/s | $\color{#d91a1a}-1.63\\%$ | | test_step_mdp_speed[True-True-True-True-False] | 43.9720μs | 13.3208μs | 75.0706 KOps/s | 74.8732 KOps/s | $\color{#35bf28}+0.26\\%$ | | test_step_mdp_speed[True-True-True-False-True] | 37.3390μs | 12.9426μs | 77.2645 KOps/s | 77.8492 KOps/s | $\color{#d91a1a}-0.75\\%$ | | test_step_mdp_speed[True-True-True-False-False] | 32.1190μs | 7.8464μs | 127.4471 KOps/s | 128.2882 KOps/s | $\color{#d91a1a}-0.66\\%$ | | test_step_mdp_speed[True-True-False-True-True] | 57.6380μs | 23.0415μs | 43.3999 KOps/s | 43.2630 KOps/s | $\color{#35bf28}+0.32\\%$ | | test_step_mdp_speed[True-True-False-True-False] | 43.9920μs | 14.4832μs | 69.0454 KOps/s | 68.7975 KOps/s | $\color{#35bf28}+0.36\\%$ | | test_step_mdp_speed[True-True-False-False-True] | 39.7840μs | 14.0706μs | 71.0703 KOps/s | 70.5241 KOps/s | $\color{#35bf28}+0.77\\%$ | | test_step_mdp_speed[True-True-False-False-False] | 31.5990μs | 9.0554μs | 110.4319 KOps/s | 110.0740 KOps/s | $\color{#35bf28}+0.33\\%$ | | test_step_mdp_speed[True-False-True-True-True] | 72.9360μs | 24.4448μs | 40.9085 KOps/s | 40.8614 KOps/s | $\color{#35bf28}+0.12\\%$ | | test_step_mdp_speed[True-False-True-True-False] | 42.6690μs | 16.0344μs | 62.3659 KOps/s | 62.7608 KOps/s | $\color{#d91a1a}-0.63\\%$ | | test_step_mdp_speed[True-False-True-False-True] | 61.7550μs | 13.8833μs | 72.0288 KOps/s | 70.9909 KOps/s | $\color{#35bf28}+1.46\\%$ | | test_step_mdp_speed[True-False-True-False-False] | 33.9530μs | 9.0386μs | 110.6371 KOps/s | 109.2877 KOps/s | $\color{#35bf28}+1.23\\%$ | | test_step_mdp_speed[True-False-False-True-True] | 57.7080μs | 25.4134μs | 39.3493 KOps/s | 38.6653 KOps/s | $\color{#35bf28}+1.77\\%$ | | test_step_mdp_speed[True-False-False-True-False] | 51.4650μs | 16.9649μs | 58.9454 KOps/s | 57.8183 KOps/s | $\color{#35bf28}+1.95\\%$ | | test_step_mdp_speed[True-False-False-False-True] | 70.9030μs | 15.0419μs | 66.4808 KOps/s | 65.3998 KOps/s | $\color{#35bf28}+1.65\\%$ | | test_step_mdp_speed[True-False-False-False-False] | 39.5130μs | 10.1498μs | 98.5242 KOps/s | 96.3010 KOps/s | $\color{#35bf28}+2.31\\%$ | | test_step_mdp_speed[False-True-True-True-True] | 62.9680μs | 24.4867μs | 40.8384 KOps/s | 40.5766 KOps/s | $\color{#35bf28}+0.65\\%$ | | test_step_mdp_speed[False-True-True-True-False] | 38.0310μs | 15.9586μs | 62.6620 KOps/s | 62.0885 KOps/s | $\color{#35bf28}+0.92\\%$ | | test_step_mdp_speed[False-True-True-False-True] | 42.2590μs | 16.4271μs | 60.8750 KOps/s | 60.8166 KOps/s | $\color{#35bf28}+0.10\\%$ | | test_step_mdp_speed[False-True-True-False-False] | 31.9100μs | 10.1776μs | 98.2545 KOps/s | 95.8099 KOps/s | $\color{#35bf28}+2.55\\%$ | | test_step_mdp_speed[False-True-False-True-True] | 78.0560μs | 25.3232μs | 39.4895 KOps/s | 39.4883 KOps/s | $+0.00\\%$ | | test_step_mdp_speed[False-True-False-True-False] | 41.9080μs | 17.0247μs | 58.7381 KOps/s | 58.6751 KOps/s | $\color{#35bf28}+0.11\\%$ | | test_step_mdp_speed[False-True-False-False-True] | 44.2420μs | 17.3815μs | 57.5326 KOps/s | 57.0580 KOps/s | $\color{#35bf28}+0.83\\%$ | | test_step_mdp_speed[False-True-False-False-False] | 42.2490μs | 11.4333μs | 87.4637 KOps/s | 85.9966 KOps/s | $\color{#35bf28}+1.71\\%$ | | test_step_mdp_speed[False-False-True-True-True] | 55.7330μs | 26.7540μs | 37.3776 KOps/s | 36.9649 KOps/s | $\color{#35bf28}+1.12\\%$ | | test_step_mdp_speed[False-False-True-True-False] | 70.0100μs | 18.2704μs | 54.7335 KOps/s | 54.3706 KOps/s | $\color{#35bf28}+0.67\\%$ | | test_step_mdp_speed[False-False-True-False-True] | 43.5810μs | 17.5219μs | 57.0715 KOps/s | 57.0788 KOps/s | $\color{#d91a1a}-0.01\\%$ | | test_step_mdp_speed[False-False-True-False-False] | 37.6400μs | 11.5078μs | 86.8972 KOps/s | 86.0738 KOps/s | $\color{#35bf28}+0.96\\%$ | | test_step_mdp_speed[False-False-False-True-True] | 39.5140μs | 28.1209μs | 35.5608 KOps/s | 35.1689 KOps/s | $\color{#35bf28}+1.11\\%$ | | test_step_mdp_speed[False-False-False-True-False] | 44.7840μs | 19.3666μs | 51.6354 KOps/s | 51.8535 KOps/s | $\color{#d91a1a}-0.42\\%$ | | test_step_mdp_speed[False-False-False-False-True] | 51.0550μs | 18.4928μs | 54.0752 KOps/s | 54.4853 KOps/s | $\color{#d91a1a}-0.75\\%$ | | test_step_mdp_speed[False-False-False-False-False] | 40.6360μs | 12.5957μs | 79.3919 KOps/s | 79.9640 KOps/s | $\color{#d91a1a}-0.72\\%$ | | test_values[generalized_advantage_estimate-True-True] | 9.8453ms | 9.1331ms | 109.4919 Ops/s | 104.9476 Ops/s | $\color{#35bf28}+4.33\\%$ | | test_values[vec_generalized_advantage_estimate-True-True] | 38.5980ms | 33.6183ms | 29.7457 Ops/s | 28.1883 Ops/s | $\textbf{\color{#35bf28}+5.53\\%}$ | | test_values[td0_return_estimate-False-False] | 0.2234ms | 0.1680ms | 5.9532 KOps/s | 6.0744 KOps/s | $\color{#d91a1a}-2.00\\%$ | | test_values[td1_return_estimate-False-False] | 25.9824ms | 23.0606ms | 43.3640 Ops/s | 43.1916 Ops/s | $\color{#35bf28}+0.40\\%$ | | test_values[vec_td1_return_estimate-False-False] | 35.6820ms | 33.6384ms | 29.7279 Ops/s | 28.2073 Ops/s | $\textbf{\color{#35bf28}+5.39\\%}$ | | test_values[td_lambda_return_estimate-True-False] | 37.9636ms | 33.1852ms | 30.1339 Ops/s | 29.8207 Ops/s | $\color{#35bf28}+1.05\\%$ | | test_values[vec_td_lambda_return_estimate-True-False] | 35.2873ms | 33.4484ms | 29.8968 Ops/s | 28.1367 Ops/s | $\textbf{\color{#35bf28}+6.26\\%}$ | | test_gae_speed[generalized_advantage_estimate-False-1-512] | 8.8803ms | 8.0192ms | 124.7003 Ops/s | 122.8795 Ops/s | $\color{#35bf28}+1.48\\%$ | | test_gae_speed[vec_generalized_advantage_estimate-True-1-512] | 4.8251ms | 2.0440ms | 489.2422 Ops/s | 553.0120 Ops/s | $\textbf{\color{#d91a1a}-11.53\\%}$ | | test_gae_speed[vec_generalized_advantage_estimate-False-1-512] | 0.4417ms | 0.3540ms | 2.8247 KOps/s | 2.8053 KOps/s | $\color{#35bf28}+0.69\\%$ | | test_gae_speed[vec_generalized_advantage_estimate-True-32-512] | 47.3112ms | 45.9829ms | 21.7472 Ops/s | 20.5132 Ops/s | $\textbf{\color{#35bf28}+6.02\\%}$ | | test_gae_speed[vec_generalized_advantage_estimate-False-32-512] | 3.7230ms | 3.0346ms | 329.5331 Ops/s | 324.2552 Ops/s | $\color{#35bf28}+1.63\\%$ | | test_dqn_speed | 1.6010ms | 1.3241ms | 755.2573 Ops/s | 724.7545 Ops/s | $\color{#35bf28}+4.21\\%$ | | test_ddpg_speed | 3.6603ms | 2.8505ms | 350.8157 Ops/s | 346.2159 Ops/s | $\color{#35bf28}+1.33\\%$ | | test_sac_speed | 10.4109ms | 8.4279ms | 118.6529 Ops/s | 117.0730 Ops/s | $\color{#35bf28}+1.35\\%$ | | test_redq_speed | 15.1175ms | 13.7209ms | 72.8815 Ops/s | 72.2563 Ops/s | $\color{#35bf28}+0.87\\%$ | | test_redq_deprec_speed | 15.5767ms | 13.6597ms | 73.2080 Ops/s | 72.9184 Ops/s | $\color{#35bf28}+0.40\\%$ | | test_td3_speed | 16.5874ms | 8.3797ms | 119.3362 Ops/s | 114.7912 Ops/s | $\color{#35bf28}+3.96\\%$ | | test_cql_speed | 37.0202ms | 36.4623ms | 27.4256 Ops/s | 26.4311 Ops/s | $\color{#35bf28}+3.76\\%$ | | test_a2c_speed | 8.8146ms | 7.4180ms | 134.8072 Ops/s | 129.0040 Ops/s | $\color{#35bf28}+4.50\\%$ | | test_ppo_speed | 8.5039ms | 7.6721ms | 130.3429 Ops/s | 125.6883 Ops/s | $\color{#35bf28}+3.70\\%$ | | test_reinforce_speed | 7.8127ms | 6.6249ms | 150.9454 Ops/s | 147.3620 Ops/s | $\color{#35bf28}+2.43\\%$ | | test_iql_speed | 34.9054ms | 33.2376ms | 30.0864 Ops/s | 30.1223 Ops/s | $\color{#d91a1a}-0.12\\%$ | | test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] | 5.6480ms | 3.6458ms | 274.2892 Ops/s | 283.9772 Ops/s | $\color{#d91a1a}-3.41\\%$ | | test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] | 1.2326ms | 0.5108ms | 1.9576 KOps/s | 2.0088 KOps/s | $\color{#d91a1a}-2.54\\%$ | | test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] | 0.6746ms | 0.4741ms | 2.1092 KOps/s | 1.9945 KOps/s | $\textbf{\color{#35bf28}+5.76\\%}$ | | test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] | 3.9777ms | 3.5372ms | 282.7096 Ops/s | 288.0734 Ops/s | $\color{#d91a1a}-1.86\\%$ | | test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] | 1.0737ms | 0.4880ms | 2.0490 KOps/s | 2.0410 KOps/s | $\color{#35bf28}+0.39\\%$ | | test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] | 0.7608ms | 0.4674ms | 2.1393 KOps/s | 2.1712 KOps/s | $\color{#d91a1a}-1.47\\%$ | | test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] | 2.5002ms | 1.6896ms | 591.8603 Ops/s | 592.6038 Ops/s | $\color{#d91a1a}-0.13\\%$ | | test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] | 4.5585ms | 1.6033ms | 623.7321 Ops/s | 624.3166 Ops/s | $\color{#d91a1a}-0.09\\%$ | | test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] | 4.1384ms | 3.5988ms | 277.8706 Ops/s | 278.2570 Ops/s | $\color{#d91a1a}-0.14\\%$ | | test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] | 0.1082s | 0.6914ms | 1.4463 KOps/s | 1.4111 KOps/s | $\color{#35bf28}+2.49\\%$ | | test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] | 1.0669ms | 0.6200ms | 1.6128 KOps/s | 1.6820 KOps/s | $\color{#d91a1a}-4.12\\%$ | | test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] | 3.7963ms | 3.5883ms | 278.6832 Ops/s | 270.9187 Ops/s | $\color{#35bf28}+2.87\\%$ | | test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] | 0.8161ms | 0.5030ms | 1.9881 KOps/s | 2.0212 KOps/s | $\color{#d91a1a}-1.64\\%$ | | test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] | 0.6168ms | 0.4712ms | 2.1224 KOps/s | 2.1090 KOps/s | $\color{#35bf28}+0.63\\%$ | | test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] | 5.1295ms | 3.4373ms | 290.9237 Ops/s | 275.8840 Ops/s | $\textbf{\color{#35bf28}+5.45\\%}$ | | test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] | 1.0242ms | 0.4890ms | 2.0449 KOps/s | 2.0110 KOps/s | $\color{#35bf28}+1.69\\%$ | | test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] | 0.7507ms | 0.4697ms | 2.1292 KOps/s | 2.1270 KOps/s | $\color{#35bf28}+0.11\\%$ | | test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] | 5.4767ms | 3.5759ms | 279.6491 Ops/s | 271.0329 Ops/s | $\color{#35bf28}+3.18\\%$ | | test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] | 0.7252ms | 0.6088ms | 1.6426 KOps/s | 1.6232 KOps/s | $\color{#35bf28}+1.20\\%$ | | test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] | 0.8500ms | 0.5979ms | 1.6725 KOps/s | 1.7124 KOps/s | $\color{#d91a1a}-2.33\\%$ | | test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] | 0.1265s | 8.3105ms | 120.3304 Ops/s | 165.8255 Ops/s | $\textbf{\color{#d91a1a}-27.44\\%}$ | | test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] | 14.7190ms | 12.4153ms | 80.5459 Ops/s | 67.5266 Ops/s | $\textbf{\color{#35bf28}+19.28\\%}$ | | test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] | 1.5316ms | 1.0395ms | 961.9597 Ops/s | 961.0530 Ops/s | $\color{#35bf28}+0.09\\%$ | | test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] | 0.1106s | 5.8387ms | 171.2704 Ops/s | 174.0567 Ops/s | $\color{#d91a1a}-1.60\\%$ | | test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] | 18.2775ms | 12.5089ms | 79.9428 Ops/s | 78.4946 Ops/s | $\color{#35bf28}+1.85\\%$ | | test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] | 1.5393ms | 1.0393ms | 962.2156 Ops/s | 883.0124 Ops/s | $\textbf{\color{#35bf28}+8.97\\%}$ | | test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] | 0.1088s | 5.9305ms | 168.6207 Ops/s | 167.6914 Ops/s | $\color{#35bf28}+0.55\\%$ | | test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] | 0.1177s | 14.6402ms | 68.3049 Ops/s | 67.2519 Ops/s | $\color{#35bf28}+1.57\\%$ | | test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] | 1.9067ms | 1.1903ms | 840.1240 Ops/s | 768.7295 Ops/s | $\textbf{\color{#35bf28}+9.29\\%}$ |
github-actions[bot] commented 4 weeks ago

$\color{#D29922}\textsf{\Large\⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 94. Improved: $\large\color{#35bf28}2$. Worsened: $\large\color{#d91a1a}6$.

Expand to view detailed results | Name | Max | Mean | Ops | Ops on Repo `HEAD` | Change | | ----------------------------------------------------------------------------------------- | --------- | --------- | -------------- | ------------------ | ----------------------------------- | | test_single | 0.1174s | 0.1170s | 8.5503 Ops/s | 8.3275 Ops/s | $\color{#35bf28}+2.67\\%$ | | test_sync | 0.1048s | 0.1039s | 9.6281 Ops/s | 9.7085 Ops/s | $\color{#d91a1a}-0.83\\%$ | | test_async | 0.1993s | 0.1007s | 9.9261 Ops/s | 10.1494 Ops/s | $\color{#d91a1a}-2.20\\%$ | | test_single_pixels | 0.1285s | 0.1283s | 7.7939 Ops/s | 7.9676 Ops/s | $\color{#d91a1a}-2.18\\%$ | | test_sync_pixels | 84.8250ms | 80.9513ms | 12.3531 Ops/s | 12.3683 Ops/s | $\color{#d91a1a}-0.12\\%$ | | test_async_pixels | 0.1588s | 69.0241ms | 14.4877 Ops/s | 14.5651 Ops/s | $\color{#d91a1a}-0.53\\%$ | | test_simple | 0.8020s | 0.8010s | 1.2484 Ops/s | 1.2472 Ops/s | $\color{#35bf28}+0.10\\%$ | | test_transformed | 1.0684s | 1.0652s | 0.9388 Ops/s | 0.9490 Ops/s | $\color{#d91a1a}-1.07\\%$ | | test_serial | 2.5546s | 2.4860s | 0.4023 Ops/s | 0.4125 Ops/s | $\color{#d91a1a}-2.49\\%$ | | test_parallel | 2.3819s | 2.3344s | 0.4284 Ops/s | 0.4288 Ops/s | $\color{#d91a1a}-0.11\\%$ | | test_step_mdp_speed[True-True-True-True-True] | 93.0920μs | 33.2061μs | 30.1150 KOps/s | 30.8427 KOps/s | $\color{#d91a1a}-2.36\\%$ | | test_step_mdp_speed[True-True-True-True-False] | 61.1210μs | 19.5556μs | 51.1363 KOps/s | 50.8376 KOps/s | $\color{#35bf28}+0.59\\%$ | | test_step_mdp_speed[True-True-True-False-True] | 85.5910μs | 18.7120μs | 53.4416 KOps/s | 53.5868 KOps/s | $\color{#d91a1a}-0.27\\%$ | | test_step_mdp_speed[True-True-True-False-False] | 52.9100μs | 11.1927μs | 89.3440 KOps/s | 89.2390 KOps/s | $\color{#35bf28}+0.12\\%$ | | test_step_mdp_speed[True-True-False-True-True] | 68.8810μs | 34.8776μs | 28.6717 KOps/s | 29.0822 KOps/s | $\color{#d91a1a}-1.41\\%$ | | test_step_mdp_speed[True-True-False-True-False] | 48.7810μs | 21.5056μs | 46.4994 KOps/s | 46.3735 KOps/s | $\color{#35bf28}+0.27\\%$ | | test_step_mdp_speed[True-True-False-False-True] | 54.9200μs | 20.4978μs | 48.7857 KOps/s | 48.4469 KOps/s | $\color{#35bf28}+0.70\\%$ | | test_step_mdp_speed[True-True-False-False-False] | 43.7800μs | 13.1059μs | 76.3016 KOps/s | 76.0362 KOps/s | $\color{#35bf28}+0.35\\%$ | | test_step_mdp_speed[True-False-True-True-True] | 82.8810μs | 36.9452μs | 27.0671 KOps/s | 27.3842 KOps/s | $\color{#d91a1a}-1.16\\%$ | | test_step_mdp_speed[True-False-True-True-False] | 56.9710μs | 23.5340μs | 42.4917 KOps/s | 42.3424 KOps/s | $\color{#35bf28}+0.35\\%$ | | test_step_mdp_speed[True-False-True-False-True] | 48.4610μs | 20.6276μs | 48.4787 KOps/s | 48.6993 KOps/s | $\color{#d91a1a}-0.45\\%$ | | test_step_mdp_speed[True-False-True-False-False] | 45.4710μs | 13.1344μs | 76.1361 KOps/s | 76.4193 KOps/s | $\color{#d91a1a}-0.37\\%$ | | test_step_mdp_speed[True-False-False-True-True] | 71.2310μs | 38.4265μs | 26.0237 KOps/s | 26.0661 KOps/s | $\color{#d91a1a}-0.16\\%$ | | test_step_mdp_speed[True-False-False-True-False] | 54.5610μs | 25.4110μs | 39.3530 KOps/s | 39.4080 KOps/s | $\color{#d91a1a}-0.14\\%$ | | test_step_mdp_speed[True-False-False-False-True] | 58.5700μs | 22.3082μs | 44.8265 KOps/s | 45.1158 KOps/s | $\color{#d91a1a}-0.64\\%$ | | test_step_mdp_speed[True-False-False-False-False] | 54.6910μs | 15.0030μs | 66.6534 KOps/s | 66.4502 KOps/s | $\color{#35bf28}+0.31\\%$ | | test_step_mdp_speed[False-True-True-True-True] | 69.4510μs | 37.1345μs | 26.9292 KOps/s | 27.5732 KOps/s | $\color{#d91a1a}-2.34\\%$ | | test_step_mdp_speed[False-True-True-True-False] | 48.3910μs | 23.5284μs | 42.5019 KOps/s | 42.2891 KOps/s | $\color{#35bf28}+0.50\\%$ | | test_step_mdp_speed[False-True-True-False-True] | 53.8110μs | 24.6671μs | 40.5398 KOps/s | 40.3190 KOps/s | $\color{#35bf28}+0.55\\%$ | | test_step_mdp_speed[False-True-True-False-False] | 46.7810μs | 14.7520μs | 67.7875 KOps/s | 65.8768 KOps/s | $\color{#35bf28}+2.90\\%$ | | test_step_mdp_speed[False-True-False-True-True] | 74.7610μs | 38.5922μs | 25.9120 KOps/s | 26.4553 KOps/s | $\color{#d91a1a}-2.05\\%$ | | test_step_mdp_speed[False-True-False-True-False] | 53.2700μs | 25.4488μs | 39.2946 KOps/s | 39.5572 KOps/s | $\color{#d91a1a}-0.66\\%$ | | test_step_mdp_speed[False-True-False-False-True] | 60.7210μs | 26.6400μs | 37.5375 KOps/s | 37.7258 KOps/s | $\color{#d91a1a}-0.50\\%$ | | test_step_mdp_speed[False-True-False-False-False] | 38.7710μs | 16.7259μs | 59.7876 KOps/s | 59.5334 KOps/s | $\color{#35bf28}+0.43\\%$ | | test_step_mdp_speed[False-False-True-True-True] | 73.0310μs | 40.7710μs | 24.5272 KOps/s | 24.6312 KOps/s | $\color{#d91a1a}-0.42\\%$ | | test_step_mdp_speed[False-False-True-True-False] | 59.1410μs | 27.3350μs | 36.5832 KOps/s | 36.7544 KOps/s | $\color{#d91a1a}-0.47\\%$ | | test_step_mdp_speed[False-False-True-False-True] | 58.5610μs | 26.7911μs | 37.3258 KOps/s | 37.3926 KOps/s | $\color{#d91a1a}-0.18\\%$ | | test_step_mdp_speed[False-False-True-False-False] | 39.9710μs | 16.7623μs | 59.6576 KOps/s | 58.6721 KOps/s | $\color{#35bf28}+1.68\\%$ | | test_step_mdp_speed[False-False-False-True-True] | 81.7210μs | 43.4757μs | 23.0014 KOps/s | 23.5998 KOps/s | $\color{#d91a1a}-2.54\\%$ | | test_step_mdp_speed[False-False-False-True-False] | 68.7510μs | 29.3635μs | 34.0559 KOps/s | 34.3424 KOps/s | $\color{#d91a1a}-0.83\\%$ | | test_step_mdp_speed[False-False-False-False-True] | 68.0910μs | 28.3441μs | 35.2808 KOps/s | 35.5106 KOps/s | $\color{#d91a1a}-0.65\\%$ | | test_step_mdp_speed[False-False-False-False-False] | 74.3410μs | 18.5306μs | 53.9649 KOps/s | 53.8326 KOps/s | $\color{#35bf28}+0.25\\%$ | | test_values[generalized_advantage_estimate-True-True] | 26.4705ms | 25.4903ms | 39.2306 Ops/s | 42.2272 Ops/s | $\textbf{\color{#d91a1a}-7.10\\%}$ | | test_values[vec_generalized_advantage_estimate-True-True] | 91.6816ms | 2.7256ms | 366.8962 Ops/s | 370.4662 Ops/s | $\color{#d91a1a}-0.96\\%$ | | test_values[td0_return_estimate-False-False] | 91.4820μs | 64.7736μs | 15.4384 KOps/s | 15.5207 KOps/s | $\color{#d91a1a}-0.53\\%$ | | test_values[td1_return_estimate-False-False] | 57.9975ms | 54.9990ms | 18.1822 Ops/s | 18.6909 Ops/s | $\color{#d91a1a}-2.72\\%$ | | test_values[vec_td1_return_estimate-False-False] | 1.3399ms | 1.0745ms | 930.6248 Ops/s | 936.4206 Ops/s | $\color{#d91a1a}-0.62\\%$ | | test_values[td_lambda_return_estimate-True-False] | 91.5727ms | 88.2609ms | 11.3301 Ops/s | 11.6948 Ops/s | $\color{#d91a1a}-3.12\\%$ | | test_values[vec_td_lambda_return_estimate-True-False] | 1.3990ms | 1.0748ms | 930.3704 Ops/s | 930.5607 Ops/s | $\color{#d91a1a}-0.02\\%$ | | test_gae_speed[generalized_advantage_estimate-False-1-512] | 26.3924ms | 26.0826ms | 38.3397 Ops/s | 42.1672 Ops/s | $\textbf{\color{#d91a1a}-9.08\\%}$ | | test_gae_speed[vec_generalized_advantage_estimate-True-1-512] | 0.9930ms | 0.7478ms | 1.3373 KOps/s | 1.4131 KOps/s | $\textbf{\color{#d91a1a}-5.36\\%}$ | | test_gae_speed[vec_generalized_advantage_estimate-False-1-512] | 0.7485ms | 0.6574ms | 1.5212 KOps/s | 1.5391 KOps/s | $\color{#d91a1a}-1.17\\%$ | | test_gae_speed[vec_generalized_advantage_estimate-True-32-512] | 1.5424ms | 1.4602ms | 684.8353 Ops/s | 687.9262 Ops/s | $\color{#d91a1a}-0.45\\%$ | | test_gae_speed[vec_generalized_advantage_estimate-False-32-512] | 0.7358ms | 0.6746ms | 1.4825 KOps/s | 1.4982 KOps/s | $\color{#d91a1a}-1.05\\%$ | | test_dqn_speed | 2.2884ms | 1.4884ms | 671.8643 Ops/s | 699.8841 Ops/s | $\color{#d91a1a}-4.00\\%$ | | test_ddpg_speed | 3.2076ms | 3.0314ms | 329.8800 Ops/s | 332.1989 Ops/s | $\color{#d91a1a}-0.70\\%$ | | test_sac_speed | 9.1307ms | 8.7658ms | 114.0797 Ops/s | 117.8104 Ops/s | $\color{#d91a1a}-3.17\\%$ | | test_redq_speed | 14.1490ms | 11.1615ms | 89.5937 Ops/s | 93.1362 Ops/s | $\color{#d91a1a}-3.80\\%$ | | test_redq_deprec_speed | 12.4334ms | 11.9773ms | 83.4911 Ops/s | 83.6837 Ops/s | $\color{#d91a1a}-0.23\\%$ | | test_td3_speed | 8.8113ms | 8.5862ms | 116.4656 Ops/s | 118.5041 Ops/s | $\color{#d91a1a}-1.72\\%$ | | test_cql_speed | 29.5955ms | 26.4931ms | 37.7457 Ops/s | 38.1679 Ops/s | $\color{#d91a1a}-1.11\\%$ | | test_a2c_speed | 5.8103ms | 5.6676ms | 176.4415 Ops/s | 175.0470 Ops/s | $\color{#35bf28}+0.80\\%$ | | test_ppo_speed | 6.2039ms | 5.9945ms | 166.8187 Ops/s | 164.3663 Ops/s | $\color{#35bf28}+1.49\\%$ | | test_reinforce_speed | 4.9493ms | 4.6381ms | 215.6070 Ops/s | 211.6826 Ops/s | $\color{#35bf28}+1.85\\%$ | | test_iql_speed | 20.6981ms | 19.9878ms | 50.0306 Ops/s | 49.2573 Ops/s | $\color{#35bf28}+1.57\\%$ | | test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] | 5.0549ms | 4.8647ms | 205.5637 Ops/s | 209.7583 Ops/s | $\color{#d91a1a}-2.00\\%$ | | test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] | 0.7289ms | 0.6008ms | 1.6645 KOps/s | 1.6819 KOps/s | $\color{#d91a1a}-1.03\\%$ | | test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] | 0.8076ms | 0.5787ms | 1.7281 KOps/s | 1.7422 KOps/s | $\color{#d91a1a}-0.81\\%$ | | test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] | 5.0282ms | 4.8116ms | 207.8327 Ops/s | 211.2381 Ops/s | $\color{#d91a1a}-1.61\\%$ | | test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] | 0.7013ms | 0.5948ms | 1.6812 KOps/s | 1.6997 KOps/s | $\color{#d91a1a}-1.09\\%$ | | test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] | 0.6775ms | 0.5703ms | 1.7535 KOps/s | 1.4770 KOps/s | $\textbf{\color{#35bf28}+18.72\\%}$ | | test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] | 2.2935ms | 2.1415ms | 466.9656 Ops/s | 475.0392 Ops/s | $\color{#d91a1a}-1.70\\%$ | | test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] | 2.2346ms | 2.0349ms | 491.4181 Ops/s | 498.7752 Ops/s | $\color{#d91a1a}-1.48\\%$ | | test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] | 5.0409ms | 4.9508ms | 201.9879 Ops/s | 204.9605 Ops/s | $\color{#d91a1a}-1.45\\%$ | | test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] | 0.8403ms | 0.7335ms | 1.3633 KOps/s | 1.3838 KOps/s | $\color{#d91a1a}-1.48\\%$ | | test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] | 4.6609ms | 0.7124ms | 1.4036 KOps/s | 1.4312 KOps/s | $\color{#d91a1a}-1.93\\%$ | | test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] | 4.9106ms | 4.8323ms | 206.9426 Ops/s | 210.5046 Ops/s | $\color{#d91a1a}-1.69\\%$ | | test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] | 0.7062ms | 0.6150ms | 1.6261 KOps/s | 1.6802 KOps/s | $\color{#d91a1a}-3.22\\%$ | | test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] | 4.4558ms | 0.5888ms | 1.6984 KOps/s | 1.7321 KOps/s | $\color{#d91a1a}-1.95\\%$ | | test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] | 5.0281ms | 4.8111ms | 207.8522 Ops/s | 210.4638 Ops/s | $\color{#d91a1a}-1.24\\%$ | | test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] | 1.3643ms | 0.6007ms | 1.6648 KOps/s | 1.6884 KOps/s | $\color{#d91a1a}-1.40\\%$ | | test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] | 0.7547ms | 0.5760ms | 1.7362 KOps/s | 1.7581 KOps/s | $\color{#d91a1a}-1.25\\%$ | | test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] | 5.1088ms | 4.9413ms | 202.3740 Ops/s | 203.5674 Ops/s | $\color{#d91a1a}-0.59\\%$ | | test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] | 0.8354ms | 0.7319ms | 1.3662 KOps/s | 1.3755 KOps/s | $\color{#d91a1a}-0.67\\%$ | | test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] | 0.8449ms | 0.7073ms | 1.4138 KOps/s | 1.4044 KOps/s | $\color{#35bf28}+0.67\\%$ | | test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] | 0.1166s | 9.5004ms | 105.2586 Ops/s | 132.3098 Ops/s | $\textbf{\color{#d91a1a}-20.45\\%}$ | | test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] | 19.3666ms | 16.8499ms | 59.3477 Ops/s | 60.2861 Ops/s | $\color{#d91a1a}-1.56\\%$ | | test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] | 2.3691ms | 1.3361ms | 748.4674 Ops/s | 734.8903 Ops/s | $\color{#35bf28}+1.85\\%$ | | test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] | 0.1053s | 7.1817ms | 139.2424 Ops/s | 140.3009 Ops/s | $\color{#d91a1a}-0.75\\%$ | | test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] | 19.1849ms | 16.8203ms | 59.4520 Ops/s | 61.5705 Ops/s | $\color{#d91a1a}-3.44\\%$ | | test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] | 2.3749ms | 1.3366ms | 748.1604 Ops/s | 688.9337 Ops/s | $\textbf{\color{#35bf28}+8.60\\%}$ | | test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] | 0.1063s | 9.3792ms | 106.6189 Ops/s | 107.0856 Ops/s | $\color{#d91a1a}-0.44\\%$ | | test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] | 19.5205ms | 16.9335ms | 59.0546 Ops/s | 62.3638 Ops/s | $\textbf{\color{#d91a1a}-5.31\\%}$ | | test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] | 7.6435ms | 1.6133ms | 619.8427 Ops/s | 663.0822 Ops/s | $\textbf{\color{#d91a1a}-6.52\\%}$ |