[Question] RuntimeError: Unable to sample before the end of the first episode. We recommend choosing a value for learning_starts that is greater than the maximum number of timesteps in the environment.

moneypi commented 5 months ago

❓ Question

I can't train "parking-v0", every time I run command python train.py --algo tqc --env parking-v0, I get an error:

(base) zhuqinghua@lcwt-DSS8440:~/workspace/rl-baselines3-zoo-master$ ./train_parking.sh
========== parking-v0 ==========
Seed: 1667912339
Loading hyperparameters from: /home/zhuqinghua/workspace/rl-baselines3-zoo-master/hyperparams/tqc.yml
Default hyperparameters for environment (ones being tuned will be overridden):
OrderedDict([('batch_size', 256),
             ('buffer_size', 300000),
             ('gamma', 0.98),
             ('learning_rate', 0.0015),
             ('n_timesteps', 100000.0),
             ('policy', 'MultiInputPolicy'),
             ('policy_kwargs', 'dict(net_arch=[512, 512, 512], n_critics=2)'),
             ('replay_buffer_class', 'HerReplayBuffer'),
             ('replay_buffer_kwargs',
              "dict( goal_selection_strategy='episode', n_sampled_goal=4, )"),
             ('tau', 0.005)])
Using 1 environments
Creating test environment
Wrapping the env in a VecTransposeImage.
Wrapping the env in a VecTransposeImage.
Using cuda device
Log path: logs/tqc/parking-v0_14
Traceback (most recent call last):
  File "train.py", line 4, in <module>
    train()
  File "/home/zhuqinghua/workspace/rl-baselines3-zoo-master/rl_zoo3/train.py", line 272, in train
    exp_manager.learn(model)
  File "/home/zhuqinghua/workspace/rl-baselines3-zoo-master/rl_zoo3/exp_manager.py", line 241, in learn
    model.learn(self.n_timesteps, **kwargs)
  File "/home/zhuqinghua/miniconda3/lib/python3.8/site-packages/sb3_contrib/tqc/tqc.py", line 304, in learn
    return super().learn(
  File "/home/zhuqinghua/miniconda3/lib/python3.8/site-packages/stable_baselines3/common/off_policy_algorithm.py", line 347, in learn
    self.train(batch_size=self.batch_size, gradient_steps=gradient_steps)
  File "/home/zhuqinghua/miniconda3/lib/python3.8/site-packages/sb3_contrib/tqc/tqc.py", line 211, in train
    replay_data = self.replay_buffer.sample(batch_size, env=self._vec_normalize_env)  # type: ignore[union-attr]
  File "/home/zhuqinghua/miniconda3/lib/python3.8/site-packages/stable_baselines3/her/her_replay_buffer.py", line 198, in sample
    raise RuntimeError(
RuntimeError: Unable to sample before the end of the first episode. We recommend choosing a value for learning_starts that is greater than the maximum number of timesteps in the environment.
(base) zhuqinghua@lcwt-DSS8440:~/workspace/rl-baselines3-zoo-master$

Checklist

[X] I have checked that there is no similar issue in the repo
[X] I have read the SB3 documentation
[X] I have read the RL Zoo documentation
[X] If code there is, it is minimal and working
[X] If code there is, it is formatted using the markdown code blocks for both code and stack traces.

araffin commented 5 months ago

Hello, please share the different versions of packages you are using.

qgallouedec commented 5 months ago

Can you share the versions you use (see the bug report issue template; next time use this template instead of the question template)

moneypi commented 5 months ago

(base) zhuqinghua@lcwt-DSS8440:~/workspace/highway-env_minimum$ python -c 'import stable_baselines3 as sb3; sb3.get_system_info()'
- OS: Linux-6.5.0-14-generic-x86_64-with-glibc2.17 # 14~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Nov 20 18:15:30 UTC 2
- Python: 3.8.18
- Stable-Baselines3: 2.2.1
- PyTorch: 2.0.0.post200
- GPU Enabled: True
- Numpy: 1.24.3
- Cloudpickle: 2.2.1
- Gymnasium: 0.28.1
- OpenAI Gym: 0.26.2

(base) zhuqinghua@lcwt-DSS8440:~/workspace/highway-env_minimum$ conda list
# packages in environment at /home/zhuqinghua/miniconda3:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                  2_kmp_llvm    conda-forge
abseil-cpp                20220623.0           h8cdb687_6    conda-forge
absl-py                   1.4.0            py38h06a4308_0
aiohttp                   3.9.0            py38h5eee18b_0
aiosignal                 1.2.0              pyhd3eb1b0_0
ale-py                    0.8.1                    pypi_0    pypi
alembic                   1.8.1            py38h06a4308_0
alsa-lib                  1.2.10               hd590300_0    conda-forge
aom                       3.6.0                h6a678d5_0
archspec                  0.2.1              pyhd3eb1b0_0
async-timeout             4.0.3            py38h06a4308_0
attr                      2.5.1                h166bdaf_1    conda-forge
attrs                     23.1.0           py38h06a4308_0
autorom                   0.6.1                    pypi_0    pypi
blas                      1.0                    openblas
blinker                   1.6.2            py38h06a4308_0
boltons                   23.0.0           py38h06a4308_0
bottleneck                1.3.5            py38h7deecbd_0
box2d-py                  2.3.8            py38h17151c0_8    conda-forge
brotli                    1.0.9                h5eee18b_7
brotli-bin                1.0.9                h5eee18b_7
brotli-python             1.0.9            py38h6a678d5_7
bzip2                     1.0.8                h7b6447c_0
c-ares                    1.19.1               h5eee18b_0
ca-certificates           2023.12.12           h06a4308_0
cachetools                4.2.2              pyhd3eb1b0_0
cairo                     1.18.0               h3faef2a_0    conda-forge
certifi                   2023.11.17       py38h06a4308_0
cffi                      1.16.0           py38h5eee18b_0
charset-normalizer        2.0.4              pyhd3eb1b0_0
click                     8.1.7            py38h06a4308_0
cloudpickle               2.2.1            py38h06a4308_0
colorlog                  5.0.1            py38h06a4308_1
conda                     23.11.0          py38h06a4308_0
conda-content-trust       0.2.0            py38h06a4308_0
conda-libmamba-solver     23.12.0            pyhd3eb1b0_1
conda-package-handling    2.2.0            py38h06a4308_0
conda-package-streaming   0.9.0            py38h06a4308_0
contourpy                 1.1.1                    pypi_0    pypi
cryptography              41.0.7           py38hdda0065_0
cuda-version              11.8                 h70ddcb2_2    conda-forge
cudatoolkit               11.8.0               h6a678d5_0
cudnn                     8.9.2.26               cuda11_0
cycler                    0.12.1                   pypi_0    pypi
dav1d                     1.2.1                h5eee18b_0
dbus                      1.13.18              hb2f20db_0
decorator                 5.1.1              pyhd3eb1b0_0
distro                    1.8.0            py38h06a4308_0
exceptiongroup            1.2.0                    pypi_0    pypi
expat                     2.5.0                h6a678d5_0
farama-notifications      0.0.4            py38h06a4308_0
ffmpeg                    4.2.2                h20bf706_0
filelock                  3.13.1           py38h06a4308_0
fluidsynth                2.3.4                h7741c18_0    conda-forge
fmt                       9.1.0                hdb19cb5_0
font-ttf-dejavu-sans-mono 2.37                 hd3eb1b0_0
font-ttf-inconsolata      2.001                hcb22688_0
font-ttf-source-code-pro  2.030                hd3eb1b0_0
font-ttf-ubuntu           0.83                 h8b1ccd4_0
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonts-anaconda            1                    h8fa9717_0
fonts-conda-ecosystem     1                    hd3eb1b0_0
fonttools                 4.47.2                   pypi_0    pypi
freetype                  2.12.1               h4a9f257_0
frozenlist                1.4.0            py38h5eee18b_0
fsspec                    2023.10.0        py38h06a4308_0
gettext                   0.21.1               h27087fc_0    conda-forge
giflib                    5.2.1                h5eee18b_3
glib                      2.78.3               hfc55251_0    conda-forge
glib-tools                2.78.3               hfc55251_0    conda-forge
gmp                       6.2.1                h295c915_3
gmpy2                     2.1.2            py38heeb90bb_0
gnutls                    3.6.15               he1e5248_0
google-auth               2.22.0           py38h06a4308_0
google-auth-oauthlib      0.5.2            py38h06a4308_0
graphite2                 1.3.14               h295c915_1
greenlet                  3.0.1            py38h6a678d5_0
grpc-cpp                  1.51.1               h27aab58_1    conda-forge
grpcio                    1.51.1           py38h5c03f07_1    conda-forge
gtest                     1.14.0               hdb19cb5_0
gym                       0.26.2                   pypi_0    pypi
gym-notices               0.0.8                    pypi_0    pypi
gymnasium                 0.28.1           py38h0df68fb_0    conda-forge
gymnasium-box2d           0.28.1           py38h578d9bd_0    conda-forge
harfbuzz                  8.3.0                h3d44ed6_0    conda-forge
highway-env               1.8.2                    pypi_0    pypi
huggingface-sb3           3.0                      pypi_0    pypi
huggingface_hub           0.20.2             pyhd8ed1ab_0    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
idna                      3.4              py38h06a4308_0
imageio                   2.31.4           py38h06a4308_0
imageio-ffmpeg            0.4.9              pyhd8ed1ab_0    conda-forge
importlib-metadata        7.0.1            py38h06a4308_0
importlib_metadata        7.0.1                hd3eb1b0_0
importlib_resources       6.1.1            py38h06a4308_1
iniconfig                 2.0.0                    pypi_0    pypi
intel-openmp              2023.1.0         hdb19cb5_46306
jack                      1.9.22               h7c63dc7_2    conda-forge
jax-jumpy                 1.0.0            py38h06a4308_0
jinja2                    3.1.2            py38h06a4308_0
jpeg                      9e                   h5eee18b_1
jsonpatch                 1.32               pyhd3eb1b0_0
jsonpointer               2.1                pyhd3eb1b0_0
kiwisolver                1.4.5                    pypi_0    pypi
krb5                      1.20.1               h143b758_1
lame                      3.100                h7b6447c_0
lcms2                     2.12                 h3be6417_0
ld_impl_linux-64          2.38                 h1181459_1
lerc                      3.0                  h295c915_0
libabseil                 20220623.0      cxx17_h05df665_6    conda-forge
libarchive                3.6.2                h6ac8c49_2
libavif                   0.11.1               h5eee18b_0
libblas                   3.9.0           16_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h5eee18b_7
libbrotlidec              1.0.9                h5eee18b_7
libbrotlienc              1.0.9                h5eee18b_7
libcap                    2.69                 h0f662aa_0    conda-forge
libcblas                  3.9.0           16_linux64_openblas    conda-forge
libcurl                   8.4.0                h251f7ec_1
libdb                     6.2.32               h6a678d5_1
libdeflate                1.17                 h5eee18b_1
libedit                   3.1.20230828         h5eee18b_0
libev                     4.33                 h7f8727e_1
libffi                    3.4.4                h6a678d5_0
libflac                   1.4.3                h59595ed_0    conda-forge
libgcc-ng                 13.2.0               h807b86a_3    conda-forge
libgcrypt                 1.10.3               hd590300_0    conda-forge
libgfortran-ng            11.2.0               h00389a5_1
libgfortran5              11.2.0               h1234567_1
libglib                   2.78.3               h783c2da_0    conda-forge
libgomp                   13.2.0               h807b86a_3    conda-forge
libgpg-error              1.47                 h71f35ed_0    conda-forge
libgrpc                   1.51.1               h4fad500_1    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
libidn2                   2.3.4                h5eee18b_0
liblapack                 3.9.0           16_linux64_openblas    conda-forge
libmad                    0.15.1b           h0b41bf4_1001    conda-forge
libmagma                  2.7.1                hc72dce7_6    conda-forge
libmagma_sparse           2.7.1                h8354cda_6    conda-forge
libmamba                  1.5.3                haf1ee3a_0
libmambapy                1.5.3            py38h2dafd23_0
libnghttp2                1.57.0               h2d74bed_0
libogg                    1.3.5                h27cfd23_1
libopenblas               0.3.21               h043d6bf_0
libopus                   1.3.1                h7b6447c_0
libpng                    1.6.39               h5eee18b_0
libprotobuf               3.21.12              hfc55251_2    conda-forge
libsndfile                1.2.2                hc60ed4a_1    conda-forge
libsolv                   0.7.24               he621ea3_0
libssh2                   1.10.0               hdbd6064_2
libstdcxx-ng              13.2.0               h7e041cc_3    conda-forge
libsystemd0               255                  h3516f8a_0    conda-forge
libtasn1                  4.19.0               h5eee18b_0
libtiff                   4.5.1                h6a678d5_0
libunistring              0.9.10               h27cfd23_0
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libvorbis                 1.3.7                h7b6447c_0
libvpx                    1.7.0                h439df22_0
libwebp                   1.3.2                h11a3e52_0
libwebp-base              1.3.2                h5eee18b_0
libxcb                    1.15                 h7f8727e_0
libxml2                   2.10.4               hf1b16e4_1
libzlib                   1.2.13               hd590300_5    conda-forge
llvm-openmp               17.0.6               h4dfa4b3_0    conda-forge
lz4                       4.3.3                    pypi_0    pypi
lz4-c                     1.9.4                h6a678d5_0
magma                     2.7.1                ha770c72_6    conda-forge
mako                      1.2.3            py38h06a4308_0
markdown                  3.4.1            py38h06a4308_0
markupsafe                2.1.3            py38h5eee18b_0
matplotlib                3.7.4                    pypi_0    pypi
matplotlib-base           3.7.2            py38h1128e8f_0
menuinst                  2.0.1            py38h06a4308_1
mkl                       2022.2.1         h84fe81f_16997    conda-forge
mkl-service               2.4.0            py38h80f09db_0    conda-forge
mkl_fft                   1.3.8            py38h15cf20e_0    conda-forge
mkl_random                1.2.4            py38h3afe8b8_2    conda-forge
moviepy                   1.0.3              pyhd8ed1ab_1    conda-forge
mpc                       1.1.0                h10f8cd9_1
mpfr                      4.0.2                hb69a4c5_1
mpg123                    1.32.4               h59595ed_0    conda-forge
mpmath                    1.3.0            py38h06a4308_0
multidict                 6.0.4            py38h5eee18b_0
munkres                   1.1.4                      py_0
nccl                      2.19.4.1             h6103f9b_0    conda-forge
ncurses                   6.4                  h6a678d5_0
nettle                    3.7.3                hbbd107a_1
networkx                  3.1              py38h06a4308_0
ninja                     1.10.2               h06a4308_5
ninja-base                1.10.2               hd09550d_5
numexpr                   2.8.4            py38hd2a5715_1
numpy                     1.24.3           py38hf838250_0
numpy-base                1.24.3           py38h1e6e340_0
nvidia-cublas-cu12        12.1.3.1                 pypi_0    pypi
nvidia-cuda-cupti-cu12    12.1.105                 pypi_0    pypi
nvidia-cuda-nvrtc-cu12    12.1.105                 pypi_0    pypi
nvidia-cuda-runtime-cu12  12.1.105                 pypi_0    pypi
nvidia-cufft-cu12         11.0.2.54                pypi_0    pypi
nvidia-curand-cu12        10.3.2.106               pypi_0    pypi
nvidia-cusparse-cu12      12.1.0.106               pypi_0    pypi
nvidia-nccl-cu12          2.18.1                   pypi_0    pypi
nvidia-nvjitlink-cu12     12.3.101                 pypi_0    pypi
nvidia-nvtx-cu12          12.1.105                 pypi_0    pypi
oauthlib                  3.2.2            py38h06a4308_0
opencv-python             4.9.0.80                 pypi_0    pypi
openh264                  2.1.1                h4ff587b_0
openjpeg                  2.4.0                h3ad879b_0
openssl                   3.0.12               h7f8727e_0
optuna                    3.5.0              pyhd8ed1ab_0    conda-forge
opusfile                  0.12                 h3358134_2    conda-forge
packaging                 23.1             py38h06a4308_0
pandas                    2.0.3            py38h1128e8f_0
pcre2                     10.42                hebb0a14_0
pillow                    10.2.0                   pypi_0    pypi
pip                       23.3.1           py38h06a4308_0
pixman                    0.43.0               h59595ed_0    conda-forge
platformdirs              3.10.0           py38h06a4308_0
pluggy                    1.0.0            py38h06a4308_1
pooch                     1.7.0            py38h06a4308_0
portaudio                 19.6.0               h7b6447c_4
portmidi                  2.0.4                h7c63dc7_2    conda-forge
proglog                   0.1.9                      py_0    conda-forge
protobuf                  4.21.12          py38h8dc9893_0    conda-forge
pulseaudio-client         16.1                 hb77b528_5    conda-forge
pyasn1                    0.4.8              pyhd3eb1b0_0
pyasn1-modules            0.2.8                      py_0
pybind11-abi              4                    hd3eb1b0_1
pycosat                   0.6.6            py38h5eee18b_0
pycparser                 2.21               pyhd3eb1b0_0
pygame                    2.5.2                    pypi_0    pypi
pyjwt                     2.4.0            py38h06a4308_0
pyopenssl                 23.2.0           py38h06a4308_0
pyparsing                 3.1.1                    pypi_0    pypi
pysocks                   1.7.1            py38h06a4308_0
pytest                    7.4.4                    pypi_0    pypi
python                    3.8.18               h955ad1f_0
python-dateutil           2.8.2              pyhd3eb1b0_0
python-tzdata             2023.3             pyhd3eb1b0_0
python_abi                3.8                      2_cp38    conda-forge
pytorch                   2.0.0           cuda112py38h5e67e12_200    conda-forge
pytorch-gpu               2.0.0           cuda112py38h9871d0b_200    conda-forge
pytz                      2023.3.post1     py38h06a4308_0
pyyaml                    6.0.1            py38h5eee18b_0
re2                       2023.02.01           hcb278e6_0    conda-forge
readline                  8.2                  h5eee18b_0
reproc                    14.2.4               h295c915_1
reproc-cpp                14.2.4               h295c915_1
requests                  2.31.0           py38h06a4308_0
requests-oauthlib         1.3.0                      py_0
rsa                       4.7.2              pyhd3eb1b0_1
ruamel.yaml               0.17.21          py38h5eee18b_0
ruamel.yaml.clib          0.2.6            py38h5eee18b_1
sb3-contrib               2.2.1              pyhd8ed1ab_0    conda-forge
scipy                     1.10.1           py38h32ae08f_1
sdl2                      2.28.5               h77f46ba_0    conda-forge
sdl2_image                2.6.3                h7b3d2e9_0    conda-forge
sdl2_mixer                2.6.3                h8830914_1    conda-forge
sdl2_ttf                  2.20.2               ha14f88b_2    conda-forge
setuptools                68.2.2           py38h06a4308_0
shimmy                    0.2.1                    pypi_0    pypi
six                       1.16.0             pyhd3eb1b0_1
sleef                     3.5.1                h9b69904_2    conda-forge
sqlalchemy                2.0.25           py38h5eee18b_0
sqlite                    3.41.2               h5eee18b_0
stable-baselines3         2.2.1              pyhd8ed1ab_0    conda-forge
sympy                     1.12             py38h06a4308_0
tbb                       2021.8.0             hdb19cb5_0
tensorboard               2.15.1             pyhd8ed1ab_0    conda-forge
tensorboard-data-server   0.7.0            py38h52d8a92_0
tk                        8.6.12               h1ccaba5_0
tomli                     2.0.1                    pypi_0    pypi
torchvision               0.10.1          py38cuda112h04b465a_0_cuda    conda-forge
tqdm                      4.65.0           py38hb070fc8_0
triton                    2.1.0                    pypi_0    pypi
typing-extensions         4.7.1            py38h06a4308_0
typing_extensions         4.7.1            py38h06a4308_0
tzdata                    2023.4                   pypi_0    pypi
urllib3                   1.26.18          py38h06a4308_0
wasabi                    1.1.2                    pypi_0    pypi
werkzeug                  2.2.3            py38h06a4308_0
wheel                     0.41.2           py38h06a4308_0
x264                      1!157.20191217       h7b6447c_0
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.1.1                hd590300_0    conda-forge
xorg-libsm                1.2.4                h7391055_0    conda-forge
xorg-libx11               1.8.7                h8ee46fc_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxrender           0.9.11               hd590300_0    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xproto               7.0.31            h27cfd23_1007
xz                        5.4.5                h5eee18b_0
yaml                      0.2.5                h7b6447c_0
yaml-cpp                  0.8.0                h6a678d5_0
yarl                      1.9.3            py38h5eee18b_0
zipp                      3.17.0           py38h06a4308_0
zlib                      1.2.13               hd590300_5    conda-forge
zstandard                 0.19.0           py38h5eee18b_0
zstd                      1.5.5                hc292b87_0
(base) zhuqinghua@lcwt-DSS8440:~/workspace/highway-env_minimum$ nvidia-smi
Mon Jan 22 09:57:58 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3090        Off | 00000000:89:00.0 Off |                  N/A |
| 53%   36C    P8              17W / 350W |    439MiB / 24576MiB |      5%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 3090        Off | 00000000:8A:00.0 Off |                  N/A |
| 52%   33C    P8               9W / 350W |      6MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   2  NVIDIA GeForce RTX 3090        Off | 00000000:C0:00.0 Off |                  N/A |
| 50%   31C    P8              13W / 350W |      6MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   3  NVIDIA GeForce RTX 3090        Off | 00000000:C1:00.0 Off |                  N/A |
| 49%   31C    P8              12W / 350W |      6MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A   1493963    C+G   /home/zhuqinghua/miniconda3/bin/python      427MiB |
+---------------------------------------------------------------------------------------+
(base) zhuqinghua@lcwt-DSS8440:~/workspace/highway-env_minimum$ cat /etc/issue
Ubuntu 22.04.3 LTS \n \l

Can you share the versions you use (see the bug report issue template; next time use this template instead of the question template)

araffin commented 5 months ago

Could you try upgrading to gymnasium 0.29.1? You should probably use -W ignore in that case: python -W ignore train.py --algo tqc --env parking-v0 --seed 1667912339.

I couldn't reproduce your issue so far. In case it happens, you can set -param learning_starts:1000 or a larger value to avoid this issue.

moneypi commented 5 months ago

Could you try upgrading to gymnasium 0.29.1? You should probably use -W ignore in that case: python -W ignore train.py --algo tqc --env parking-v0 --seed 1667912339.

I couldn't reproduce your issue so far. In case it happens, you can set -param learning_starts:1000 or a larger value to avoid this issue.

Thanks, learning_starts:1000 works fine for me. Thank you so much.

DLR-RM / rl-baselines3-zoo

[Question] RuntimeError: Unable to sample before the end of the first episode. We recommend choosing a value for learning_starts that is greater than the maximum number of timesteps in the environment. #433

❓ Question

Checklist