search

sakakibara-yuuki / rl

A modular, primitive-first, python-first PyTorch library for Reinforcement Learning.
MIT License
0 stars 0 forks source link

# RL examples #3

Open sakakibara-yuuki opened 10 months ago

sakakibara-yuuki commented 10 months ago

setting environment

そもそもexamplesのために環境設定するのか.... EXAMPLES.mdに従う.

sakakibara-yuuki commented 10 months ago

pip install

pip install hydra-core

Successfully installed antlr4-python3-runtime-4.9.3 hydra-core-1.3.2 omegaconf-2.3.0
sakakibara-yuuki commented 10 months ago

好きなディレクトリに行ってから実行しろと言われた が、EXAMPLES.mdに書いたほうがよいのでは

cd sac
python sac.py

👎 Error

Traceback (most recent call last):
  File "/home/sakakibara/project/rl/examples/sac/sac.py", line 19, in <module>
    import tqdm
ModuleNotFoundError: No module named 'tqdm'
sakakibara-yuuki commented 10 months ago

対応:

pip install tqdm

もう一度

python sac.py

しかし

/home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
  ret = run_job(
Error executing job with overrides: []
Traceback (most recent call last):
  File "/home/sakakibara/project/rl/examples/sac/sac.py", line 41, in main
    logger = get_logger(
  File "/home/sakakibara/project/rl/torchrl/record/loggers/utils.py", line 51, in get_logger
    logger = WandbLogger(
  File "/home/sakakibara/project/rl/torchrl/record/loggers/wandb.py", line 54, in __init__
    raise ImportError("wandb could not be imported")
ImportError: wandb could not be imported

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
sakakibara-yuuki commented 10 months ago

しかたないので

pip install wandb

wandbはチームで使うと有料サービスなのでなるべく使いたくない

sakakibara-yuuki commented 10 months ago 
$ find ./ -name wandb.py
> ./torchrl/record/loggers/wandb.py

どうやらwandbのスクリプトがあるっぽい?

sakakibara-yuuki commented 10 months ago

いや、そもそも https://github.com/sakakibara-yuuki/rl/issues/3#issuecomment-1713180478 で

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

とあるので環境変数を追加してみる

export HYDRA_FULL_ERROR=1
sakakibara-yuuki commented 10 months ago

再度実行

python sac.py

result:

/home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
  ret = run_job(
Error executing job with overrides: []
Traceback (most recent call last):
  File "/home/sakakibara/project/rl/examples/sac/sac.py", line 180, in <module>
    main()
  File "/home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/sakakibara/project/rl/examples/sac/sac.py", line 41, in main
    logger = get_logger(
  File "/home/sakakibara/project/rl/torchrl/record/loggers/utils.py", line 51, in get_logger
    logger = WandbLogger(
  File "/home/sakakibara/project/rl/torchrl/record/loggers/wandb.py", line 54, in __init__
    raise ImportError("wandb could not be imported")
ImportError: wandb could not be imported

WTF

sakakibara-yuuki commented 10 months ago

もしかして

python setup.py install

しなきゃいけないかも

sakakibara-yuuki commented 10 months ago 
pip install tensordict-nightly
python setup.py develop

今度はうまくいった. HYDRAの環境変数の影響?

-- PyTorch dependency: torch
running develop
running egg_info
writing torchrl.egg-info/PKG-INFO
writing dependency_links to torchrl.egg-info/dependency_links.txt
writing requirements to torchrl.egg-info/requires.txt
writing top-level names to torchrl.egg-info/top_level.txt
/home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/torch/utils/cpp_extension.py:476: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
  warnings.warn(msg.format('we could not find ninja.'))
reading manifest file 'torchrl.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'torchrl.egg-info/SOURCES.txt'
running build_ext
copying build/lib.linux-x86_64-3.9/torchrl/_torchrl.so -> torchrl
Creating /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/torchrl.egg-link (link to .)
Adding torchrl 0.1.1+dc573fd to easy-install.pth file

Installed /home/sakakibara/project/rl
Processing dependencies for torchrl==0.1.1+dc573fd
Searching for tensordict>=0.1.1
Reading https://pypi.org/simple/tensordict/
Downloading https://files.pythonhosted.org/packages/dd/b1/06de15b9c3a38b4f133fbb4e2d6228ad22f075fdbc61d0174f31ebb7e227/tensordict-0.1.2-py3-none-any.whl#sha256=09723530487cc223a15baeebcf5bc15f7be7027849386f4c36c8df8df9a6de5e
Best match: tensordict 0.1.2
Processing tensordict-0.1.2-py3-none-any.whl
Installing tensordict-0.1.2-py3-none-any.whl to /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Adding tensordict 0.1.2 to easy-install.pth file

Installed /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/tensordict-0.1.2-py3.9.egg
Searching for packaging
Reading https://pypi.org/simple/packaging/
Downloading https://files.pythonhosted.org/packages/ab/c3/57f0601a2d4fe15de7a553c00adbc901425661bf048f2a22dfc500caf121/packaging-23.1-py3-none-any.whl#sha256=994793af429502c4ea2ebf6bf664629d07c1a9fe974af92966e4b8d2df7edc61
Best match: packaging 23.1
Processing packaging-23.1-py3-none-any.whl
Installing packaging-23.1-py3-none-any.whl to /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Adding packaging 23.1 to easy-install.pth file

Installed /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/packaging-23.1-py3.9.egg
Searching for cloudpickle==2.2.1
Best match: cloudpickle 2.2.1
Adding cloudpickle 2.2.1 to easy-install.pth file

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for numpy==1.25.2
Best match: numpy 1.25.2
Adding numpy 1.25.2 to easy-install.pth file
Installing f2py script to /home/sakakibara/project/rl/.rl/bin
Installing f2py3 script to /home/sakakibara/project/rl/.rl/bin
Installing f2py3.9 script to /home/sakakibara/project/rl/.rl/bin

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for torch==2.0.1
Best match: torch 2.0.1
Adding torch 2.0.1 to easy-install.pth file
Installing convert-caffe2-to-onnx script to /home/sakakibara/project/rl/.rl/bin
Installing convert-onnx-to-caffe2 script to /home/sakakibara/project/rl/.rl/bin
Installing torchrun script to /home/sakakibara/project/rl/.rl/bin

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for nvidia-curand-cu11==10.2.10.91
Best match: nvidia-curand-cu11 10.2.10.91
Adding nvidia-curand-cu11 10.2.10.91 to easy-install.pth file

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for nvidia-cusparse-cu11==11.7.4.91
Best match: nvidia-cusparse-cu11 11.7.4.91
Adding nvidia-cusparse-cu11 11.7.4.91 to easy-install.pth file

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for filelock==3.12.3
Best match: filelock 3.12.3
Adding filelock 3.12.3 to easy-install.pth file

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for triton==2.0.0
Best match: triton 2.0.0
Adding triton 2.0.0 to easy-install.pth file

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for nvidia-nvtx-cu11==11.7.91
Best match: nvidia-nvtx-cu11 11.7.91
Adding nvidia-nvtx-cu11 11.7.91 to easy-install.pth file

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for nvidia-nccl-cu11==2.14.3
Best match: nvidia-nccl-cu11 2.14.3
Adding nvidia-nccl-cu11 2.14.3 to easy-install.pth file

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for sympy==1.12
Best match: sympy 1.12
Adding sympy 1.12 to easy-install.pth file
Installing isympy script to /home/sakakibara/project/rl/.rl/bin

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for nvidia-cufft-cu11==10.9.0.58
Best match: nvidia-cufft-cu11 10.9.0.58
Adding nvidia-cufft-cu11 10.9.0.58 to easy-install.pth file

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for nvidia-cudnn-cu11==8.5.0.96
Best match: nvidia-cudnn-cu11 8.5.0.96
Adding nvidia-cudnn-cu11 8.5.0.96 to easy-install.pth file

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for Jinja2==3.1.2
Best match: Jinja2 3.1.2
Adding Jinja2 3.1.2 to easy-install.pth file

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for networkx==3.1
Best match: networkx 3.1
Adding networkx 3.1 to easy-install.pth file

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for typing-extensions==4.7.1
Best match: typing-extensions 4.7.1
Adding typing-extensions 4.7.1 to easy-install.pth file

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for nvidia-cublas-cu11==11.10.3.66
Best match: nvidia-cublas-cu11 11.10.3.66
Adding nvidia-cublas-cu11 11.10.3.66 to easy-install.pth file

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for nvidia-cuda-runtime-cu11==11.7.99
Best match: nvidia-cuda-runtime-cu11 11.7.99
Adding nvidia-cuda-runtime-cu11 11.7.99 to easy-install.pth file

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for nvidia-cuda-nvrtc-cu11==11.7.99
Best match: nvidia-cuda-nvrtc-cu11 11.7.99
Adding nvidia-cuda-nvrtc-cu11 11.7.99 to easy-install.pth file

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for nvidia-cuda-cupti-cu11==11.7.101
Best match: nvidia-cuda-cupti-cu11 11.7.101
Adding nvidia-cuda-cupti-cu11 11.7.101 to easy-install.pth file

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for nvidia-cusolver-cu11==11.4.0.1
Best match: nvidia-cusolver-cu11 11.4.0.1
Adding nvidia-cusolver-cu11 11.4.0.1 to easy-install.pth file

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for wheel==0.41.2
Best match: wheel 0.41.2
Adding wheel 0.41.2 to easy-install.pth file
Installing wheel script to /home/sakakibara/project/rl/.rl/bin

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for setuptools==58.1.0
Best match: setuptools 58.1.0
Adding setuptools 58.1.0 to easy-install.pth file

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for lit==16.0.6
Best match: lit 16.0.6
Adding lit 16.0.6 to easy-install.pth file
Installing lit script to /home/sakakibara/project/rl/.rl/bin

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for cmake==3.27.4.1
Best match: cmake 3.27.4.1
Adding cmake 3.27.4.1 to easy-install.pth file
Installing cmake script to /home/sakakibara/project/rl/.rl/bin
Installing cpack script to /home/sakakibara/project/rl/.rl/bin
Installing ctest script to /home/sakakibara/project/rl/.rl/bin

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for mpmath==1.3.0
Best match: mpmath 1.3.0
Adding mpmath 1.3.0 to easy-install.pth file

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Searching for MarkupSafe==2.1.3
Best match: MarkupSafe 2.1.3
Adding MarkupSafe 2.1.3 to easy-install.pth file

Using /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
Finished processing dependencies for torchrl==0.1.1+dc573fd
sakakibara-yuuki commented 10 months ago

https://qiita.com/ledmonster/items/74886ac2f51fb2e5fd64 の記事を見てわかった

pip install -e .[utils]

だ..

pip list | grep wandb

と思ったら失敗した

Obtaining file:///home/sakakibara/project/rl
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: torch in ./.rl/lib/python3.9/site-packages (from torchrl==0.1.1+dc573fd) (2.0.1)
Requirement already satisfied: numpy in ./.rl/lib/python3.9/site-packages (from torchrl==0.1.1+dc573fd) (1.25.2)
Requirement already satisfied: packaging in ./.rl/lib/python3.9/site-packages/packaging-23.1-py3.9.egg (from torchrl==0.1.1+dc573fd) (23.1)
Requirement already satisfied: cloudpickle in ./.rl/lib/python3.9/site-packages (from torchrl==0.1.1+dc573fd) (2.2.1)
Requirement already satisfied: tensordict>=0.1.1 in ./.rl/lib/python3.9/site-packages/tensordict-0.1.2-py3.9.egg (from torchrl==0.1.1+dc573fd) (0.1.2)
Collecting tensorboard (from torchrl==0.1.1+dc573fd)
  Obtaining dependency information for tensorboard from https://files.pythonhosted.org/packages/bc/a2/ff5f4c299eb37c95299a76015da3f30211468e29d8d6f1d011683279baee/tensorboard-2.14.0-py3-none-any.whl.metadata
  Using cached tensorboard-2.14.0-py3-none-any.whl.metadata (1.8 kB)
Collecting wandb (from torchrl==0.1.1+dc573fd)
  Obtaining dependency information for wandb from https://files.pythonhosted.org/packages/fe/10/18b03623c460fd433525d9b4739af58c5e69f5974328dcdd037cfbc855d7/wandb-0.15.10-py3-none-any.whl.metadata
  Using cached wandb-0.15.10-py3-none-any.whl.metadata (9.6 kB)
Collecting tqdm (from torchrl==0.1.1+dc573fd)
  Obtaining dependency information for tqdm from https://files.pythonhosted.org/packages/00/e5/f12a80907d0884e6dff9c16d0c0114d81b8cd07dc3ae54c5e962cc83037e/tqdm-4.66.1-py3-none-any.whl.metadata
  Using cached tqdm-4.66.1-py3-none-any.whl.metadata (57 kB)
Collecting hydra-core>=1.1 (from torchrl==0.1.1+dc573fd)
  Using cached hydra_core-1.3.2-py3-none-any.whl (154 kB)
Collecting hydra-submitit-launcher (from torchrl==0.1.1+dc573fd)
  Downloading hydra_submitit_launcher-1.2.0-py3-none-any.whl (5.2 kB)
INFO: pip is looking at multiple versions of torchrl[utils] to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement git; extra == "utils" (from torchrl[utils]) (from versions: none)
ERROR: No matching distribution found for git; extra == "utils"
sakakibara-yuuki commented 10 months ago

utilsというディスとりがないからinstall できないって言われてる気がす

pip install torchrl[utils]

result:

pip list | grep wandb
wandb                    0.15.10

やっとこさ...

sakakibara-yuuki commented 10 months ago

念願の

python sac.py

result:

Traceback (most recent call last):
  File "/home/sakakibara/project/rl/examples/sac/sac.py", line 21, in <module>
    from torchrl.envs.utils import ExplorationType, set_exploration_type
  File "/home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/torchrl/__init__.py", line 32, in <module>
    import torchrl.collectors
  File "/home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/torchrl/collectors/__init__.py", line 6, in <module>
    from .collectors import (
  File "/home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/torchrl/collectors/collectors.py", line 40, in <module>
    from torchrl.envs.common import EnvBase
  File "/home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/torchrl/envs/__init__.py", line 10, in <module>
    from .transforms import (
  File "/home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/torchrl/envs/transforms/__init__.py", line 6, in <module>
    from .r3m import R3MTransform
  File "/home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/torchrl/envs/transforms/r3m.py", line 19, in <module>
    from torchrl.envs.transforms.transforms import (
  File "/home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/torchrl/envs/transforms/transforms.py", line 17, in <module>
    from tensordict.nn.common import _seq_of_nested_key_check
ImportError: cannot import name '_seq_of_nested_key_check' from 'tensordict.nn.common' (/home/sakakibara/project/rl/.rl/lib/python3.9/site-packages/tensordict/nn/common.py)

....もしかしてpip install torchrl[utils]の時に変なのinstallしちゃったからlocalのtorchrlやtensordictが正しく認識されなくなった　?

sakakibara-yuuki commented 10 months ago 
pip list

すると

tensordict               0.1.2
tensordict-nightly       2023.9.10
torch                    2.0.1
torchrl                  0.1.1      /home/sakakibara/project/rl/.rl/lib/python3.9/site-packages
sakakibara-yuuki commented 10 months ago

https://github.com/sakakibara-yuuki/rl/issues/3#issuecomment-1713220865 について

git の versionが低いと入らない？

現在:

git/jammy-updates,now 1:2.34.1-1ubuntu1.10 amd64 [installed]

update:

sudo add-apt-repository ppa:git-core/ppa
sudo apt update
sudo apt upgrade
git --version
> git version 2.42.0

ref: https://qiita.com/cointoss1973/items/1c01837e65b937fc0761

sakakibara-yuuki commented 10 months ago

いったんすべて消してやりなおし

pip freeze > hoge && pip uninstall -y -r hoge && rm hoge
pip install tensordict-nightly
python setup.py develop
pip install -e .[utils]

results:

Obtaining file:///home/sakakibara/project/rl
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: torch in ./.rl/lib/python3.9/site-packages (from torchrl==0.1.1+dc573fd) (2.0.1)
Requirement already satisfied: numpy in ./.rl/lib/python3.9/site-packages (from torchrl==0.1.1+dc573fd) (1.25.2)
Requirement already satisfied: packaging in ./.rl/lib/python3.9/site-packages/packaging-23.1-py3.9.egg (from torchrl==0.1.1+dc573fd) (23.1)
Requirement already satisfied: cloudpickle in ./.rl/lib/python3.9/site-packages (from torchrl==0.1.1+dc573fd) (2.2.1)
Requirement already satisfied: tensordict>=0.1.1 in ./.rl/lib/python3.9/site-packages/tensordict-0.1.2-py3.9.egg (from torchrl==0.1.1+dc573fd) (0.1.2)
Collecting tensorboard (from torchrl==0.1.1+dc573fd)
  Obtaining dependency information for tensorboard from https://files.pythonhosted.org/packages/bc/a2/ff5f4c299eb37c95299a76015da3f30211468e29d8d6f1d011683279baee/tensorboard-2.14.0-py3-none-any.whl.metadata
  Using cached tensorboard-2.14.0-py3-none-any.whl.metadata (1.8 kB)
Collecting wandb (from torchrl==0.1.1+dc573fd)
  Obtaining dependency information for wandb from https://files.pythonhosted.org/packages/fe/10/18b03623c460fd433525d9b4739af58c5e69f5974328dcdd037cfbc855d7/wandb-0.15.10-py3-none-any.whl.metadata
  Using cached wandb-0.15.10-py3-none-any.whl.metadata (9.6 kB)
Collecting tqdm (from torchrl==0.1.1+dc573fd)
  Obtaining dependency information for tqdm from https://files.pythonhosted.org/packages/00/e5/f12a80907d0884e6dff9c16d0c0114d81b8cd07dc3ae54c5e962cc83037e/tqdm-4.66.1-py3-none-any.whl.metadata
  Using cached tqdm-4.66.1-py3-none-any.whl.metadata (57 kB)
Collecting hydra-core>=1.1 (from torchrl==0.1.1+dc573fd)
  Using cached hydra_core-1.3.2-py3-none-any.whl (154 kB)
Collecting hydra-submitit-launcher (from torchrl==0.1.1+dc573fd)
  Using cached hydra_submitit_launcher-1.2.0-py3-none-any.whl (5.2 kB)
INFO: pip is looking at multiple versions of torchrl[utils] to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement git; extra == "utils" (from torchrl[utils]) (from versions: none)
ERROR: No matching distribution found for git; extra == "utils"

hahaha

sakakibara-yuuki commented 10 months ago 
pip install -e .["tests"]

はうまくいった。いみわからん。

sakakibara-yuuki commented 10 months ago 
setup.py
のgitをコメントアウトしてみる
pip install -e .["utils"]

うまくいった