Closed Kaszanas closed 1 month ago
@Kaszanas I think the blog post could have pointed to the full implementation here. Hopefully this has all the code you were missing.
Regarding the environment issue: Make sure to create a fresh virtual environment with virtualenv or conda. Make sure that, after activating the environment, which python
and which pip
returns the python executable from your environment, not the system python. You want to make sure that when you pip install lightning
, it installs it into your environment, not the system-wide packages. The warning message we print "INFO: Lightning is running from outside your current environment. Switching to your current environment." is basically complaining about that. It sees that you are in an environment but it recognizes that the python command runs from outside of it.
I will let you know later if I am able to fix it i create my virtual environments using python -m venv venv
, and activate it manually venv/Scripts/activate
In the environment dump you can see that all of the packages installed, and this doesn't take into consideration that pip install lightning
doesn't come with CUDA support out of the box torch with CUDA needs to be installed separately on top of that.
Additionally, the full implementation imports parts of code that are not exposed when installing lightning
. So these examples are not accessible for anyone that wouldn't like to clone the repository. Even when cloning the repository anyone that wants to work with the examples needs to get to know all of the lightning
repository setup. There's no good way to work with that in my opinion.
Okay, I went back to diagnose this problem.
lightning is in fact installed as an .exe file in my virtual environment (notice it at the bottom):
I am sure I am running with my virtual environment activated.
See the following:
(venv) PS G:\SomePath\SomeProject> .\venv\Scripts\lightning.exe run model --accelerator=gpu --strategy=ddp --devices=1 src/main.py --capture-video --env-id CartPole-v1 --total-timesteps 100000 --num-envs 2 --num-steps 512
INFO: Lightning is running from outside your current environment. Switching to your current environment.
The lightning package is not installed. Would you like to install it? [Y/n (exit)]:
If you need any more information, let me know.
Where does where lightning
, where pip
, where python
point to?
Result of the command (with paths anonymized) lightning:
G:\SomePath\SomeProject\venv\Scripts\lightning.exe
pip:
G:\SomePath\SomeProject\venv\Scripts\pip.exe
... Other paths
python
G:\SomePath\SomeProject\venv\Scripts\python.exe
... Other paths
In that regard the first path should take priority, that is why I did not include all of the other paths, and it seems that first returned path for my directory is in a correct place.
This looks good, so I am surprised. This must mean that
python -c "import sys; print(sys.executable)"
is probably returning a different path (to a different python?). Can you verify this?
And what happens if you put this line into an empty file test_import.py
import lightning
and run python test_import.py
?
Running the command:
python -c "import sys; print(sys.executable)"
Returns:
G:\SomePath\SomeProject\venv\Scripts\python.exe
Running a test script from any path:
python .\src\test_import.py
cd src
python test_import.py
finishes execution without errors. See that is why I was confused as to why the lightning
command from the CLI fails.
same problem, is there any solution? @awaelchli
The lightning
command no longer does this in recent versions. So the solution is to simply upgrade the lightning version. To run fabric scripts, simply run them with Python, or use the fabric run
command now.
I will try to verify if this is resolved.
Hopefully I can get it to work 😉
Bug description
This issue is apparent when attempting to run the following tutorial: https://lightning.ai/pages/community/tutorial/how-to-train-reinforcement-learning-model-to-play-game-using-proximal-policy-optimization-ppo-algorithm/ but might be a sign of a more serious issue where
lightning
command fails to run from within a virtual environment.Unfortunately the code snippets in the tutorial alone do not point towards a final running solution of the tutorial. I have had to reach into the repository code to find out what is happening. My replication of this tutorial is attached in the code section below.
When attempting to run the final tutorial command there are issues as highlighted below:
What version are you seeing the problem on?
v2.0
How to reproduce the bug
Naturally some util functions were also needed:
agent.py
file contents:Error messages and logs
Running the commands as highlighted above ends in the following error when running from a virtual environment with lightning installed, and torch cuda installed:
When trying to go through with installation, the lightning is installed (somewhere), and attempts to run the code but fails due to GPU (allegedly) not being available.
Environment
Current environment
* CUDA: - GPU: - NVIDIA GeForce GTX 1660 SUPER - available: True - version: 11.8 * Lightning: - lightning: 2.0.3 - lightning-cloud: 0.5.36 - lightning-utilities: 0.8.0 - pytorch-lightning: 2.0.3 - torch: 2.0.1+cu118 - torchaudio: 2.0.2+cu118 - torchmetrics: 0.11.4 - torchvision: 0.15.2+cu118 * Packages: - absl-py: 1.4.0 - aiohttp: 3.8.4 - aiosignal: 1.3.1 - ansicon: 1.89.0 - anyio: 3.7.0 - arrow: 1.2.3 - async-timeout: 4.0.2 - attrs: 23.1.0 - beautifulsoup4: 4.12.2 - black: 23.3.0 - blessed: 1.20.0 - box2d-py: 2.3.5 - cachetools: 5.3.1 - certifi: 2023.5.7 - charset-normalizer: 3.1.0 - click: 8.1.3 - cloudpickle: 2.2.1 - colorama: 0.4.6 - croniter: 1.3.15 - dateutils: 0.6.12 - decorator: 4.4.2 - deepdiff: 6.3.0 - farama-notifications: 0.0.4 - fastapi: 0.88.0 - filelock: 3.12.0 - frozenlist: 1.3.3 - fsspec: 2023.5.0 - google-auth: 2.19.1 - google-auth-oauthlib: 1.0.0 - grpcio: 1.54.2 - gymnasium: 0.28.1 - h11: 0.14.0 - idna: 3.4 - imageio: 2.31.0 - imageio-ffmpeg: 0.4.8 - inquirer: 3.1.3 - itsdangerous: 2.1.2 - jax-jumpy: 1.0.0 - jinja2: 3.1.2 - jinxed: 1.2.0 - lightning: 2.0.3 - lightning-cloud: 0.5.36 - lightning-utilities: 0.8.0 - markdown: 3.4.3 - markdown-it-py: 2.2.0 - markupsafe: 2.1.3 - mdurl: 0.1.2 - moviepy: 1.0.3 - mpmath: 1.3.0 - multidict: 6.0.4 - mypy-extensions: 1.0.0 - networkx: 3.1 - numpy: 1.24.3 - oauthlib: 3.2.2 - ordered-set: 4.1.0 - packaging: 23.1 - pathspec: 0.11.1 - pillow: 9.5.0 - pip: 22.3 - platformdirs: 3.5.1 - proglog: 0.1.10 - protobuf: 4.23.2 - psutil: 5.9.5 - pyasn1: 0.5.0 - pyasn1-modules: 0.3.0 - pydantic: 1.10.9 - pygame: 2.1.3 - pygments: 2.15.1 - pyjwt: 2.7.0 - python-dateutil: 2.8.2 - python-editor: 1.0.4 - python-multipart: 0.0.6 - pytorch-lightning: 2.0.3 - pytz: 2023.3 - pyyaml: 6.0 - readchar: 4.0.5 - requests: 2.31.0 - requests-oauthlib: 1.3.1 - rich: 13.4.1 - rsa: 4.9 - setuptools: 65.5.0 - six: 1.16.0 - sniffio: 1.3.0 - soupsieve: 2.4.1 - starlette: 0.22.0 - starsessions: 1.3.0 - swig: 4.1.1 - sympy: 1.12 - tensorboard: 2.13.0 - tensorboard-data-server: 0.7.0 - torch: 2.0.1+cu118 - torchaudio: 2.0.2+cu118 - torchmetrics: 0.11.4 - torchvision: 0.15.2+cu118 - tqdm: 4.65.0 - traitlets: 5.9.0 - typing-extensions: 4.6.3 - urllib3: 1.26.16 - uvicorn: 0.22.0 - wcwidth: 0.2.6 - websocket-client: 1.5.2 - websockets: 11.0.3 - werkzeug: 2.3.6 - wheel: 0.40.0 - yarl: 1.9.2 * System: - OS: Windows - architecture: - 64bit - WindowsPE - processor: AMD64 Family 23 Model 113 Stepping 0, AuthenticAMD - python: 3.11.0 - release: 10 - version: 10.0.19041More info
If possible it would be great to be able to run the examples locally from within a virtual environment.
cc @carmocca @justusschock @awaelchli