Assets not pulled yet. Waiting for 10 seconds... forever

jakethesnake420 commented 7 months ago

Was working yesterday if I install with pip install metadrive-simulator and run: python -m metadrive.examples.profile_metadrive

i get:

Start to profile the efficiency of MetaDrive with 1000 maps and ~4 vehicles!
[INFO] Environment: MetaDriveEnv
[INFO] MetaDrive version: 0.4.2.2
[INFO] Sensors: [lidar: Lidar(), side_detector: SideDetector(), lane_line_detector: LaneLineDetector()]
[INFO] Render Mode: none
[INFO] Horizon (Max steps per agent): None
[INFO] Another instance of this program is already running. Wait for the asset pulling finished from another program...
[INFO] Assets not pulled yet. Waiting for 10 seconds...
[INFO] Assets not pulled yet. Waiting for 10 seconds...
[INFO] Assets not pulled yet. Waiting for 10 seconds...

this repeats forever. I know its not my machine because github action tests that worked yesterday are failing today.

if i install master from git and run it i get:

/home/user/.pyenv/versions/3.11.4/lib/python3.11/site-packages/torch/cuda/__init__.py:141: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() > 0
Start to profile the efficiency of MetaDrive with 1000 maps and ~4 vehicles!
[INFO] Environment: MetaDriveEnv
[INFO] MetaDrive version: 0.4.2.2
[INFO] Sensors: [lidar: Lidar(), side_detector: SideDetector(), lane_line_detector: LaneLineDetector()]
[INFO] Render Mode: none
[INFO] Horizon (Max steps per agent): None
[WARNING] Assets folder doesn't exist. Begin to download assets... (base_engine.py:753)
[INFO] Pull assets from https://github.com/metadriverse/metadrive/releases/download/MetaDrive-0.4.2.2/assets.zip to /home/user/Projects/metadrive/metadrive/assets.zip
Traceback (most recent call last):                                                                                                                         |
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/user/Projects/metadrive/metadrive/examples/profile_metadrive.py", line 17, in <module>
    obs, _ = env.reset()
             ^^^^^^^^^^^
  File "/home/user/Projects/metadrive/metadrive/envs/base_env.py", line 519, in reset
    self.lazy_init()  # it only works the first time when reset() is called to avoid the error when render
    ^^^^^^^^^^^^^^^^
  File "/home/user/Projects/metadrive/metadrive/envs/base_env.py", line 406, in lazy_init
    initialize_engine(self.config)
  File "/home/user/Projects/metadrive/metadrive/engine/engine_utils.py", line 12, in initialize_engine
    cls.singleton = cls(env_global_config)
                    ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/Projects/metadrive/metadrive/engine/base_engine.py", line 54, in __init__
    self.try_pull_asset()
  File "/home/user/Projects/metadrive/metadrive/engine/base_engine.py", line 754, in try_pull_asset
    pull_asset(update=False)
  File "/home/user/Projects/metadrive/metadrive/pull_asset.py", line 82, in pull_asset
    urllib.request.urlretrieve(ASSET_URL, zip_path, *extra_arg)
  File "/home/user/.pyenv/versions/3.11.4/lib/python3.11/urllib/request.py", line 270, in urlretrieve
    block = fp.read(bs)
            ^^^^^^^^^^^
  File "/home/user/.pyenv/versions/3.11.4/lib/python3.11/http/client.py", line 466, in read
    s = self.fp.read(amt)
        ^^^^^^^^^^^^^^^^^
  File "/home/user/.pyenv/versions/3.11.4/lib/python3.11/socket.py", line 706, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.pyenv/versions/3.11.4/lib/python3.11/ssl.py", line 1269, in recv_into
    self._checkClosed()
  File "/home/user/.pyenv/versions/3.11.4/lib/python3.11/ssl.py", line 1116, in _checkClosed
    raise Exception("Lost Connection")
Exception: Lost Connection

QuanyiLi commented 7 months ago

@pengzhenghao Could it be raised due to the new lock system?

jakethesnake420 commented 7 months ago

import unittest
import filelock
import os
import time
import urllib.request
import zipfile
from unittest.mock import patch
import requests
from requests.exceptions import RequestException
import traceback

VERSION = "0.4.2.2"
class TestFileLockWithRealDownload(unittest.TestCase):
    def setUp(self):
        self.zip_lock = 'test.lock'
        self.zip_path = 'test.zip'
        self.ASSET_URL = "https://github.com/metadriverse/metadrive/releases/download/MetaDrive-{}/assets.zip".format(VERSION)  # Replace with your version
        self.TARGET_DIR = 'test_extraction_dir'

        # Create target directory if it doesn't exist
        if not os.path.exists(self.TARGET_DIR):
            os.makedirs(self.TARGET_DIR)

    def tearDown(self):
        # Cleanup downloaded and extracted files
        if os.path.exists(self.zip_path):
            os.remove(self.zip_path)
        if os.path.exists(self.zip_lock):
            os.remove(self.zip_lock)

    def test_file_lock_with_real_download_fails(self):
        lock = filelock.FileLock(self.zip_lock, timeout=1)
        try:
            with lock.acquire():
                print("Pull assets from {} to {}".format(self.ASSET_URL, self.zip_path))
                try:
                    urllib.request.urlretrieve(self.ASSET_URL, self.zip_path)
                except Exception as e:
                    print("Error during file download: {}".format(e))
                    traceback.print_exc()  # This will print the stack trace
                with zipfile.ZipFile(self.zip_path, 'r') as zip_ref:
                    zip_ref.extractall(self.TARGET_DIR)
                print("Successfully downloaded and extracted assets")

        except filelock.Timeout:
            print("Another instance of this program is already running...")
            while os.path.exists(self.zip_lock):
                print("Assets not pulled yet. Waiting for 10 seconds...")
                time.sleep(10)

            print("Assets are now available.")

    def test_file_lock_with_real_download_works(self):
        lock = filelock.FileLock(self.zip_lock, timeout=1)
        try:
            with lock.acquire():
                # Simulate logger
                print("Pull assets from {} to {}".format(self.ASSET_URL, self.zip_path))

                # Implementing retry mechanism
                max_retries = 3
                for _ in range(max_retries):
                    try:
                        response = requests.get(self.ASSET_URL, stream=True, timeout=10)
                        with open(self.zip_path, 'wb') as f:
                            for chunk in response.iter_content(chunk_size=8192):
                                f.write(chunk)
                        break
                    except RequestException as e:
                        print(f"Failed to download file: {e}")
                        time.sleep(5)  # wait before retrying
                else:
                    raise Exception("Failed to download file after multiple attempts")

                with zipfile.ZipFile(self.zip_path, 'r') as zip_ref:
                    zip_ref.extractall(self.TARGET_DIR)
                print("Successfully downloaded and extracted assets")

        except filelock.Timeout:
            print("Another instance of this program is already running. Wait for the asset pulling finished from another program...")
            while os.path.exists(self.zip_lock):
                print("Assets not pulled yet. Waiting for 10 seconds...")
                time.sleep(10)

            print("Assets are now available.")

if __name__ == '__main__':
    unittest.main()

pengzhenghao commented 7 months ago

Hi did you install metadrive via git clone or pip install metadrive-simulator? It should be fixed by #607

Thanks for reminding me maybe I should create a new release so the user install via pip install can avoid this issue.

pengzhenghao commented 7 months ago

Now I've made a new release #622

Could you try update MD and see if that works?

pengzhenghao commented 7 months ago

@jakethesnake420 I ran your code it works well with me

jakethesnake420 commented 7 months ago

@jakethesnake420 I ran your code it works well with me

Interesting, I get the Exception: Lost Connection when i run test_file_lock_with_real_download_fails(). It must be some dependency issue in my openpilot enviroment on github actions and my local PC. I will test your latest fixes.

Hi did you install metadrive via git clone or pip install metadrive-simulator

I tried both pip package and git master installations.

jakethesnake420 commented 7 months ago

Tried the latest pip version 0.4.2.3 .

Just doing pip install metadrive-simulator did not fix the issue. I had to do pip-autoremove metadrive-simulator and then pip install metadrive-simulator --no-cache-dir to make it work.

jakethesnake420 commented 7 months ago

So actually when I install it in openpilot, i still get the same Exception: Lost Connection even after updating poetry and pip-autoremove metadrive-simulator and then pip install metadrive-simulator --no-cache-dir

The only way its workingis if i manually download and unzip.

I figure there is some dependency that's not updating due to another package using it in openpilot and its causing this.

These are my dependency versions for metadrive-simulator

pipdeptree -p metadrive-simulator

Warning!! Cyclic dependencies found:
* poetry-plugin-export => poetry => poetry-plugin-export
* poetry => poetry-plugin-export => poetry
------------------------------------------------------------------------
metadrive-simulator==0.4.2.3
├── filelock [required: Any, installed: 3.13.1]
├── geopandas [required: Any, installed: 0.14.2]
│   ├── fiona [required: >=1.8.21, installed: 1.9.5]
│   │   ├── attrs [required: >=19.2.0, installed: 23.1.0]
│   │   ├── certifi [required: Any, installed: 2023.11.17]
│   │   ├── click [required: ~=8.0, installed: 8.1.7]
│   │   ├── click-plugins [required: >=1.0, installed: 1.1.1]
│   │   │   └── click [required: >=4.0, installed: 8.1.7]
│   │   ├── cligj [required: >=0.5, installed: 0.7.2]
│   │   │   └── click [required: >=4.0, installed: 8.1.7]
│   │   ├── setuptools [required: Any, installed: 65.5.0]
│   │   └── six [required: Any, installed: 1.16.0]
│   ├── packaging [required: Any, installed: 23.2]
│   ├── pandas [required: >=1.4.0, installed: 2.2.0]
│   │   ├── numpy [required: >=1.23.2,<2, installed: 1.24.2]
│   │   ├── python-dateutil [required: >=2.8.2, installed: 2.8.2]
│   │   │   └── six [required: >=1.5, installed: 1.16.0]
│   │   ├── pytz [required: >=2020.1, installed: 2023.3.post1]
│   │   └── tzdata [required: >=2022.7, installed: 2023.4]
│   ├── pyproj [required: >=3.3.0, installed: 3.6.1]
│   │   └── certifi [required: Any, installed: 2023.11.17]
│   └── shapely [required: >=1.8.0, installed: 2.0.2]
│       └── numpy [required: >=1.14, installed: 1.24.2]
├── gymnasium [required: >=0.28,<0.29, installed: 0.28.1]
│   ├── cloudpickle [required: >=1.2.0, installed: 3.0.0]
│   ├── Farama-Notifications [required: >=0.0.1, installed: 0.0.4]
│   ├── jax-jumpy [required: >=1.0.0, installed: 1.0.0]
│   │   └── numpy [required: >=1.18.0, installed: 1.24.2]
│   ├── numpy [required: >=1.21.0, installed: 1.24.2]
│   └── typing-extensions [required: >=4.3.0, installed: 4.9.0]
├── lxml [required: Any, installed: 5.1.0]
├── matplotlib [required: Any, installed: 3.8.2]
│   ├── contourpy [required: >=1.0.1, installed: 1.2.0]
│   │   └── numpy [required: >=1.20,<2.0, installed: 1.24.2]
│   ├── cycler [required: >=0.10, installed: 0.12.1]
│   ├── fonttools [required: >=4.22.0, installed: 4.47.2]
│   ├── kiwisolver [required: >=1.3.1, installed: 1.4.5]
│   ├── numpy [required: >=1.21,<2, installed: 1.24.2]
│   ├── packaging [required: >=20.0, installed: 23.2]
│   ├── Pillow [required: >=8, installed: 10.1.0]
│   ├── pyparsing [required: >=2.3.1, installed: 3.1.1]
│   └── python-dateutil [required: >=2.7, installed: 2.8.2]
│       └── six [required: >=1.5, installed: 1.16.0]
├── numpy [required: >=1.21.6,<=1.24.2, installed: 1.24.2]
├── opencv-python [required: Any, installed: 4.9.0.80]
│   ├── numpy [required: >=1.21.2, installed: 1.24.2]
│   ├── numpy [required: >=1.23.5, installed: 1.24.2]
│   ├── numpy [required: >=1.17.0, installed: 1.24.2]
│   ├── numpy [required: >=1.17.3, installed: 1.24.2]
│   └── numpy [required: >=1.19.3, installed: 1.24.2]
├── panda3d [required: ==1.10.13, installed: 1.10.13]
├── panda3d-gltf [required: ==0.13, installed: 0.13]
│   ├── panda3d [required: >=1.10.8, installed: 1.10.13]
│   └── panda3d-simplepbr [required: >=0.6, installed: 0.11.2]
│       ├── panda3d [required: >=1.10.8, installed: 1.10.13]
│       └── typing-extensions [required: ~=4.7, installed: 4.9.0]
├── pandas [required: Any, installed: 2.2.0]
│   ├── numpy [required: >=1.23.2,<2, installed: 1.24.2]
│   ├── python-dateutil [required: >=2.8.2, installed: 2.8.2]
│   │   └── six [required: >=1.5, installed: 1.16.0]
│   ├── pytz [required: >=2020.1, installed: 2023.3.post1]
│   └── tzdata [required: >=2022.7, installed: 2023.4]
├── Pillow [required: Any, installed: 10.1.0]
├── progressbar [required: Any, installed: 2.5]
├── psutil [required: Any, installed: 5.9.7]
├── pygame [required: Any, installed: 2.5.2]
├── pygments [required: Any, installed: 2.17.2]
├── pytest [required: Any, installed: 7.4.4]
│   ├── iniconfig [required: Any, installed: 2.0.0]
│   ├── packaging [required: Any, installed: 23.2]
│   └── pluggy [required: >=0.12,<2.0, installed: 1.4.0]
├── requests [required: Any, installed: 2.31.0]
│   ├── certifi [required: >=2017.4.17, installed: 2023.11.17]
│   ├── charset-normalizer [required: >=2,<4, installed: 3.3.2]
│   ├── idna [required: >=2.5,<4, installed: 3.6]
│   └── urllib3 [required: >=1.21.1,<3, installed: 2.1.0]
├── scipy [required: Any, installed: 1.11.4]
│   └── numpy [required: >=1.21.6,<1.28.0, installed: 1.24.2]
├── seaborn [required: Any, installed: 0.13.2]
│   ├── matplotlib [required: >=3.4,!=3.6.1, installed: 3.8.2]
│   │   ├── contourpy [required: >=1.0.1, installed: 1.2.0]
│   │   │   └── numpy [required: >=1.20,<2.0, installed: 1.24.2]
│   │   ├── cycler [required: >=0.10, installed: 0.12.1]
│   │   ├── fonttools [required: >=4.22.0, installed: 4.47.2]
│   │   ├── kiwisolver [required: >=1.3.1, installed: 1.4.5]
│   │   ├── numpy [required: >=1.21,<2, installed: 1.24.2]
│   │   ├── packaging [required: >=20.0, installed: 23.2]
│   │   ├── Pillow [required: >=8, installed: 10.1.0]
│   │   ├── pyparsing [required: >=2.3.1, installed: 3.1.1]
│   │   └── python-dateutil [required: >=2.7, installed: 2.8.2]
│   │       └── six [required: >=1.5, installed: 1.16.0]
│   ├── numpy [required: >=1.20,!=1.24.0, installed: 1.24.2]
│   └── pandas [required: >=1.2, installed: 2.2.0]
│       ├── numpy [required: >=1.23.2,<2, installed: 1.24.2]
│       ├── python-dateutil [required: >=2.8.2, installed: 2.8.2]
│       │   └── six [required: >=1.5, installed: 1.16.0]
│       ├── pytz [required: >=2020.1, installed: 2023.3.post1]
│       └── tzdata [required: >=2022.7, installed: 2023.4]
├── shapely [required: Any, installed: 2.0.2]
│   └── numpy [required: >=1.14, installed: 1.24.2]
├── tqdm [required: Any, installed: 4.66.1]
└── yapf [required: Any, installed: 0.40.2]
    ├── importlib-metadata [required: >=6.6.0, installed: 7.0.0]
    │   └── zipp [required: >=0.5, installed: 3.17.0]
    ├── platformdirs [required: >=3.5.1, installed: 3.11.0]
    └── tomli [required: >=2.0.1, installed: 2.0.1]

metadriverse / metadrive

Assets not pulled yet. Waiting for 10 seconds... forever #621