search

pytest-dev / pytest-forked

extracted --boxed from pytest-xdist to ensure backward compat
MIT License
62 stars 21 forks source link

Missing warnings with pytest --forked option #63

Open kir0ul opened 2 years ago

kir0ul commented 2 years ago

Hi,

I found out in https://github.com/openai/gym/issues/2647 that the pytest --forked option seems to miss to report many warnings.

With the --forked option I get only 2 warnings:

❯ docker run gym-docker pytest --forked --import-mode=append
[...]
============================= test session starts ==============================
platform linux -- Python 3.10.2, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /usr/local/gym
plugins: forked-1.4.0
collected 623 items / 5 skipped / 618 selected
[...]
=============================== warnings summary ===============================
../lib/python3.10/site-packages/gym/wrappers/monitoring/video_recorder.py:9
  /usr/local/lib/python3.10/site-packages/gym/wrappers/monitoring/video_recorder.py:9: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
    import distutils.spawn

../lib/python3.10/site-packages/gym/spaces/box.py:78
  /usr/local/lib/python3.10/site-packages/gym/spaces/box.py:78: UserWarning: WARN: Box bound precision lowered by casting to float32
    logger.warn(f"Box bound precision lowered by casting to {self.dtype}")

-- Docs: https://docs.pytest.org/en/stable/warnings.html
================= 621 passed, 7 skipped, 2 warnings in 39.42s ==================

Now without the --forked option I get ~1500 warnings:

❯ docker run gym-docker pytest --import-mode=append
[...]
============================= test session starts ==============================
platform linux -- Python 3.10.2, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /usr/local/gym
plugins: forked-1.4.0
collected 623 items / 5 skipped / 618 selected
[...]
=============================== warnings summary ===============================
../lib/python3.10/site-packages/gym/wrappers/monitoring/video_recorder.py:9
  /usr/local/lib/python3.10/site-packages/gym/wrappers/monitoring/video_recorder.py:9: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
    import distutils.spawn

../lib/python3.10/site-packages/gym/spaces/box.py:78
tests/wrappers/test_normalize.py::test_normalize_observation[False]
tests/wrappers/test_normalize.py::test_normalize_observation[True]
tests/wrappers/test_normalize.py::test_normalize_reset_info
tests/wrappers/test_normalize.py::test_normalize_return
tests/wrappers/test_normalize.py::test_normalize_observation_vector_env
tests/wrappers/test_normalize.py::test_normalize_return_vector_env
  /usr/local/lib/python3.10/site-packages/gym/spaces/box.py:78: UserWarning: WARN: Box bound precision lowered by casting to float32
    logger.warn(f"Box bound precision lowered by casting to {self.dtype}")

tests/envs/test_determinism.py: 80 warnings
tests/envs/test_envs.py: 137 warnings
tests/spaces/test_spaces.py: 266 warnings
tests/spaces/test_utils.py: 176 warnings
tests/utils/test_env_checker.py: 2 warnings
tests/vector/test_async_vector_env.py: 16 warnings
tests/vector/test_numpy_utils.py: 64 warnings
tests/vector/test_shared_memory.py: 128 warnings
tests/vector/test_sync_vector_env.py: 8 warnings
tests/wrappers/test_normalize.py: 6 warnings
tests/wrappers/test_record_episode_statistics.py: 181 warnings
tests/wrappers/test_record_video.py: 398 warnings
tests/wrappers/test_time_aware_observation.py: 2 warnings
tests/wrappers/test_transform_observation.py: 1 warning
tests/wrappers/test_transform_reward.py: 14 warnings
  /usr/local/lib/python3.10/site-packages/gym/utils/seeding.py:47: DeprecationWarning: WARN: Function `rng.randint(low, [high, size, dtype])` is marked as deprecated and will be removed in the future. Please use `rng.integers(low, [high, size, dtype])` instead.
    deprecation(

tests/envs/test_envs.py::test_env[spec1]
tests/envs/test_envs.py::test_env[spec2]
tests/envs/test_envs.py::test_env[spec10]
tests/envs/test_envs.py::test_env[spec11]
  /usr/local/lib/python3.10/site-packages/gym/utils/env_checker.py:116: UserWarning: WARN: Agent's minimum observation space value is -infinity. This is probably too low.
    logger.warn(

tests/envs/test_envs.py::test_env[spec1]
tests/envs/test_envs.py::test_env[spec2]
tests/envs/test_envs.py::test_env[spec10]
tests/envs/test_envs.py::test_env[spec11]
  /usr/local/lib/python3.10/site-packages/gym/utils/env_checker.py:120: UserWarning: WARN: Agent's maxmimum observation space value is infinity. This is probably too high
    logger.warn(

tests/envs/test_envs.py::test_env[spec4]
tests/envs/test_envs.py::test_env[spec14]
  /usr/local/lib/python3.10/site-packages/gym/utils/env_checker.py:162: UserWarning: WARN: We recommend you to use a symmetric and normalized Box action space (range=[-1, 1]) cf https://stable-baselines3.readthedocs.io/en/master/guide/rl_tips.html
    logger.warn(

tests/envs/test_envs.py::test_env[spec14]
tests/envs/test_envs.py::test_env[spec14]
tests/envs/test_envs.py::test_env[spec14]
tests/envs/test_envs.py::test_env[spec14]
  /usr/local/lib/python3.10/site-packages/gym/envs/classic_control/pendulum.py:183: DeprecationWarning: In future, it will be an error for 'np.bool_' scalars to be interpreted as an index
    scale_img = pygame.transform.flip(scale_img, is_flip, True)

tests/spaces/test_spaces.py::test_box_dtype_check
tests/vector/test_spaces.py::test_iterate[Box0]
tests/vector/test_spaces.py::test_iterate[Box4]
tests/vector/test_spaces.py::test_iterate[Dict1]
tests/vector/test_spaces.py::test_iterate_custom_space[Tuple]
  /usr/local/lib/python3.10/site-packages/gym/spaces/box.py:153: UserWarning: WARN: Casting input x to numpy array.    logger.warn("Casting input x to numpy array.")

tests/vector/test_async_vector_env.py::test_custom_space_async_vector_env_shared_memory
  /usr/local/lib/python3.10/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning: Exception ignored in: <function AsyncVectorEnv.__del__ at 0x7f0366be9ea0>

  Traceback (most recent call last):
    File "/usr/local/lib/python3.10/site-packages/gym/vector/async_vector_env.py", line 631, in __del__
      self.close(terminate=True)
    File "/usr/local/lib/python3.10/site-packages/gym/vector/vector_env.py", line 193, in close
      self.close_extras(**kwargs)
    File "/usr/local/lib/python3.10/site-packages/gym/vector/async_vector_env.py", line 543, in close_extras
      if self._state != AsyncState.DEFAULT:
  AttributeError: 'AsyncVectorEnv' object has no attribute '_state'

    warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))

tests/wrappers/test_record_video.py::test_record_video_reset_return_info
  /usr/local/lib/python3.10/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning: Exception ignored in: <function VideoRecorder.__del__ at 0x7f0366c06b90>

  Traceback (most recent call last):
    File "/usr/local/lib/python3.10/site-packages/gym/wrappers/monitoring/video_recorder.py", line 193, in __del__
      self.close()
    File "/usr/local/lib/python3.10/site-packages/gym/wrappers/monitoring/video_recorder.py", line 182, in close
      self.write_metadata()
    File "/usr/local/lib/python3.10/site-packages/gym/wrappers/monitoring/video_recorder.py", line 188, in write_metadata
      with open(self.metadata_path, "w") as f:
  FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/gym/videos/rl-video-step-0.meta.json'

    warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))

tests/wrappers/test_video_recorder.py::test_record_breaking_render_method
  /usr/local/lib/python3.10/site-packages/gym/wrappers/monitoring/video_recorder.py:136: UserWarning: WARN: Env returned None on render(). Disabling further rendering for video recorder by marking as disabled: path=/tmp/tmpqrycjchl.mp4 metadata_path=/tmp/tmpqrycjchl.meta.json
    logger.warn(

-- Docs: https://docs.pytest.org/en/stable/warnings.html
================ 621 passed, 7 skipped, 1509 warnings in 21.03s ================

Steps to reproduce:

git clone git@github.com:openai/gym.git
cd gym
wget https://www.roboti.us/file/mjkey.txt
docker build -f py.Dockerfile --build-arg MUJOCO_KEY=./mjkey.txt --build-arg PYTHON_VERSION=3.10 --tag gym-docker .

Then run either docker run gym-docker pytest --forked --import-mode=append or docker run gym-docker pytest --import-mode=append

Let me know if I can do anything else to help troubleshoot!

RonnyPfannschmidt commented 2 years ago

This is unlikely to be fixed unless someone takes over maintenance