[RLlib] Silence external warnings

[stefanbschneider]

Description

It would be great to silence warnings from external libraries inside Ray, eg, from numpy.

Here, warnings.filterwarnings("ignore", category=DeprecationWarning) does not work and the warnings from Ray are still shown. Also, ray.init(log_to_driver=False) or reducing verbosity do not help.

Discussion and suggested approach on Discourse: https://discuss.ray.io/t/silence-numpy-deprecation-warnings/5705

Use case

I am trying to use Ray 1.8 with Numpy 1.20+ and my logs are being flooded with deprecation warnings, e.g., DeprecationWarning:np.objectis a deprecated alias for the builtinobject. To silence this warning, useobjectby itself. Doing this will not modify any behavior and is safe.

While I do plan upgrading Ray in the mid/long-term, I can't do it right now. I also can't downgrade Numpy. And with the current excessive logging, my CI pipelines fail because the logs overflow. So it would be great if there's an option to silence such external warnings through Ray.

[sven1977]

Hey @stefanbschneider , thanks for raising this issue. We actually don't use np.object anymore in RLlib, but I understand this issue isn't really about that, but rather a better way to generall silence those warnings in the future.

[cool-RR]

This is especially needed because the scientific packages in Python are always built on top of each other with lots of problems between the dependencies. For example, I just installed the latest version of rllib, and I'm getting several different warnings:

/home/ramrachum/.venvs/ray_env/lib/python3.10/site-packages/gym/wrappers/monitoring/video_recorder.py:9: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  import distutils.spawn
/home/ramrachum/.venvs/ray_env/lib/python3.10/site-packages/ray/_private/ray_option_utils.py:266: DeprecationWarning: Setting 'object_store_memory' for actors is deprecated since it doesn't actually reserve the required object store memory. Use object spilling that's enabled by default (https://docs.ray.io/en/releases-2.0.0/ray-core/objects/object-spilling.html) instead to bypass the object store memory size limitation.
  warnings.warn(
(pid=510148) /home/ramrachum/.venvs/ray_env/lib/python3.10/site-packages/gym/wrappers/monitoring/video_recorder.py:9: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
(pid=510148)   import distutils.spawn

None of these deprecation warnings are something I have anything to do about as a user. For example, the one in Gym was probably already fixed, but RLlib (or one of its dependencies) specifies an older version of Gym. While package maintainers hash that out, I want to have a clean log.

[cool-RR]

[ArturNiederfahrenhorst]

You can use runtime environments to filter Python warnings. -> "env_vars": {"PYTHONWARNINGS: "ignore"} should do the job!

[cool-RR]

@ArturNiederfahrenhorst That does not do the job. Warnings are important and we should see them. Then we should be able to silence them one-by-one when we find that they're not because of our mistakes, but because of incompatibilities between third-party packages. It's important to silence specific warnings so when a relevant warning does come up, it won't be missed.

I wrote some hacky code that filters sys.stderr and removes a denylist of warnings, but I wish there was a less hacky solution.

[zriddle-eiq]

@cool-RR Is your hacky solution still the best one for this? If so, can you share part of it?

[cool-RR]

@zriddle-eiq I don't know of any other solution. Here you go: https://r.rachum.com/filtros

Call filtros.activate() before you import ray. You may want to edit some of ignored warnings.

By the way, I recently upgraded to Ray 2.6.1 and noticed that there are so many deprecation warnings that are caused by Ray itself! This is pretty crazy. It's impossible to use stock Ray without nothing else and not get deprecation warnings. I have no idea how other people are dealing with this. I imagine most of them just tolerate a huge torrent of deprecation warnings in their stderr that sometimes drowns out important information.

[smolboii]

@cool-RR thanks for that :). I ended up adding "WARNING deprecation.py:50 -- DeprecationWarning:" as that was common to all of my deprecation warnings.

[cool-RR]

@smolboii Sure but if you add that one, you might be ignoring actually useful deprecation warnings. That's why I prefer going over them one-by-one.

[smolboii]

@cool-RR yea true - but imo deprecated is just another word for supported :P

[stefanbschneider]

We just ran into this same issue again: We have a ton of deprecation warnings caused by another dependency that are flooding our logs. We can't resolve them right away and want to silence/suppress them for now.

Unfortunately, I still don't know of an easy way to do that with Ray. warnings.filterwarnings("ignore", category=DeprecationWarning) seems to be ignored by the workers.

[virtualluke]

We are seeing this as well except for strings which aren't raw throwing SyntaxWarnings and overloading ray (we don't have control of the strings to add a r"" so that isn't an option). This is "fixed" with a log_to_driver=False in the ray.init but this causes us to miss vital logs we are expecting. Any resolution other than the (log_to_driver=False) working for anyone?