[RLLib] Algorithm compute_actions does not work with Connector API enabled.

[n30111]

What happened + What you expected to happen

When running inference or computing actions using an instance of algorithm, algorithm.compute_actions, with connector API enabled, it throws error.

The same issue does not occur with config["enable_connectors"]=False

Traceback (most recent call last):
  File "rllib_ppo.py", line 54, in <module>
    test_ppo()
  File "rllib_ppo.py", line 50, in test_ppo
    outputs = algo.compute_actions({"1": obj})
  File "/lib/python3.8/site-packages/ray/rllib/algorithms/algorithm.py", line 1687, in compute_actions
    preprocessed = worker.preprocessors[policy_id].transform(ob)
AttributeError: 'NoneType' object has no attribute 'transform'

Versions / Dependencies

Ray=2.3

Reproduction script

import ray.rllib.algorithms.ppo as ppo
from ray.tune.logger import JsonLogger

import gymnasium as gym

from ray.tune.logger import UnifiedLogger

def test_ppo():
    """Test whether PPO can be built with all frameworks w/ connectors."""

    # Build a PPOConfig object.
    config = (
        ppo.PPOConfig().training(
            num_sgd_iter=2,
            train_batch_size=128,
        ).rollouts(
            num_rollout_workers=2,
            compress_observations=True,
            enable_connectors=True,
        )
        .framework("tf2"))  # For checking lr-schedule correctness.

    num_iterations = 1

    env ="LunarLanderContinuous-v2"
    algo = config.build(
        env=env,
        logger_creator=lambda config: UnifiedLogger(
            config, "/tmp/params_test", loggers=[JsonLogger]))
    for i in range(num_iterations):
        result = algo.train()
        # print(result)
        print(i)
    env_obj = gym.make(env)
    obj, _ = env_obj.reset()
    outputs = algo.compute_actions({"1": obj})
    print(outputs)
    algo.stop()

test_ppo()

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[n30111]

@hora-anyscale can you please share whats the conclusion here? Is it user error?

[hora-anyscale]

@ArturNiederfahrenhorst ^^^

[ArturNiederfahrenhorst]

Hi @n30111 Thanks for raising this I'm working on a solution.

[paehal]

@ArturNiederfahrenhorst

I am also getting an error regarding enable_connectors, and if I set enable_connectors=False, the error does not occur.

The error is different from this issue, but I am asking here because the content may be related. What is the background of enable_connectors=True being default after updating to Ray ver 2.3?

https://github.com/ray-project/ray/blob/f1b8bfd219a98da69a612558cbae32441d98ca60/rllib/algorithms/algorithm_config.py#L226

https://github.com/ray-project/ray/blob/58d262068908eec7799bffa4db71ecf80a438147/rllib/algorithms/algorithm_config.py#L297

[sven1977]

I see, I think we should simply deprecate these APIs altogether. Actions should be computed on individual Policies (old stack) or RLModules (new stack) only. These APIs are too leaky and not well defined (multi-agent vs preprocessor vs ...).

For now, as a quick workaround, could you simply use:

algo.compute_single_action(obs)

instead?

In case you need a certain policy (in a multi-agent case, you can also do):

algo.compute_single_action(obs, policy_id=[xyz])