Catheter action for RL

[MarkWrobelUCL]

Hi, I am trying to use sofa-gym in order to train an agent to move a catheter from one area of a model heart to another. Previously I have been using imitation learning techniques which have been relatively successful. My scene is based on this scene in the beam adapter example. This uses 'InterventionalRadiologyController' to move the catheter. Now I am following the CartPole example and am unsure how to implement the class 'ApplyAction' for my purposes, in order to move the catheter. Any help would be appreciated. Thanks, Mark.

[samuelmyoussef]

Hello,

Your scene is probably similar to the CTR environment in SofaGym. You can look at the implementation here to help you, specifically the startCmd, start_cmd, and action_to_command functions. You don't need to explicitly implement the ApplyAction class but it helps with making your code more organized. The main difference between the CTR env and the scene you mentioned is that the CTR controls 3 instruments instead of one in the case of the single beam example.

We will also add a new example of a catheter soon, based on the 3instrument scene from beamadapter, which is similar to your scene but with 3 concentric tubes instead of a single one, so keep an eye on this.

[MarkWrobelUCL]

Thank you so much for the help @samuelmyoussef . I now have something working but I still have a few questions. In the CTREnv.py in line 61 there is the variable dim_state and I'm not quite sure what it means. I understand why there are 12 actions in this example, but not what dim_state represents. Also it appears the goal is randomized, does this only happen when you start to train i.e. it is fixed for the entirety of each training attempt?

Also do you happen to know what the variable xtip in the InterventionalRadiologyController represents?

Again many thanks for the help.

[samuelmyoussef]

dim_state represents the dimension of the observation or the state of the environment that is returned after each step. In this case it's 12 because the observation includes the xtips and rotations of the 3 instruments, the xyz position of the beam's tip, and the xyz position of the goal. You can see it in the code here returned by the getState method.

https://github.com/SofaDefrost/SofaGym/blob/main/sofagym/envs/CTR/CTRToolbox.py#L216

The goal is randomized for each episode not for the whole training attempt, so when an episode ends and the environment is reset, the goal will be different for the new episode.

For xtip, as given by the description of the variable it's the curvilinear abscissa of the tip of each interventional radiology instrument, so basically a measure of the curvature of each of the instruments of the catheter.

[MarkWrobelUCL]

Thanks for the explanations @samuelmyoussef . I now have something that trains well (it is quite a simple task). The catheter consistently gets to the goal position within the heart during training and is also using very few actions to get there. This sounds great however when I try to evaluate the trained agent, it just rotates at the start point which is not correct. Do you have any idea why the agent is very good at consistently hitting the maximum reward during training but then not when testing the trained agent? It might be useful to also mention that the simulation of the agent between epochs never hits the goal position either even though the agent is consistently hitting that same goal during the epoch. The goal i have set is the same across all epochs of training.

If you have any ideas about this issue I would be grateful. Thanks once more!

[samuelmyoussef]

We are aware of this issue and it's been fixed in this PR https://github.com/SofaDefrost/SofaGym/pull/45 but it hasn't been merged to the main branch yet. Until then, you can try to use my branch from the PR or apply the modifications manually to see if it works and give us your feedback.

[MarkWrobelUCL]

Thanks for letting me know. When I try to run any RL training (even the examples) with this branch I get a bunch of errors. Now I have tried with this version from Aug 28, the RL runs however, it gets stuck at 99/1000 during epoch 1 even for the example environments.

Please advise if I am doing something wrong. Thanks

[samuelmyoussef]

Can you post the errors log, please?

[MarkWrobelUCL]

Here is the error log for the up to date branch:

`[ERROR] [SofaRuntime] AssertionError: The render_mode must be 'rgb_array', not None Traceback (most recent call last): File "/home/mark/Documents/SofaGym-fix_model_load/rl.py", line 113, in agent.fit(total_timesteps) File "/home/mark/Documents/SofaGym-fix_model_load/agents/SB3Agent.py", line 309, in fit self.model.learn(total_timesteps=total_timesteps, reset_num_timesteps=False, File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/ppo/ppo.py", line 308, in learn return super().learn( File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/on_policy_algorithm.py", line 246, in learn total_timesteps, callback = self._setup_learn( File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/base_class.py", line 436, in _setup_learn callback = self._init_callback(callback, progress_bar) File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/base_class.py", line 380, in _init_callback callback.init_callback(self) File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 63, in init_callback self._init_callback() File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 194, in _init_callback callback.init_callback(self.model) File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 63, in init_callback self._init_callback() File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 194, in _init_callback callback.init_callback(self.model) File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 155, in init_callback self.callback.init_callback(self.model) File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 63, in init_callback self._init_callback() File "/home/mark/Documents/SofaGym-fix_model_load/agents/callbacks.py", line 153, in _init_callback self.eval_env = VecVideoRecorder(self.eval_env, self.save_path, File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/vec_env/vec_video_recorder.py", line 50, in init assert self.env.render_mode == "rgb_array", f"The render_mode must be 'rgb_array', not {self.env.render_mode}"

########## SIG 15 - SIGTERM: a termination request was sent to the program ##########

########## SIG 15 - SIGTERM: a termination request was sent to the program ########## sofa::helper::BackTrace::sig(int) read _Py_read _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall PyObject_Call _PyEval_EvalFrameDefault PyEval_EvalCode PyRun_StringFlags PyRun_SimpleStringFlags Py_RunMain Py_BytesMain libc_start_main sofa::helper::BackTrace::sig(int) read _Py_read _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall PyObject_Call _PyEval_EvalFrameDefault PyEval_EvalCode PyRun_StringFlags PyRun_SimpleStringFlags Py_RunMain Py_BytesMain libc_start_main [ERROR] [SofaRuntime] ConnectionRefusedError: [Errno 111] Connection refused Traceback (most recent call last): File "/home/mark/Documents/SofaGym-fix_model_load/sofagym/rpc_client.py", line 181, in task = s.getNextTask(stateId) File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1122, in call return self.send(self.name, args) File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1464, in request response = self.transport.request( File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1166, in request return self.single_request(host, handler, request_body, verbose) File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1178, in single_request http_conn = self.send_request(host, handler, request_body, verbose) File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1291, in send_request self.send_content(connection, request_body) File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1321, in send_content connection.endheaders(request_body) File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 1277, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 1037, in _send_output self.send(msg) File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 975, in send self.connect() File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 941, in connect self.sock = self._create_connection( File "/home/mark/miniconda3/lib/python3.10/socket.py", line 845, in create_connection raise err File "/home/mark/miniconda3/lib/python3.10/socket.py", line 833, in create_connection sock.connect(sa)

[ERROR] [SofaRuntime] ConnectionRefusedError: [Errno 111] Connection refused Traceback (most recent call last): File "/home/mark/Documents/SofaGym-fix_model_load/sofagym/rpc_client.py", line 181, in task = s.getNextTask(stateId) File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1122, in call return self.send(self.name, args) File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1464, in request response = self.transport.request( File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1166, in request return self.single_request(host, handler, request_body, verbose) File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1178, in single_request http_conn = self.send_request(host, handler, request_body, verbose) File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1291, in send_request self.send_content(connection, request_body) File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1321, in send_content connection.endheaders(request_body) File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 1277, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 1037, in _send_output self.send(msg) File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 975, in send self.connect() File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 941, in connect self.sock = self._create_connection( File "/home/mark/miniconda3/lib/python3.10/socket.py", line 845, in create_connection raise err File "/home/mark/miniconda3/lib/python3.10/socket.py", line 833, in create_connection sock.connect(sa)

Exception ignored in: <function VecVideoRecorder.del at 0x7f9a5ff6ce50> Traceback (most recent call last): File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/vec_env/vec_video_recorder.py", line 113, in del self.close_video_recorder() File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/vec_env/vec_video_recorder.py", line 103, in close_video_recorder if self.recording: File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 392, in getattr return self.getattr_recursive(name) File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 415, in getattr_recursive attr = self.venv.getattr_recursive(name) File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 415, in getattr_recursive attr = self.venv.getattr_recursive(name) File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 417, in getattr_recursive attr = getattr(self.venv, name) AttributeError: 'SubprocVecEnv' object has no attribute 'recording'`

[samuelmyoussef]

Seems you are using a newer version of stable baseline 3, which doesn't work with SofaGym for now. Can you try with version 1.7.0

[MarkWrobelUCL]

Thank you, this has fixed the errors. However, the training still stops at 99/1000 during epoch 1. There are no error messages. The environment simply resets every so often but remains on 99/1000.

[samuelmyoussef]

If you left the hyperparameters in the yaml file as they are, then it's running 5 evaluations every 100 steps, so it's probably just running this evaluation callback not stuck so it should continue after some time if you leave it running. You should change the eval_freq for whichever algorithm you're using if you want the evaluation to happen less frequently.

[MarkWrobelUCL]

I have left it running for longer now and I indeed do get an error after about 3 minutes.

`[ERROR] [SofaRuntime] EOFError Traceback (most recent call last): File "/home/mark/Documents/SofaGym-fix_model_load/rl.py", line 113, in agent.fit(total_timesteps) File "/home/mark/Documents/SofaGym-fix_model_load/agents/SB3Agent.py", line 309, in fit self.model.learn(total_timesteps=total_timesteps, reset_num_timesteps=False, File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/dqn/dqn.py", line 265, in learn return super().learn( File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/off_policy_algorithm.py", line 334, in learn rollout = self.collect_rollouts( File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/off_policy_algorithm.py", line 575, in collect_rollouts if callback.on_step() is False: File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 100, in on_step return self._on_step() File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 204, in _on_step continue_training = callback.on_step() and continue_training File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 100, in on_step return self._on_step() File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 204, in _on_step continue_training = callback.on_step() and continue_training File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 100, in on_step return self._on_step() File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 447, in _on_step episode_rewards, episode_lengths = evaluate_policy( File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/evaluation.py", line 89, in evaluate_policy observations, rewards, dones, infos = env.step(actions) File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 163, in step return self.step_wait() File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/vec_monitor.py", line 76, in step_wait obs, rewards, dones, infos = self.venv.step_wait() File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/vec_normalize.py", line 175, in step_wait obs, rewards, dones, infos = self.venv.step_wait() File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/subproc_vec_env.py", line 121, in step_wait results = [remote.recv() for remote in self.remotes] File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/subproc_vec_env.py", line 121, in results = [remote.recv() for remote in self.remotes] File "/home/mark/miniconda3/lib/python3.10/multiprocessing/connection.py", line 250, in recv buf = self._recv_bytes() File "/home/mark/miniconda3/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes buf = self._recv(4) File "/home/mark/miniconda3/lib/python3.10/multiprocessing/connection.py", line 383, in _recv raise EOFError

########## SIG 15 - SIGTERM: a termination request was sent to the program ########## sofa::helper::BackTrace::sig(int) read _Py_read _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall _PyEval_EvalFrameDefault _PyFunction_Vectorcall PyObject_Call _PyEval_EvalFrameDefault PyEval_EvalCode PyRun_StringFlags PyRun_SimpleStringFlags Py_RunMain Py_BytesMain __libc_start_main [ERROR] [SofaRuntime] ConnectionRefusedError: [Errno 111] Connection refused Traceback (most recent call last): File "/home/mark/Documents/SofaGym-fix_model_load/sofagym/rpc_client.py", line 181, in task = s.getNextTask(stateId) File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1122, in call return self.send(self.name, args) File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1464, in request response = self.transport.request( File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1166, in request return self.single_request(host, handler, request_body, verbose) File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1178, in single_request http_conn = self.send_request(host, handler, request_body, verbose) File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1291, in send_request self.send_content(connection, request_body) File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1321, in send_content connection.endheaders(request_body) File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 1277, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 1037, in _send_output self.send(msg) File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 975, in send self.connect() File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 941, in connect self.sock = self._create_connection( File "/home/mark/miniconda3/lib/python3.10/socket.py", line 845, in create_connection raise err File "/home/mark/miniconda3/lib/python3.10/socket.py", line 833, in create_connection sock.connect(sa)

Exception ignored in: <function VecVideoRecorder.del at 0x7f34331bbb50> Traceback (most recent call last): File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/vec_video_recorder.py", line 113, in del File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/vec_video_recorder.py", line 109, in close File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 279, in close File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/vec_monitor.py", line 100, in close File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 279, in close File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/subproc_vec_env.py", line 144, in close File "/home/mark/miniconda3/lib/python3.10/multiprocessing/connection.py", line 250, in recv File "/home/mark/miniconda3/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes File "/home/mark/miniconda3/lib/python3.10/multiprocessing/connection.py", line 383, in _recv EOFError: Exception ignored in: <function tqdm.del at 0x7f348b5f67a0> Traceback (most recent call last): File "/home/mark/.local/lib/python3.10/site-packages/tqdm/std.py", line 1145, in del File "/home/mark/.local/lib/python3.10/site-packages/tqdm/rich.py", line 120, in close File "/home/mark/.local/lib/python3.10/site-packages/rich/progress.py", line 1177, in exit File "/home/mark/.local/lib/python3.10/site-packages/rich/progress.py", line 1163, in stop File "/home/mark/.local/lib/python3.10/site-packages/rich/live.py", line 155, in stop File "/home/mark/.local/lib/python3.10/site-packages/rich/console.py", line 1136, in line File "/home/mark/.local/lib/python3.10/site-packages/rich/console.py", line 1673, in print File "/home/mark/.local/lib/python3.10/site-packages/rich/console.py", line 1534, in _collect_renderables File "/home/mark/.local/lib/python3.10/site-packages/rich/protocol.py", line 28, in rich_cast ImportError: sys.meta_path is None, Python is likely shutting down`

[samuelmyoussef]

Is this error happening when you train your own environment or one of the examples?

[MarkWrobelUCL]

This is with the gripper-v0 environment, also happening with trunk-v0

Edit: It actually does not crash for the trunk-v0 env. Every 100 steps it launches two windows and runs simulations (both windows show exactly the same thing). This takes about a minute until it starts training again.

My own environment and the gripper-v0 env both crash after the first 100 steps with the errors posted above.

[samuelmyoussef]

The trunk seems to be working right, the rendering windows are opened to record a video of the evaluation callback.

Can you try again with the latest commit to the branch and see if it works.

[MarkWrobelUCL]

I have now tested it with the latest commit. The trunk env does work however, I still have the same issues with the gripper env and my own catheter env.

[samuelmyoussef]

I tested this issue more on the gripper environment. The error happens due to a timeout because the simulation stops responding and stops sending new observations after some time. In the case of the gripper, this is due to a wrong behavior in the simulation most probably because of very large deformation of the fingers causing it to crash. We will work on fixing this but for now If you limit the max_episode_steps parameter to like 40 or 50, it should work fine since the env doesn't reach this state anymore.

In the case of your own catheter env, I can't tell what's wrong with it but to help you debug, you could change the rendering during the callback to True in this line to have a better visualization of what's happening in the simulation during the evaluation. https://github.com/SofaDefrost/SofaGym/blob/f94578f6e078123eafe188dd57c3fbdba86ed0f2/agents/SB3Agent.py#L294

[MarkWrobelUCL]

Thanks for the help. My own env crashes as soon as it starts to evaluate the trained model. This does not happen when I use the code from the main branch. This happens regardless of whether I have render set to True or False. I will paste the error bellow as it is different form the others i have posted thus far. Please let me know if you have any other ideas on how to fix. Thanks.

[ERROR] [SofaRuntime] ValueError: not enough values to unpack (expected 4, got 1) Traceback (most recent call last): File "/home/mark/Documents/SofaGym-fix_model_load_2/rl.py", line 114, in <module> agent.fit(total_timesteps) File "/home/mark/Documents/SofaGym-fix_model_load_2/agents/SB3Agent.py", line 310, in fit self.model.learn(total_timesteps=total_timesteps, reset_num_timesteps=False, File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/ppo/ppo.py", line 307, in learn return super().learn( File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/on_policy_algorithm.py", line 248, in learn continue_training = self.collect_rollouts(self.env, callback, self.rollout_buffer, n_rollout_steps=self.n_steps) File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/on_policy_algorithm.py", line 181, in collect_rollouts if callback.on_step() is False: File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 100, in on_step return self._on_step() File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 204, in _on_step continue_training = callback.on_step() and continue_training File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 100, in on_step return self._on_step() File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 204, in _on_step continue_training = callback.on_step() and continue_training File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 100, in on_step return self._on_step() File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 447, in _on_step episode_rewards, episode_lengths = evaluate_policy( File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/evaluation.py", line 125, in evaluate_policy env.render() File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 282, in render return self.venv.render(mode=mode) File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 282, in render return self.venv.render(mode=mode) File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 184, in render bigimg = tile_images(imgs) File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 33, in tile_images n_images, height, width, n_channels = img_nhwc.shape

[samuelmyoussef]

From the error log, it seems like a problem with the rendering because the image is not received correctly but I'm not really sure what's causing this. If you don't mind sharing the codes for your environment, it would make it easier for me to help you debug this issue. Can you also try to run your environment using the test script test_env.py and see if it works or not?

[MarkWrobelUCL]

Sorry for the late reply, I am now trying on the new main sofagym repo. I still get the same error during the start of eval. It works on the test_env but it initially is in an odd state then corrects itself very quickly (I'm not sure if this is causing the issue). I have attached a simple version of my catheter environment. Thanks for the continued help.

Cath.zip

[MarkWrobelUCL]

Additionally I am still working with the older version of sofagym (the one with broken evaluation). I am using the in built NatureCnn to process images as the input to a PPO. I am slightly confused by the overall network architecture (below). I don't understand the reason for the vf_features_extractor and pi_features_extractor layers. I realize that these are for the actor/critic however there are already features_extractor layers processing the image and it seems to me like it should only be one set of these layers. Are the pi/vf feature extractor layers necessary? Many thanks for the help.