Load model to Re-train. " 'NoneType' object has no attribute 'reset' "

[malik-ben]

Hi,

I might have been coding for too long now but I can't figure out this issue.

I train a model. Save it. Then load it in order to train it again on new data. But when I run the .train_model() function, I get the following error. Maybe I'm missing something.

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/tmp/ipykernel_486/2121854827.py in <module>
----> 1 trained_ppo = agent.train_model(model = ppo_loaded, 
      2                              tb_log_name='ppo',
      3                              total_timesteps=80000)

~/rl/finrl/drl_agents/stablebaselines3/models.py in train_model(self, model, tb_log_name, total_timesteps)
    100 
    101     def train_model(self, model, tb_log_name, total_timesteps=5000):
--> 102         model = model.learn(
    103             total_timesteps=total_timesteps,
    104             tb_log_name=tb_log_name,

~/miniconda3/envs/X/lib/python3.8/site-packages/stable_baselines3/ppo/ppo.py in learn(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps)
    297     ) -> "PPO":
    298 
--> 299         return super(PPO, self).learn(
    300             total_timesteps=total_timesteps,
    301             callback=callback,

~/miniconda3/envs/X/lib/python3.8/site-packages/stable_baselines3/common/on_policy_algorithm.py in learn(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps)
    227         iteration = 0
    228 
--> 229         total_timesteps, callback = self._setup_learn(
    230             total_timesteps, eval_env, callback, eval_freq, n_eval_episodes, eval_log_path, reset_num_timesteps, tb_log_name
    231         )

~/miniconda3/envs/X/lib/python3.8/site-packages/stable_baselines3/common/base_class.py in _setup_learn(self, total_timesteps, eval_env, callback, eval_freq, n_eval_episodes, log_path, reset_num_timesteps, tb_log_name)
    424         # Avoid resetting the environment when calling ``.learn()`` consecutive times
    425         if reset_num_timesteps or self._last_obs is None:
--> 426             self._last_obs = self.env.reset()  # pytype: disable=annotation-type-mismatch
    427             self._last_episode_starts = np.ones((self.env.num_envs,), dtype=bool)
    428             # Retrieve unnormalized observation for saving into the buffer

AttributeError: 'NoneType' object has no attribute 'reset'

[malik-ben]

Not sure if it's correct but i assigned the env I wanted and it seems to work. Guess it's still an issue isnt it or should I close it?

[Miffyli]

You should not call train_model (instead, calling learn function), but also yes, you should assign the training env before continuing learning. See docs on this. Also I highly recommend changing to stable-baselines3 if possible as it is more up-to-date :)

Closing as resolved.

[malik-ben]

Thank you. I had reached this conclusion and it works since then. The error was on assigning the env before... Also just for uriosity, I manged to use the train function since it does only trigger the learn function... but i'll use learn directly.

Thank you again

[rambo1111]

https://github.com/hill-a/stable-baselines/issues/1192