Error running notebook

[jiadingfang]

Hi,

I'm very interesting in your code, but I''m having trouble running the two notebooks.

For the tensorforce-VPG notebook, the error happens at block 7, and the error message is

ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-7-445b09d9425b> in <module>()
     22 env_test.seed = 0
     23 
---> 24 from tensorforce.environments.openai_gym import OpenAIGym
     25 environment = OpenAIGym('CartPole-v0')
     26 environment.gym = env

ModuleNotFoundError: No module named 'tensorforce.environments.openai_gym'

Anything to do with your tensorforce version? As for now, it's "use my branch with prioritised ppo, untill mergeed"

https://github.com/wassname/tensorforce/archive/merged_6b.zip

For the tensorforce-PPO notebook, the error happens at in Training session, block 19

episodes = int(6e6 / 30)
runner.run(
    episodes=episodes,
    max_timesteps=200,
    episode_finished=EpisodeFinishedTQDM(
        #steps=1,
        log_intv=100, 
        episodes=episodes,
        log_dir=log_dir,
        session=runner.agent.model.session
    )
)

First time, the error message is

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-20-2c7dac94069d> in <module>()
      8         episodes=episodes,
      9         log_dir=log_dir,
---> 10         session=runner.agent.model.session
     11     )
     12 )

TypeError: __init__() got an unexpected keyword argument 'episodes'

so I change it to episode which I find in tensroforce file.

Second time, the error message is

TypeError                                 Traceback (most recent call last)
<ipython-input-21-02d9c6e080ce> in <module>()
      8         episode=episodes,
      9         log_dir=log_dir,
---> 10         session=runner.agent.model.session
     11     )
     12 )

TypeError: __init__() missing 1 required positional argument: 'steps'

so I ad an argument steps, and set it to some random constant 1.

Third time, the error message is

If you're reading this message in Jupyter Notebook or JupyterLab, it may mean that the widgets JavaScript is still loading. If this message persists, it likely means that the widgets JavaScript library is either not installed or not enabled. See the Jupyter Widgets Documentation for setup instructions.
If you're reading this message in another notebook frontend (for example, a static rendering on GitHub or NBViewer), it may mean that your frontend doesn't currently support widgets.
TensorBoardLogger started. Run `tensorboard --logdir=/home/jfang16/Dropbox/rl-portfolio-management/logs` to visualize
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-23-e11fc16557ce> in <module>()
      8         episode=episodes,
      9         log_dir=log_dir,
---> 10         session=runner.agent.model.session
     11     )
     12 )

~/anaconda3/envs/rl-portfolio/lib/python3.6/site-packages/tensorforce/execution/runner.py in run(self, episodes, max_timesteps, episode_finished)
    165                 self.agent.save_model(self.save_path)
    166 
--> 167             if episode_finished and not episode_finished(self):
    168                 return
    169             if self.cluster_spec is None:

~/Dropbox/rl-portfolio-management/rl_portfolio_management/callbacks/tensorforce.py in __call__(self, r)
    146         weights = dict(zip(oai_env.src.asset_names, np.round(oai_env.sim.w0, 4).tolist()))
    147         exploration = r.agent.exploration.get('action', lambda x, y: 0)(
--> 148             r.episode, np.sum(r.episode_timesteps))
    149 
    150         desc = "ep reward: {reward: 2.8f} [{rewards_min: 2.8f}, {rewards_max: 2.8f}], portfolio_value: {portfolio_value: 2.4f} mdd={mdd:2.2%} sharpe={sharpe:2.4f}, expl={exploration: 2.2%} eps={episode:} weights={weights:}".format(

AttributeError: 'Runner' object has no attribute 'episode_timesteps'

I'm not sure if I should continue.

Please help,

Thanks,

[wassname]

No problems, I'll fix this today

[wassname]

Updated, want to give it ago with the latest code? You can install the package versions from requirements/requirements.txt (especially tensorforce)

[jiadingfang]

Cool, it works now, thanks! But I'm surprised to see it having a run time of 50 hours(I know RL algo usually takes a lot of time to run though), is that the same situation with you?

[wassname]

It is, and this implementation is not built to be parallel. You can usually see it converging (or not) after about 8 hours or you can change the max steps.

[wassname]

It should actually be 9 hours now, since I optimised the environment a bit. Just pull the latest code.

[jiadingfang]

It's about 6 hours for me now. Cool updates!

[jiadingfang]

Not sure if I did anything wrong, but when the training is done and it's going to view the history and save the agent, error happens.

In block 19, though there is not error, but the output is essentially a blank image;

In block 20, the error message goes like,

RuntimeError                              Traceback (most recent call last)
<ipython-input-20-c5d9f08f990b> in <module>()
----> 1 agent.save_model('./outputs/tensorforce-PPO-prioritised/tensorforcePPO')

~/anaconda3/envs/rl-portfolio/lib/python3.6/site-packages/tensorforce/agents/agent.py in save_model(self, directory, append_timestep)
    345             Checkpoint path were the model was saved.
    346         """
--> 347         return self.model.save(directory=directory, append_timestep=append_timestep)
    348 
    349     def restore_model(self, directory=None, file=None):

~/anaconda3/envs/rl-portfolio/lib/python3.6/site-packages/tensorforce/models/model.py in save(self, directory, append_timestep)
    919             meta_graph_suffix='meta',
    920             write_meta_graph=True,
--> 921             write_state=True
    922         )
    923 

~/anaconda3/envs/rl-portfolio/lib/python3.6/site-packages/tensorflow/python/training/saver.py in save(self, sess, save_path, global_step, latest_filename, meta_graph_suffix, write_meta_graph, write_state)
   1450     if global_step is not None:
   1451       if not isinstance(global_step, compat.integral_types):
-> 1452         global_step = training_util.global_step(sess, global_step)
   1453       checkpoint_file = "%s-%d" % (save_path, global_step)
   1454       if self._pad_step_number:

~/anaconda3/envs/rl-portfolio/lib/python3.6/site-packages/tensorflow/python/training/training_util.py in global_step(sess, global_step_tensor)
     56     The global step value.
     57   """
---> 58   return int(sess.run(global_step_tensor))
     59 
     60 

~/anaconda3/envs/rl-portfolio/lib/python3.6/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
    893     try:
    894       result = self._run(None, fetches, feed_dict, options_ptr,
--> 895                          run_metadata_ptr)
    896       if run_metadata:
    897         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

~/anaconda3/envs/rl-portfolio/lib/python3.6/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1049     # Check session.
   1050     if self._closed:
-> 1051       raise RuntimeError('Attempted to use a closed Session.')
   1052     if self.graph.version == 0:
   1053       raise RuntimeError('The Session graph is empty.  Add operations to the '

RuntimeError: Attempted to use a closed Session.

Can you please check it up? Anything I can do to fix this?

[wassname]

I always end up stopping mine with keyboard interupt. But you let it run the whole time, tensorforce runs agent.close which shuts down the tensorflow session. Not sure how to fix that right now as I haven't had time to have a look. But if you were using the notebook that has

   saver_spec = dict(
        directory=save_path, 
        steps=100000, 
    ),

in the config, then it was saving as it went, and will resume upon running the notebook.

Not sure why the history didn't show.

What you can do it: restart the notebook, execute the cells again in order (except the one which trains it). If you're using the latest notebook it will resume from the last save and you might be able to test and view history.

[wassname]

I just fixed a critical bug with the input data (introduced when I made the speedup) so you might need to run again sorry

[jiadingfang]

This time the error happens at the training block, see here

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-17-c4353544343c> in <module>()
     10         mean_of=1000,
     11         log_dir=log_dir,
---> 12         session=runner.agent.model.session,
     13     )
     14 )

~/anaconda3/envs/rl-portfolio/lib/python3.6/site-packages/tensorforce/execution/runner.py in run(self, timesteps, episodes, max_episode_timesteps, deterministic, episode_finished)
    133                 break
    134 
--> 135         self.agent.close()
    136         self.environment.close()

~/anaconda3/envs/rl-portfolio/lib/python3.6/site-packages/tensorforce/agents/agent.py in close(self)
    194 
    195     def close(self):
--> 196         self.model.close()
    197 
    198     def initialize_model(self, states_spec, actions_spec, config):

~/anaconda3/envs/rl-portfolio/lib/python3.6/site-packages/tensorforce/models/model.py in close(self)
    417         if self.saver_directory is not None:
    418             self.save(append_timestep=True)
--> 419         self.monitored_session.close()
    420 
    421     def initialize(self, custom_getter):

~/anaconda3/envs/rl-portfolio/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py in close(self)
    524 
    525   def close(self):
--> 526     self._close_internal()
    527 
    528   def __enter__(self):

~/anaconda3/envs/rl-portfolio/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py in _close_internal(self, exception_type)
    564       if not exception_type:
    565         for h in self._hooks:
--> 566           h.end(self._coordinated_creator.tf_sess)
    567     finally:
    568       try:

~/anaconda3/envs/rl-portfolio/lib/python3.6/site-packages/tensorflow/python/training/basic_session_run_hooks.py in end(self, session)
    443 
    444   def end(self, session):
--> 445     last_step = session.run(training_util.get_global_step())
    446     if last_step != self._timer.last_triggered_step():
    447       self._save(last_step, session)

~/anaconda3/envs/rl-portfolio/lib/python3.6/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
    893     try:
    894       result = self._run(None, fetches, feed_dict, options_ptr,
--> 895                          run_metadata_ptr)
    896       if run_metadata:
    897         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

~/anaconda3/envs/rl-portfolio/lib/python3.6/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1107     # Create a fetch handler to take care of the structure of fetches.
   1108     fetch_handler = _FetchHandler(
-> 1109         self._graph, fetches, feed_dict_tensor, feed_handles=feed_handles)
   1110 
   1111     # Run request and get response.

~/anaconda3/envs/rl-portfolio/lib/python3.6/site-packages/tensorflow/python/client/session.py in __init__(self, graph, fetches, feeds, feed_handles)
    411     """
    412     with graph.as_default():
--> 413       self._fetch_mapper = _FetchMapper.for_fetch(fetches)
    414     self._fetches = []
    415     self._targets = []

~/anaconda3/envs/rl-portfolio/lib/python3.6/site-packages/tensorflow/python/client/session.py in for_fetch(fetch)
    228     if fetch is None:
    229       raise TypeError('Fetch argument %r has invalid type %r' %
--> 230                       (fetch, type(fetch)))
    231     elif isinstance(fetch, (list, tuple)):
    232       # NOTE(touts): This is also the code path for namedtuples.

TypeError: Fetch argument None has invalid type <class 'NoneType'>

[wassname]

The error happens when it stops by itself. It's stopping because you've reached max steps/= or episodes probably (including saved data)

[jiadingfang]

OK, but how may I fix it? I see the same error happens in the notebook provided in the github repository.

BTW, do you find anywhere needs to be modified after checking up the the official released code? Would like to hear about that, because I still like more about the gym environment building strategy here.

[wassname]

No fix yet, I just work around it. I haven't had time to check the official code yet sorry.

[december-soul]

Ok i got the same error as kudo1026.

I just took the next step. Keep walking.

But then I get an error during the test at

episodes=1
steps=environment_test.gym.env.env.src.steps*episodes
runner_test.run(
    episodes=episodes,
    timesteps=steps,
    deterministic=True,
    episode_finished=episode_finished,
)

AttributeError                            Traceback (most recent call last)
<ipython-input-29-de2ce0c0e318> in <module>()
      5     timesteps=steps,
      6     deterministic=True,
----> 7     episode_finished=episode_finished,
      8 )

/usr/local/lib/python3.5/dist-packages/tensorforce/execution/runner.py in run(self, timesteps, episodes, max_episode_timesteps, deterministic, episode_finished)
     75         self.start_time = time.time()
     76 
---> 77         self.agent.reset()
     78 
     79         self.episode = self.agent.episode

/usr/local/lib/python3.5/dist-packages/tensorforce/agents/agent.py in reset(self)
    204         timestep counter, internal states,  and resets preprocessors.
    205         """
--> 206         self.episode, self.timestep, self.next_internals = self.model.reset()
    207         self.current_internals = self.next_internals
    208 

/usr/local/lib/python3.5/dist-packages/tensorforce/models/model.py in reset(self)
    811             initializations.
    812         """
--> 813         episode, timestep = self.monitored_session.run(fetches=(self.episode, self.timestep))
    814         return episode, timestep, list(self.internal_inits)
    815 

/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py in run(self, fetches, feed_dict, options, run_metadata)
    513       Same as `tf.Session.run()`.
    514     """
--> 515     return self._sess.run(fetches,
    516                           feed_dict=feed_dict,
    517                           options=options,

AttributeError: 'NoneType' object has no attribute 'run'

[wassname]

Should be fixed in the latest commit