Question - Tensorflow-gpu optional?

[Purdy8TV]

Hi,

I was wanting to use your program and alter the algorithms behind it (employing reinforcement learning) and largely leave the base code untouched.

Checking 'tensorforce-PPO-IEET' I don't see tensorflow-gpu being used, is this assumption wrong? (Wanting to work on this on OSX which lost support for tfgpu at 1.1)

EDIT: Okay I've tried installing some different versions of the libraries in your requirements as I couldn't get them to install (particularly the tensorflow-gpu)

Reduced the step count significantly along with epsilon requirement.

However when I'm getting to the Test block I'm getting the following error.

`AttributeError                            Traceback (most recent call last)
<ipython-input-23-a605848dcb2f> in <module>()
      1 # Create an agent
      2 agent = PPOAgent(
----> 3     states_spec=environment.states,
      4     actions_spec=actions_spec,
      5     network_spec=network_spec,

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorforce/contrib/openai_gym.py in states(self)
     77     @property
     78     def states(self):
---> 79         return OpenAIGym.state_from_space(space=self.gym.observation_space)
     80 
     81     @staticmethod

AttributeError: 'NoneType' object has no attribute 'observation_space'`

Thank you

[wassname]

From memory I think it works without the gpu, and at about the same speed since it's bound by the speed of the environment - not the optimisation.

Make sure you're installing the versions from requirements/requirements.txt, otherwise you will get errors. Not sure why you want to install different version.

[Purdy8TV]

Sorry I should have been more clear it was only tensorflow-gpu (I'm just using tensor flow 1.5.0 and it doesn't appear to be causing issues) and tensorforce.

it would appear tensorforce is giving me the issue as shown above, I was able to install the specific commit in your requirements by manual install but mine still reports back as '0.3.5.1' rather than '0.3.5.2'

AttributeError                            Traceback (most recent call last)
<ipython-input-22-a605848dcb2f> in <module>()
      1 # Create an agent
      2 agent = PPOAgent(
----> 3     states_spec=environment.states,
      4     actions_spec=actions_spec,
      5     network_spec=network_spec,

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorforce-0.3.5.1-py3.6.egg/tensorforce/contrib/openai_gym.py in states(self)
     77     @property
     78     def states(self):
---> 79         return OpenAIGym.state_from_space(space=self.gym.observation_space)
     80 
     81     @staticmethod

AttributeError: 'NoneType' object has no attribute 'observation_space'

I shall try to investigate this further

[wassname]

That makes sense. Mine actually says 0.3.5.1 too, so maybe that's a typo in requirements.txt (fixed).

Do you want to check what version of gym you have? They have refactored it recently and that could be causing the error (should be 0.9.3).

[Purdy8TV]

Yeah I've double checked my versions of everything. I'm going to get to grips with using a virtual environment and use one for this to try and figure this out instead of hassling you with basic queries.

Thanks for the replies.

[Purdy8TV]

Okay I have set up virtualenvs on both W10 and OSX and I'm reproducing the same error above on both OS.

Deviations from the requirements.txt

• tensorflow(1.5.0) instead of tensorflow-gpu(1.3.0)
• I have to download the tensorforce link and delete the build file within it and install manually.
• See installed modules in virtualenv

Error once again, not sure if you have anything to suggest?

AttributeError                            Traceback (most recent call last)
<ipython-input-22-a605848dcb2f> in <module>()
      1 # Create an agent
      2 agent = PPOAgent(
----> 3     states_spec=environment.states,
      4     actions_spec=actions_spec,
      5     network_spec=network_spec,

d:\daniel\google drive\python\envtesting\testenvw10\lib\site-packages\tensorforce-0.3.5.1-py3.6.egg\tensorforce\contrib\openai_gym.py in states(self)
     77     @property
     78     def states(self):
---> 79         return OpenAIGym.state_from_space(space=self.gym.observation_space)
     80 
     81     @staticmethod

AttributeError: 'NoneType' object has no attribute 'observation_space'

[wassname]

No worries, seems like it's not a basic question and you've tried all the obvious things. I'll try to replicate on my desktop when I'm at home.

For now I'm thinking try earlier/later version of gym or tensorforce.

I've also got a DDPG pytorch notebook, which is a lot more fun (and pythonic) to debug and play with. That's my latest attempt. I'm not sure if you would prefer that? I can upload it for you.

Edit: here's a gist of it, but it might not work with the requirements in the project, but I can fix that in the weekend. However you might have an easier time getting it working. Yeah I've been trying DDPG because the inherent randomness in the output of PPO didn't seem to be doing me any favors.

[Purdy8TV]

I shall play around with earlier versions of gym or tesorforce and see where it gets me.

Thanks, I will certainly try to understand and see if I can get it working.

[Purdy8TV]

Had a brief play around with the DDPG pytorch but had a few issues.

Currently have it running as reading from one of the other questions this has a run time ~5 hours? I'm wondering if my adjustment of the steps/epsilon value could be causing issue? (Considering I can replicate it on each system) Hopefully have an update in a few hours.

[wassname]

Yeah that's about right, it takes a while

Do you reckon it's easier to use? If you improve the notebook feel free to submit it back as a PR or something, for the next person.

[Purdy8TV]

Do I think the DDPG one is easier to use? If so I haven't given it enough time to be honest.

I let the PPO run for it appears ~16 hours going by my command prompt saves. This was the output which it has stopped at for ~3 hours - https://gist.github.com/Purdy8TV/75dceb3f6fb5ae0910bb493f9d1ce4c8

I've been given your repo to work on as a final year project, however my course wasn't programming focused (mathematical finance) and my only python knowledge comes from a few small projects I made myself, but yes my goal is to get this working and improve upon it (granted I see you're quite active with RL so that may be difficult to achieve) but will share anything.

[wassname]

How long do you have to spend on the final project? RL can be a bit of a time sink since lots of times you run a ton of experiments and get no signal. I always used to bite off more than I can chew, but as long as you have your sights set on a doable experiment and some nice graphs it should be fine. I'm pretty sure you can change it or change the scope if you want since it's your project.

I would definitely try the pytorch one as I found it much easier to modify and play with compared to tensorflow based algorithms which become a headache to debug. Seem's like your good at programming so I'm sure you'll work it out either way.

Some links you might find usefull... If you want a quick intro to RL I quite liked the deeprl bootcamp, but other people prefer Silver's course because it walks you through it more comprehensively. I found it helped a lot working out hyper-parameters etc. Eventually you might want RL tips and there are a bunch here on reddit.

It'l be cool to see what you find, you should send me a copy of your write up when you get to that stage.

[wassname]

From your gist it doesn't look like it didn't learn much because the output weights look like random noise. Or at least that's my interpretation. PPO learns a mean action and a standard deviation. One common failure case it where it leans mean=0 and std=small. This avoids most fees but, but because of the standard deviation, you see the weights change a little during training.

[Purdy8TV]

I have about 12 weeks from now and have budgeted around 250 hours roughly.

Cheers, I will take your advice and give it the time tomorrow to get it going (pytorch). Also thanks for the link of resources, been using David Silver so far but happy to repeat some material to solidify my understandings. I think I need to play around with some simpler programs also.

Okay I'll need to take a closer look at that to understand why this occurred. Edit - scrap that I wasn’t familiar with that notation for x10^x!

[hmingkwan]

Hi guys, I encounter the same AttributeError, may I know that if it is fixed already?

[wassname]

I haven't had time yet sorry, hopefully this weekend

[Purdy8TV]

Hi,

Don't suppose you've experienced the following with the pytorch ddpg before - https://gist.github.com/Purdy8TV/82ea432b4cff4ad10d81829eefd53901 ?

Perhaps a Python3 issue?

[wassname]

Better not to mix issues on this thread, so is it alright if (in future) you make a separate issue for different problems. Or if it's code that's not up on the repo (like that gist), please email me instead (wassname at wassname then org) and I'll help if I have time.

Looks like that was an old version of that notebook, I updated the gist with a version that runs, (but testing doesn't work yet). Basically the enviroment returns a dict, but I'm not using a openai.gym.wrapper to transform the state into an array. In the notebook you just tried it was trying to slice a dict. To debug those error I find it useful to make a new cells with %debug which enters python debugging mode.

[wassname]

I had a go at replicating that AttributeError, but I couldn't. I can't see how it can be happening with the current version of the notebook.

@Purdy8TV could you please check if you were using the latest version of the notebook (same as on the repo)? And could you also run conda list and pip freeze and show me the output?

Mine (with lots of random stuff):

affine==2.1.0
apipkg==1.4
arrow==0.10.0
atari-py==0.1.1
attrs==17.4.0
autopep8==1.3.2
backoff==1.4.3
backports.weakref==1.0rc1
-e git+https://github.com/openai/baselines.git@b05be68c55d416f7aa71f745aaf3c2999d83b83c#egg=baselines
bcolz==1.1.2
beautifulsoup4==4.6.0
bench==2.8
BitcoinExchangeFH==0.2.3
bleach==2.1.2
boto3==1.4.6
botocore==1.6.3
bs4==0.0.1
certifi==2017.7.27.1
chardet==3.0.4
click==6.7
click-plugins==1.0.3
cligj==0.4.0
cntk==2.2
coverage==4.4.1
cvxopt==1.1.9
cycler==0.10.0
Cython==0.26.1
deap==1.2.2
decorator==4.2.1
dill==0.2.7.1
dnspython==1.15.0
docopt==0.6.2
docutils==0.14
entrypoints==0.2.3
enum34==1.1.6
execnet==1.4.1
flake8==3.4.1
future==0.16.0
graphviz==0.8.1
gym==0.9.3
h5py==2.7.0
html5lib==1.0.1
-e git+https://github.com/hyperopt/hyperopt.git@7e247f5963d05f6e7e9299ff59bc90b126ef7987#egg=hyperopt
idna==2.6
imageio==2.2.0
ipykernel==4.8.0
ipython==6.2.1
ipython-genutils==0.2.0
ipywidgets==7.0.5
jedi==0.11.1
Jinja2==2.10
jmespath==0.9.3
joblib==0.11
jsonpickle==0.9.5
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.2.2
jupyter-console==5.2.0
jupyter-contrib-core==0.3.1
jupyter-contrib-nbextensions==0.2.8
jupyter-core==4.4.0
jupyter-highlight-selected-word==0.0.11
jupyter-latex-envs==1.3.8.4
jupyter-nbextensions-configurator==0.2.5
jupyterthemes==0.18.2
Keras==2.1.2
-e git+git@github.com:wassname/keras-contrib.git@6379a5a9b3d58e409b57b6df7cac89347ce6f1a8#egg=keras_contrib
keras-rl==0.3.1
keras-tqdm==2.0.1
lesscpy==0.12.0
Markdown==2.6.10
MarkupSafe==1.0
matplotlib==2.1.2
mccabe==0.6.1
mistune==0.8.3
mock==2.0.0
mpi4py==2.0.0
mujoco-py==0.5.7
munch==2.2.0
mysqlclient==1.3.12
nbconvert==5.3.1
nbformat==4.4.0
nbstripout==0.3.1
netaddr==0.7.19
networkx==1.11
nose==1.3.7
notebook==5.3.0
numexpr==2.6.2
numpy==1.14.0
olefile==0.44
opencv-python==3.3.0.10
osim-rl==1.5.1
pandas==0.20.3
pandas-datareader==0.5.0
pandas-profiling==1.4.0
pandocfilters==1.4.2
parso==0.1.1
path.py==10.3.1
patsy==0.4.1
pbr==3.1.1
pep8==1.7.0
pexpect==4.3.1
pickleshare==0.7.4
Pillow==5.0.0
pluggy==0.6.0
ply==3.10
poloniex==0.4.6
pprint==0.1
progressbar2==3.34.3
prompt-toolkit==1.0.15
protobuf==3.5.0.post1
psutil==5.2.2
ptyprocess==0.5.2
py==1.5.2
pycodestyle==2.3.1
pydot==1.2.3
pyflakes==1.5.0
pyglet==1.2.4
Pygments==2.2.0
pymongo==3.5.1
PyMySQL==0.7.11
PyOpenGL==3.1.0
pyparsing==2.2.0
pyproj==1.9.5.1
pytest==3.4.0
pytest-cache==1.0
pytest-cov==2.5.1
pytest-forked==0.2
pytest-pep8==1.0.6
pytest-xdist==1.20.0
python-coveralls==2.9.1
python-dateutil==2.6.1
python-utils==2.2.0
pytz==2017.3
PyWavelets==0.5.2
PyYAML==3.12
pyzmq==16.0.4
qgrid==1.0.0
qPython==1.2.2
qtconsole==4.3.1
rasterio==0.36.0
redis==2.10.6
requests==2.18.4
requests-file==1.4.2
requests-ftp==0.3.1
rlp==0.6.0
-e git+https://github.com/openai/roboschool.git@6dce1eacef85a5ac14f6121fa6e152a54becea69#egg=roboschool
s3transfer==0.1.10
sacred==0.7.2
scikit-image==0.13.0
scikit-learn==0.19.1
scikit-optimize==0.5
scipy==1.0.0
seaborn==0.8
selenium==3.7.0
Send2Trash==1.4.2
Shapely==1.6.0
simplegeneric==0.8.1
six==1.11.0
sklearn==0.0
snuggs==1.4.1
SQLAlchemy==1.1.15
statsmodels==0.8.0
stopit==1.1.1
tables==3.4.2
tensorboard-logger==0.0.4
tensorboardX==0.8
tensorflow==1.3.0
tensorflow-gpu==1.4.1
tensorflow-tensorboard==0.4.0rc3
tensorforce==0.3.5.1
terminado==0.8.1
testpath==0.3.1
tflearn==0.3.2
Theano==0.9.0
timeout-decorator==0.4.0
timer==0.1
torch==0.3.0.post4
torchvision==0.1.8
tornado==4.5.3
tox==2.9.1
TPOT==0.9.2
tqdm==4.19.5
traitlets==4.3.2
universal-portfolios==0.3.2
update-checker==0.16
urllib3==1.22
virtualenv==15.1.0
visdom==0.1.5
wcwidth==0.1.7
webencodings==0.5.1
websocket-client==0.44.0
Werkzeug==0.13
widgetsnbextension==3.0.8
wrapt==1.10.11
xgboost==0.7.post3
xlrd==1.1.0
yapf==0.16.3
zmq==0.0.0

[hmingkwan]

Hi @wassname, thanks for updating the tensorforce-PPO-IEET.ipynb, is the testing part not ready yet?

[wassname]

It's working in the latest notebook

[hmingkwan]

@wassname When I try to run the tensorforce-PPO-IEET.ipynb, there is an AttributeError.

[wassname]

Please make a separate issue for new issues, and provide details needed to replicate it. Make sure you have the right versions of everything and updated files from the repo.