search

AminHP / gym-anytrading

The most simple, flexible, and comprehensive OpenAI Gym trading environment (Approved by OpenAI Gym)
MIT License
2.13k stars 467 forks source link

Balance, order volume is not explained in the docs. Reward gaming behavior observed in some models. #90

Open astrologos opened 1 year ago

astrologos commented 1 year ago

Hi @AminHP @bionicles @super-pirata , could you please update the docs to include an explanation of the agent's asset pool and how the volume of an order is determined? An example on how to change these would also be helpful, as the README is opaque in this regard.

I have observed what I suspect is reward-gaming behavior in my trained agents, and I'm wondering if it is due to an under-specified environment. My agents (tested on DQN, PPO and A2C) fall into a stable valley where their optimal behavior is to first sell, then buy and continue buying. This results in profit slightly shy of 1.

Clearly this should not be valid behavior in any trading environment.

Thanks! Jack

astrologos commented 1 year ago

873b8264-0ca6-404f-b70e-2079a6047691 Training loss associated with this policy is 0.0008.

astrologos commented 1 year ago

087e6047-fbd1-4ac8-975b-749c3befe08a

astrologos commented 1 year ago

Hi, I am writing to report that I've created an alternative trading environment that is more advanced, highly customizable, and simpler to use, render and evaluate. It dodges the issues above.

You can find the repo here: https://www.github.com/astrologos/tradinggym