search

ClementPerroud / Gym-Trading-Env

A simple, easy, customizable Gymnasium environment for trading.
https://gym-trading-env.readthedocs.io/
MIT License
267 stars 56 forks source link

Agents #4

Open InTheta opened 1 year ago

InTheta commented 1 year ago

Hi, can we add some functionality to add agents?

Thanks

ClementPerroud commented 1 year ago

Yes of course ! I work on a from scratch implementation of a Rainbow agent. But I would personally recommend using a library like RLLIB. What would you like to see ?

InTheta commented 1 year ago

pettingzoo and RLLib looks great.

Maybe use something like quant stats also?

Would be good to get a boiler plate environment that can be inherited or modified with overrides and really basic buy / sell at random so we can build custom envs easy.

Happy to work on some once I figure it out.

On Mon, 15 May 2023 at 22:06, Clément Perroud @.***> wrote:

Yes of course ! I work on a from scratch implementation of a Rainbow agent. But I would personally recommend using a library like RLLIB. What would you like to see ?

— Reply to this email directly, view it on GitHub https://github.com/ClementPerroud/Gym-Trading-Env/issues/4#issuecomment-1548578278, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEDMDPB5K4CAUROCZXTL4P3XGKLFVANCNFSM6AAAAAAYCE22WE . You are receiving this because you authored the thread.Message ID: @.***>

kvrban commented 1 year ago

I currently use it with stable-baselines3 and vanilla DQN. rllib seemed a bit overkill for me to get started.

ClementPerroud commented 1 year ago
  1. "Supra-classing" the TradingEnv class with a really basic trading environment class that would be that "BaseTradingEnv" that would be a simple buy/sell (without the idea of position ?) environment. Is that what you were thinking of ?
  2. We need to decide which RL library to recommend. IMO :
    • stable-baselines3 is easy to use but a bit out-dated in terms of agents
    • RLlib is probably the more efficient lib, but it lacks flexibility and looks hard to use I would prefer using RLLIB in tutorial, what do you think ?
InTheta commented 1 year ago

Hey

I have been using stable baselines, it’s versatile and functional and compatible with Gymnasium >= 2.0

Getting GPU functionality would be important factor here.

Petting zoo would be pretty crazy later on.

1) Yes, I think we need something simpler to build on. A positions dictionary, PnL tracking in some base class maybe?

On Tue, 16 May 2023 at 15:09, Clément Perroud @.***> wrote:

  1. "Supra-classing" the TradingEnv class with a really basic trading environment class that would be that "BaseTradingEnv" that would be a simple buy/sell (without the idea of position ?) environment. Is that what you were thinking of ?

  2. We need to decide which RL library to recommend. IMO :

    • stable-baselines3 is easy to use but a bit out-dated in terms of agents
    • RLlib is probably the more efficient lib, but it lacks flexibility and looks hard to manage I would prefer using RLLIB in tutorial, what do you think ?

— Reply to this email directly, view it on GitHub https://github.com/ClementPerroud/Gym-Trading-Env/issues/4#issuecomment-1549753544, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEDMDPBCU4XCX62E5HOU4STXGODADANCNFSM6AAAAAAYCE22WE . You are receiving this because you authored the thread.Message ID: @.***>

ClementPerroud commented 1 year ago

Hi thanks for your reply,

Yes petting zoo is a good idea !

I used to work with stable_baselines3 when I was learning RL as it is easy to implement. But the main problem to me is that you can not really implement recurrent networks, neither transformers networks. And it is not up-to-date on stage of the art DQN agents (C51, Rainbow, Categorical, R2D2 ...), which probably are the best agents for trading agent. I'll take a closer look that verify what I'm saying

I actually do not understand what you really expect from my lib :/

Can you explain from the beginning what you would like ? Or maybe, can you tell us about your personal project behind this, so I can understand your needs ?

Happy to help ;)

ihopethiswillfi commented 1 year ago

There's also DI-engine. I'm using RLlib myself but DI-engine is on my shortlist because its list of agent implementations is very impressive: https://github.com/opendilab/DI-engine#algorithm-versatility . It looks more lightweight than RLlib, so may be good for a reference implementation.

InTheta commented 1 year ago

Thanks for the info.

I have built my own env with charting and stuff, but was thinking to be able to build environments with base functions easily and select and back test multiple agents.

doesn’t seem too much to get that.

Base portfolio tracker Base rewards tracking Base history / trades

I have found the management of rewards and pnl extremely challenging this week.

I am happy to contribute

ClementPerroud commented 1 year ago

Hi,

Thanks all of you for your ideas and suggestions ! I am a little busy these times, but I will begin to implement this in a few days !

1 - For the "vanilla env", this is not as easy as it sounds, as I need to revise the structure of my classes (especially the interaction between the TradingEnv class and the Portfolio class). This would allow custom portfolio management behavior really easy to implement without having to modify the environment mechanism (so you will still be able to track all type of metrics with the logs and the renders) ! I think this is the solution.

2 - DI-Engine looks really promising ! Do you have feedback about it ? Does the agents works fine ?

ihopethiswillfi commented 1 year ago

2 - DI-Engine looks really promising ! Do you have feedback about it ? Does the agents works fine ?

Unfortunately me too, I need to find time to try it. I was unable to find feedback, even on reddit. Which I find strange, for a repo that leaves such a good first impression. I just suggested it here because it would be what I'd try first if I were starting from scratch, and it looks like it could be suitable for a relatively 'simple' reference implementation while still using state-of-the-art agents.

ClementPerroud commented 1 year ago

The only thing I can find is a post on Reddit a few weeks ago. According to the Github stars, the package is really famous. It probably is popular among the Asian RL community.

The more I dig, the more I find this package amazing. We need to try this

ClementPerroud commented 1 year ago

Hello @InTheta, I have been working, on something to allow custom portfolio management. Now the portfolio management process (trades, margin, borrow ...) and the environment are completely separated. By initializing the environment, you need to specify a Portfolio Manager object that will handle "actions" to perform portfolio operations. By default, the portfolio manager is the PositionBasedPortfolioManager that works with positions as it already was. If you want to build your own PortfolioManager, it is preferable that it inherits from the CorePortfolioManager to simplify the process. As it already handles a lot of stuff including and .... I will make a tutorial on that part soon. What do you think about this ?

InTheta commented 1 year ago

Hey, Sounds interesting! Would like to see this! Thanks for sharing your knowledge!

ClementPerroud commented 1 year ago

Hi, I will release it soon (today or tomorrow) ! Thanks

Le ven. 23 juin 2023 à 10:39, InTheta @.***> a écrit :

Hey, Sounds interesting! Would like to see this! Thanks for sharing your knowledge!

— Reply to this email directly, view it on GitHub https://github.com/ClementPerroud/Gym-Trading-Env/issues/4#issuecomment-1603936265, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOFZ2ZD4VQI7BT62GBSWSP3XMVI4LANCNFSM6AAAAAAYCE22WE . You are receiving this because you modified the open/close state.Message ID: @.***>

Karlheinzniebuhr commented 11 months ago

recurrent networks

You can use recurrent PPO with sb3-contrib haven't tried it but my vote goes for SB3. It is highly performant on a local GPU with a bit of parameter tuning.

sebnapi commented 10 months ago

recurrent networks

You can use recurrent PPO with sb3-contrib haven't tried it but my vote goes for SB3. It is highly performant on a local GPU with a bit of parameter tuning.

Did you try using it with this environment?

talvasconcelos commented 3 months ago

  1. "Supra-classing" the TradingEnv class with a really basic trading environment class that would be that "BaseTradingEnv" that would be a simple buy/sell (without the idea of position ?) environment. Is that what you were thinking of ?

    1. We need to decide which RL library to recommend. IMO :

    • stable-baselines3 is easy to use but a bit out-dated in terms of agents

    • RLlib is probably the more efficient lib, but it lacks flexibility and looks hard to use I would prefer using RLLIB in tutorial, what do you think ?

I'm interested in this! I'm trying to create RL model but i'd like for it to only buy and sell... long/short seems an overkill to what i'm trying to achieve! Any environment i can look at with just that?

I'm using stable-baselines PPO, BTW!