RayActorError in FinRL_PaperTrading_Demo when using AlpacaPaperTrading

[NicoleRichards1998]

Thank you so much for creating the FinRL_PaperTrading_Demo, it is so cool!

For the project I'm doing I've been trying to use Ray Tune to tune the rrlib agent parameters and then deploy it to Alpaca and see if I could improve performance. But I've been getting an issue with the AlpacaPaperTrading function when drl_lib="rllib".

I've been looking into this for a week now, but I'm very lost. Any ideas on how I can fix or even approach this would be appreciated.

[Athe-kunal]

Hi @NicoleRichards1998 Can you try this notebook? Here we can do hyperparameter tuning of stock market data using Ray tune. Also, share your results. Due to computational limitations, I was not able to stress-test this one.

[NicoleRichards1998]

I've been looking at that notebook as well, that is actually where I got the idea to use RLlib in the paper trading application. My goal in a way is to combine the two notebooks. The issue comes in when the trained (and tuned) agent is deployed on the live paper trading environment...

[Athe-kunal]

For that we have checkpointing enabled in tune.run(). So tune is checkpointing all the models used for tuning with various hyperparameter configuration. You can get it by analysis.best_checkpoint. Load that model and deploy it to live trading environment. Does this answer your question?

[NicoleRichards1998]

Hi Athe-kunal, thank you for coming back to my issue. Unfortunately I am not even at that step yet. I just trained the model with the default parameters to first sort out AlpacaPaperTrading, but I keep getting stuck. I think the issue is with the StockEnvEmpty that is passed to PPOTrainer, but I can't figure it out from there

[Athe-kunal]

Ohh okay, I haven't tested it Please do one thing, in the notebook for Ray tune hyperparameter tuning, change the data source to Alpaca and time interval to 1Min. Then you can do end-to-end hyperparameter tuning. If you just want to pass the default parameters and not do hyperparameter tuning, then follow these steps

1. In tune.run(), pass your default hyperparameters to config attribute
2. Change num_samples = 1 Then you will imitate RLlib, instead of ray tune. If you get stuck somewhere, let me know

[NicoleRichards1998]

Okay so I changed the notebook as you asked (except the default hyperparameters, because it gets upset when it receives those). Here is the link FinRL_Raytune_with_Alpaca_Paper_Trading.ipynb You can see at the end of the notebook that I get a RayActorError. It's exactly the same one I get when I use a regular rllib training function. Any ideas on how I can fix it?

[Athe-kunal]

Hi @NicoleRichards1998 You are using TensorFlow, right? If you are then in cell 19, please change the "framework": "torch" to "framework": "tf2" inside tune.run()

[NicoleRichards1998]

Hi Anthe, thanks for catching that. I changed it to tf2 but there was not much of a difference. I still get 8 error trails and most importantly I still get the KeyError: 'price_array' when running AlpacaPaperTrading, so I still can't run the live paper trading on Alpaca https://github.com/NicoleRichards1998/FinRL/blob/master/FinRL_Raytune_with_Alpaca_Paper_Trading.ipynb Here is the link for the up to date notebook that I ran this morning

[Athe-kunal]

Hi @NicoleRichards1998 So the tuning using Ray tune is done via StockTradingEnv_np, and while running the AlpacaPaperTrading, you can see the StockTradingEnv_empty. The later environment does not have price_array in its environment initialization. You can either change the StockTradingEnv_empty to accommodate what Ray tune wants in cell 35. Let me know if it works or else we will get in touch to solve the issue.

[NicoleRichards1998]

Hi @Athe-kunal unfortunately it didn't work. When I use the StockEnvEmpty (which works with Elegant RL) I get a RayActorError that says AttributeError: 'NoneType' object has no attribute 'dtype'. This was the original issue that I aksed about two weeks ago, here is the code for that https://github.com/NicoleRichards1998/FinRL/blob/9c8966c12033553d2da2d05dad67af041193bdc6/FinRL_Raytune_with_Alpaca_Paper_Trading_(StockEnvEmpty).ipynb Then I tried rather using StockTradingEnv (as it works during training and testing) but then I get a RayActorError with IndexError: tuple index out of range. I don't think this env will work in this setting as I need to fill it with data beforehand not during trading as with alpaca paper trading. Here is the link to that code if you want to check it out: https://github.com/NicoleRichards1998/FinRL/blob/9c8966c12033553d2da2d05dad67af041193bdc6/FinRL_Raytune_with_Alpaca_Paper_Trading_(StockTradingEnv).ipynb

I'm sorry to keep on nagging on this, but I have no idea how to fix this and I really need it to work to complete my project