ZhengyaoJiang / PGPortfolio

PGPortfolio: Policy Gradient Portfolio, the source code of "A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem"(https://arxiv.org/pdf/1706.10059.pdf).
GNU General Public License v3.0
1.74k stars 750 forks source link

failed to download data or train #119

Open franklili3 opened 5 years ago

franklili3 commented 5 years ago

When I download data or train without downloading data, there is a error " File "/home/ubuntu/PGPortfolio/pgportfolio/marketdata/globaldatamatrix.py", line 199, in __fill_data if c["date"] > 0:TypeError: string indices must be integers". What can I do?

franklili3 commented 5 years ago

I have get the trade paires and volumes." pair price volume coin
ETH BTC_ETH 0.035572 833820.402758 LTC BTC_LTC 0.011680 669757.028884 XRP BTC_XRP 0.000082 465788.871570 reversed_USDT USDT_BTC 0.000263 200241.369072 ETC BTC_ETC 0.001111 165836.536867 DASH BTC_DASH 0.021320 145825.768286 XMR BTC_XMR 0.012554 82908.949198 XEM BTC_XEM 0.000011 69047.989435 FCT BTC_FCT 0.001690 63216.478905"

hn2 commented 5 years ago

I have the same problem pgportfolio\marketdata\globaldatamatrix.py", line 199, in __fill_data if c["date"] > 0: TypeError: string indices must be integers

ghost commented 5 years ago

I made it work by cloning the dev repo, change the exchange in config file to binance, changing the dates in base config and train_package NNAgent config to a shorter frame such as 2019/1 -> 2019/3

JayWu7 commented 5 years ago

This is the reason and solution: {'error': 'Data requested is too large. Please specify a longer period or a shorter date range, or use resolution=auto to automatically calculate the shortest supported period for your date range.'}

atangfan commented 5 years ago

Same here. How to set resolution to auto? No resolution field in net_config.json. In https://github.com/ZhengyaoJiang/PGPortfolio/blob/master/user_guide.md, author mentioned that

Note that using the this file, you shouldn't make any changes to input data configuration(For example start_date, end_date or coin_number) otherwise incorrect result might be presented.

but looks like I can only shorten the start_date and end_date in the config to run the download. (I tried 2019/1/1 -> 2019/3/1 which is downloadable)

johannes73 commented 5 years ago

but looks like I can only shorten the start_date and end_date in the config to run the download. (I tried 2019/1/1 -> 2019/3/1 which is downloadable)

Same here. I also tried shortening the period, and it worked. What would you think would be the best way to download for a longer period? To download manually multiple times, each time for a short period?

JayWu7 commented 5 years ago

Same here. How to set resolution to auto? No resolution field in net_config.json. In https://github.com/ZhengyaoJiang/PGPortfolio/blob/master/user_guide.md, author mentioned that

Note that using the this file, you shouldn't make any changes to input data configuration(For example start_date, end_date or coin_number) otherwise incorrect result might be presented.

but looks like I can only shorten the start_date and end_date in the config to run the download. (I tried 2019/1/1 -> 2019/3/1 which is downloadable)

If you follow the code, you will find that there is a update_data method in the globaldatamatrix.py which will alter the end and period. The reason I think is that the local database have the former data and it will influence the parse of the current data you want to training. This is a bug of this code I think.

Solution: Just delete the Data.db, then training the data online!!! or make sure the data of the asset that u want training is not appeared in the Data.db.

This is my thinking, and it run well in my machine, hope you guys will have a better way to solve this.

JayWu7 commented 5 years ago

but looks like I can only shorten the start_date and end_date in the config to run the download. (I tried 2019/1/1 -> 2019/3/1 which is downloadable)

Same here. I also tried shortening the period, and it worked. What would you think would be the best way to download for a longer period? To download manually multiple times, each time for a short period?

After doing the things I said above, I can train the period as long as I want.

pbezz1 commented 5 years ago

Hi everyone, can anybody tell me where I can get the data to run this repository please? I'm using "python main.py --mode=download_data" and it gets stuck after 121K.

Thanks.

myknowntime commented 5 years ago

Can anyone offer a complete Data.db? I think the main cause of this error is due to my internet connection. Even I followed the instructions mentioned above.

dexhunter commented 4 years ago

there is one data.db at release

0xdarkman commented 4 years ago

I have experienced this TypeError "TypeError: string indices must be integers" with recent dev branch repo. My json config market is set 'binance'. So, I spotted there is market default hardcoded to poloniex in datamatrices.py in DataMatrices class in init constructor.

This market value is not defined so it is not overwritten in main.py in line 71. there shall be market value added to read from config.

yyqxnice commented 3 years ago

Hello everyone, can anyone download the data from 2018 to 2020? I may not be able to download it due to network problems. If someone downloads it, can you share it, thank you.