Reward doesn't go up

[star277hk]

After each episode, I expect agent learnt something from previous, but I plot the reward of 1000 Episode, the reward doesn't go up. Is it mean the agent can't learn from the dataset ? I used EURUSD 2010 - 2015 total 1567 records with 3 window size

[yeshengyi]

Same here, I think the training process should filter out those with less than expected profits.

[madytyoo]

I would suggest to adjust the normalization function, the implementation computes the sigmoid of returns:

sigmoid(block[i + 1] - block[i])

The returns in the sample stock file range between +/-100. For currencies the returns range is between +/-0.001 if you are using a raw price data set. You could try to express the return in pips:

sigmoid( (block[i + 1] - block[i])*10000)

[MonaTanggg]

@madytyoo sorry tried, but the reward didn't go up after times 10000

[madytyoo]

@MonaTanggg I did a test with a EURUSD data 2009-2017, 1000 episodes logging the reward on Tensor board. I suggest to check your data set.

[star277hk]

@madytyoo did you edit anything from your source code/dataset , I wonder why I got a total different result, may I know your tensorflow version as well ? Not sure which part go wrong. Thanks in advance.

[madytyoo]

@star277hk I did the changes suggested by xtr33me (see: https://github.com/llSourcell/Reinforcement_Learning_for_Stock_Prediction/pull/8). In addition I multiplied the EURUSD data by 1000, in the getStockDataVec function.

vec.append(float(line.split(",")[4])*1000)

As I mentioned the currencies the returns range is between +/-0.001 is small to get a result. I run a test without multiplying the price by 1000 (see below) and I got a result similar to your chart. Cheers

[star277hk]

@madytyoo yes I used xtr33me #8 added times 1000 at vec.append(float(line.split(",")[4])*1000) still got no luck. I used EURUSD ASK 01.07.2010-01.07.2015 dataframe 1 day and window size 50 with tensorflow 1.3. May I know your setting ?

[madytyoo]

I'm using EURUSD BID 01.01.2010-31.12.2017 1 day, window size 10, 1000 episodes, tensorflow 1.3.

[colbyham]

I added all columns of the dataset, high, low, volume, etc. to the input layer and i've been achieving max gain of ~60% over ten years for the GSPC dataset which is not that great but at least it's progress. Average drawdown of around 25-35%. I think adding actions to the agent that are related to how much of your portfolio you should hold, sell, or buy up to, etc would be helpful. In other words, bucketed ranges of sizes of trades based on portfolio and or buying power.

[MonaTanggg]

@madytyoo Strange, already using almost same setting but agent still can't learn. Are you just learning from the close price or added other values? If you don't mine, hope you can share me your project that can be raise reward, I wanna figure out where is the problem, thanks.

[xxdaggerxx]

kind off retarded to post code that doesn't work