ageron
commented
3 years ago Hi @ChrisChiasson ,
Great catch, thanks a lot! I believe you are referring to the naive baseline in the "Forecasting Several Steps Ahead" section. I'm not sure why I defined it as Y_naive_pred = Y_valid[:, -1:], I'm guessing I first copied the naive prediction from the first section (y_pred = X_valid[:, -1], which was correct) but then something went wrong, it really makes no sense at all.
Recall that in this section we are trying to forecast t+1, t+2, ..., t+10 based on t-n, t-n+1, ..., t-1, t. In other words, we are trying to predict 10 values at once, based only on past values. We should not be using any future values to make our predictions, which means we should not be using Y_valid at all (as my code was doing), only X_valid.
The most naive prediction for this use case is simply repeating the last known value 10 times (it's like predicting that the temperature for the next 10 days will be equal to today's temperature). Here's a simple way to implement this:
Y_naive_pred = np.tile(X_valid[:, -1], 10)This takes the last time step from each sequence (X_valid[:, -1]) and repeats it 10 times (np.tile(..., 10)). Let's check that it works well for the first sequence:
>>> X_valid[0, -1]
array([-0.24104503], dtype=float32)
>>> Y_naive_pred[0]
array([-0.24104503, -0.24104503, -0.24104503, -0.24104503, -0.24104503,
-0.24104503, -0.24104503, -0.24104503, -0.24104503, -0.24104503],
dtype=float32)Looks good! Now we can compute the error:
>>> np.mean(keras.metrics.mean_squared_error(Y_valid, Y_naive_pred))
0.25697407Luckily, it's not too far from the value I got when using the wrong code! So the neural net really is 10x better than the naive prediction.
Now regarding the first solution you propose:
actual_Y_naive=np.array([np.concatenate([x,y]) for x,y in zip(X_valid[:,-1],Y_valid[:,:-1])])In this case you are using Y_valid to define the naive predictions (like my code was), but that's incorrect as explained above: it uses future values to predict future values (it's kind of cheating, which explains why the error is so low!). Specifically, it's predicting t, t+1, t+2, ..., t+9.
Regarding the second solution:
Y_naive_pred = Y_valid[:, -2]
np.mean(keras.metrics.mean_squared_error(Y_valid[:,-1], Y_naive_pred))Again, this code uses Y_valid to define Y_naive_pred, which is not allowed. Moreover, it's only predicting a single value, but in this section we're trying to predict the next 10 values at once.
Lastly, the third solution:
Y_naive_pred=X_valid[:,-1,0]
np.mean(keras.metrics.mean_squared_error(Y_valid[:,0],Y_naive_pred))This code only predicts a single value, when we want to predict 10 values at once. At least it's not using future values, though.
I'll fix the notebook now.
Thanks again for catching this error! 👍
ChrisChiasson
The naive prediction in Chapter 15 reproduced below is incorrect. It also raises a visual red flag because the error from the network was about 10x smaller (about 0.027). Quoting from Chapter 15, https://github.com/ageron/handson-ml2/blob/master/15_processing_sequences_using_rnns_and_cnns.ipynb
An accurate way to calculate the naive prediction error would be as follows. The naive error is actually smaller than the RNN walk forward error. This makes sense because it is actually very hard to beat the naive prediction, even when walking forward with real data as the notebook did. (For example, 70% of the naive error would be excellent. 10% of the naive error is ludicrous.)
The main problem with the code in the existing notebook is that it is not dimensionally consistent. Here are some accurate ways to calculate subsets of the naive error to make it easier to see.
This is just using the last validation value vs. the 2nd to last validation value.
This is just using the first validation value vs. the last value in the input.