Weird results from predict_dt

[orih711]

Hey,

I'm trying to use predict_dt with a DBN I trained and as a result I get an almost constant value. I looked at the result data and for some reason it has constant values even for the features I'm not trying to predict (which I'm assuming is the problem). Did you ever come across something like that?

I tried it for a large number of samples for training and testing but it also happens for the following script that only uses one sample: df <- read.csv("C:\Temp Data\1.csv") sample_data_dt <- as.data.table(df) dt_train <- sample_data_dt[1:300] dt_val <- sample_data_dt[301:454]

obj <- c("glideslope_error_deg_t_0") net1 <- learn_dbn_struc(dt_train, size) f_dt_train <- fold_dt(dt_train, size) f_dt_val <- fold_dt(dt_val, size) fit1 <- fit_dbn_params(net1, f_dt_train, method = "mle-g") res <- suppressWarnings(predict_dt(fit1, f_dt_val, obj_nodes = obj, verbose = T))

A small part of "res":

[dkesada]

Hi, I'm afraid to say that I haven't seen this particular behaviour before. The code you provided seems to be correct, so I'm inclined to think that there's some kind of issue with the training data underneath. By the way, can you show me a screenshot of the network structure with plot_dynamic_network()? This behaviour could be explained if none of the nodes in t_0 are connected to any previous nodes, and so they always predict their mean. But this is an extremely rare scenario. The parameters of the nodes in t_0 could also give us some insight on what is going on underneath when performing predictions.

From the information you provided, I can give you a couple suggestions on how could you obtain different results:

• I do not know what value you used for the size parameter, but changing it could maybe offer you more consistent results. Values of 2, 3 or 4 tend to be enough for most cases. There were some very rare cases in the past where networks trained with the DMMHC algorithm returned weird results for size 3, but those cases should be fixed by now.
• Changing the structure learning algorithm will return different DBNs, which may be worth trying to see if the problem is network related or not. Using learn_dbn_struc(dt_train, size, method="psoho") or learn_dbn_struc(dt_train, size, method="natPsoho") will give you a different structure from the default DMMHC algorithm.

Let me know if any of this suggestions get you better results. Also, If you provide me a minimal reproducible example, I'll be able to take a closer look at it to see what's going on.

[orih711]

Thank you for your quick response!

I tried different values for size (2,3,4) all return this weird thing and I wasn't able to try the other two methods for structure learning - "psoho" makes R choke and "natPsoho" returns this error:

I actually think the problem is wrong data type or something like that - the image of res I attached in the original post is of features I'm not trying to predict - so why wouldn't them match the data?

As for your other questions - the structure I get for different sizes (well connected as you can see and all return the constant guess):

The coefficients are extremely small (for anything that's not the same type of node):

But, I still think the issue is somewhere else (a data type thing) - I tried to predict nodes that have significant coefficients to other nodes and still got the same result of constant guess. Again, I think that for some reason that I do not understand the model try to predict all the features instead of the features I actually asked it to predict or something else that's weird.

[dkesada]

Regarding the predictions, the behaviour of predicting all variables in t_0 is normal for the model. When you use the predict_dt() function, you predict the values of all the variables in t_0 using the values of the variables in t_1, t_2 and so on. This is the default behaviour now in dbnR because using the values of other variables in t_0 while predicting some variables in t_0 is introducing look ahead bias. In other words, you would be using information from the future to predict that same future, which would be imposible in a real world scenario. Therefore, all variables in t_0 are forecasted simultaneously regardless of the objective variables that you set. However, there is the lookahead argument which you can use to avoid this behaviour. If you use predict_dt(fit1, f_dt_val, obj_nodes = obj, verbose = T, look_ahead = T) then only the variables you set in obj_nodes will be predicted and all others will use the values in the dataset. That will return the results that you are looking for, just be careful with the look ahead bias.

As for the other two learning algorithms failing while the DMMHC works, that is concerning. All 3 algorithms should be interchangeable, unless there is some problem with the dataset, so this probably is the issue. Keep in mind that your data has to be numeric, because dbnR does not work with categorical data. You cannot have constant columns or duplicated columns either, so please check that there are no problems of that kind in your dataset. I will add checks for this things inside function calls in the future, because they are a common ocurrence in the issues with dbnR. In the meantime, you could maybe try to use only a few of the variables in your dataset (like the "glideslope_error_deg" variable that you showed in your first post, that one looks ok) and then try to train the model to see if the problem persists. Adding some white noise with rnorm() to the columns in your dataset can also be a possible solution, but it would be best to first figure out the reason why it is not working properly. By the way, what version of dbnR are you using? Just to rule out other possible errors.

[orih711]

Thank you so much! There were a few columns that made the model pretty angry (not just constant value ones, also columns with huge\tiny values because of measurement equipment hiccups) - once they were gone it performed perfectly!

This is obviously over-fitted - the full data set has WAY MORE variance but at least everything functions as it should now!

In case that is still relevant for you to know the dbnR version I'm using is 0.7.8

[dkesada]

Great news! I'll close the issue then. As a final note, keep in mind that the predict_dt() function only makes predictions of the next instant for each row and so it appears like a very good fit, but this can be misleading. Normally, when you perform forecast with the forecast_ts() function, the performance degrades the longer you predict into the future.