Compare model and loss function

[star77sa]

Hello, I’m leaving an issue as I have a few questions.

While tuning the parameters of PDE Loss and Mono Loss, I set Alpha=0 and Beta=0, and the performance turned out to be better than what was reported in the paper for Batch 1 of the XJTU Dataset. (MAPE=0.0065, RMSE=0.0087 (seed=42))

With Alpha and Beta set to 0, it would practically be almost identical to the comparison model, MLP. However, since the results came out better than the performance reported in the paper, I reviewed the comparison model’s code and found two issues:

1. It was using only half the data compared to the proposed model.

2. Even if early stopping was set to a different value, it was inherently limited to 10.

I believe this leads to an unfair comparison in the experiments. Was this considered? Also, with Alpha and Beta set to 0, the performance on XJTU batch=1 was better than what was reported in the paper, so I'm not sure if PDE Loss and Mono Loss are actually contributing.

Additionally, it seems that in a previous issue, Mono Loss was changed from using the True SOH value to the Estimated SOH value.

However, when using the revised Estimated SOH, the performance does not match the paper’s results. If I use the True SOH, which was considered a code error, the paper's performance can be reproduced.

I would like to know your thoughts on these issues.

Thank you for your attention to this matter.

[wang-fujin]

Thank you for your attention. For question 1: We did not use only half of the data. In the dataloader, there is an overlap between the two items. For example, dataloader[1] includes (x1,x2,y1,y2), and dataloader[2] includes (x2,x3,y2,y3). Therefore, in the proposed model, loss1 is calculated twice for each $x_i$, and all have a coefficient of 0.5. The comparison method only calculates it once. For question 2: This is a small issue. At the beginning, we gave the value of early_stop in the train function. When we reorganized the code later, we put it in args, but ignored the modification of the train function. In fact, the early_stop value of all models is set to the same, and the results are consistent with those in the paper. We re-ran the experiment to confirm it again. In addition, you said that the results of our method and the comparison MLP are not much different, which is indeed the case (such as Table 2 in the paper, especially in the MIT dataset and HUST dataset). But in this paper, we are more concerned about variance, as paper's title says stable modeling. Although the MLP method has a chance to produce results that are similar to ours, it has a higher variance (as shown in Figure 4 in the article).