Open pengzhi1998 opened 3 weeks ago
Hi @pengzhi1998 , interesting findings. Here are my thoughts:
Different loss across two experiments: the loss is calculated by the negative log probability of the GMM head, and the variance of the GMM will significantly influence the log probability. I guess the learned policies among these two experiments have different final mean & variance, thus the loss is so different.
In lifelong learning, yes generally the results on previously learned tasks are worse than the latest tasks. You can see that the loss of the latest task is always the lowest among the 10 learned tasks. However, for decision-making tasks, the success rate is NOT always proportional to the loss because the decision-making process is a sequential process. Easier tasks may remain high success rates even with a higher loss. Besides, the task order may also have some influences. This is one of the reasons why we build this LIBERO benchmark for the community to study why there is a mismatch between the loss and success rate.
Hope these can answer your concerns.
Thank you, Chongkai, for your reply and clear explanation! @HeegerGao
Thank you so much again!!
Dear Authors,
Sorry to bother you again. I have a question regarding the performance evaluation on the
libero_object
dataset. I ran LIBERO with the following command:I conducted two evaluations with slightly different configurations. First Evaluation Configuration:
Results of the 10 tasks:
Second Evaluation Configuration:
Results:
I'm confused about two points:
Best regards, Pengzhi