Problems with the evaluation part

[shahaamirbader]

Hi,

I am trying to run the evaluation using the commands provided on the readme. Seems like there is no config file in the checkpoints folder, just the model.pt and model_best.pt. Here is the list of all files in the directory

When I try to run the Eval_cdt.py I get the following error where it says that config file location is not a directory. What am I doing wrong here?

The path to my model and best_models are "/home/xubuntu/Desktop/homelocal/3rd Year/DT_RL/OSRL/logs/OfflineCarPush1Gymnasium-v0-cost-5/CDT_cost5-eea0/CDT_cost5-eea0/checkpoint/mode.pt "

I even tried to give the path argument as you have mentioned in the command in your readme but still the same issue. See the below screenshot please.

In this screenshot I entered the model path inside the eval_cdt.py file and hence path argument is not used here

[Ja4822]

@shahaamirbader Hi, there should be a config.yaml in this directory: /home/xubuntu/Desktop/homelocal/3rd Year/DT_RL/OSRL/logs/OfflineCarPush1Gymnasium-v0-cost-5/CDT_cost5-eea0/CDT_cost5-eea0/ if you didn't comment out this line.

To evaluate the trained model, run this command:

cd <directory>/OSRL
python example/eval/eval_cdt.py --path logs/OfflineCarPush1Gymnasium-v0-cost-5/CDT_cost5-eea0/CDT_cost5-eea0/

you may also need to comment out these two lines. Sorry for the inconvenience, we will fix these bugs regarding to gym in our future updates.

[shahaamirbader]

Thanks. This helps. I was finally able to run the evaluation on the trained model for Car-cirlcle. Here is the output. I see in one of your issue response that you used a TR550.0 for Car-circle. I also included a TR550.0 in the eval_cdt.py. But the results do not seem to match Table 1 results. Can you help me understand how can this result be checked against the benchmarks results that you show in the paper? The normalized cost is somewhat close but the reward is not.

I tried with other datasets and seems like the results in terminal are not matching those of the paper

Looking forward for the response.

[liuzuxin]

Which paper's results are you referring to? The datasets we released are for the benchmark paper, where we average the results w.r.t three seeds and three different target cost thresholds. Regarding the original datasets used in the CDT paper, please check this issue.

[shahaamirbader]

Thanks, I was referring to the CDT paper, table 1. I wanted to replicate the results shown in table 1 for the dataset. Let me have a deeper look at the issue you shared. Closing it for now.