search

OpenDriveLab / DriveLM

[ECCV 2024 Oral] DriveLM: Driving with Graph Visual Question Answering
https://opendrivelab.com/DriveLM/
Apache License 2.0
870 stars 58 forks source link

Are test_eval and test_llama the same data? #109

Open Camellia-hz opened 3 months ago

Camellia-hz commented 3 months ago

Dear Author, Hello, When I followed the data preparation in the challenge/readme documentation, I realized that test_eval.json and test_llama.json are essential the same data (derived from test.json), and if I train my model using test_llama.json and then generate the output.json, and then ultimately evaluate it according to the documented methods (using output.json and test_eval.json), wouldn't that be equivalent to assess my model with the training set? Is my understanding wrong?

Camellia-hz commented 3 months ago

@DevLinyan

Camellia-hz commented 3 months ago

@ChonghaoSima

DevLinyan commented 3 months ago

The files test_eval.json and test_llama.json contain the same data but in different formats. The evaluation can only be conducted using the specific format in test_eval.json.

Camellia-hz commented 3 months ago

Thanks for your reply, if so is the evaluation valid? Because I am using test_llama.json to train my model, if I then use test_eval.json to evaluate it, what about the training set and validation set are the same?

Camellia-hz commented 3 months ago

@DevLinyan

ChonghaoSima commented 3 months ago

Not sure what you mean by "the training set and validation set are the same".

The evaluationo is valid as long as you use our provided test file and submit to our official test server.