search

CityBrainChallenge / KDDCup2021-CityBrainChallenge-starter-kit

75 stars 40 forks source link

`delay_index` calculated by `evaluate.py` only consider the vehicles on the road net #43

Closed fstqwq closed 3 years ago

fstqwq commented 3 years ago

process_delay_index method only consider the output of environment, while the environment deletes all vehicles finished journey in the output.

It may be a critical issue, since the evaluation may be exploit by the following method: stuck all the vehicles with small delay index when it's going to evaluate delay, in order to decrease the mean of all vehicle in the road net.

zhyliu00 commented 3 years ago

Thanks for your comment. For process_delay_index, it use info_step *.json to calculate the delay index, rather than obs['info']. For info_step *.json，it contains all the vehicles (including those vehicles that have left the road network).

fstqwq commented 3 years ago

Sorry for that. Since we noticed that process_delay_index has a parameter called step, while it never be used at all, we misread the logic and now it's clear.