PR #14: Implementation for issue #32

[harveyvn]

• Setup experiments to address noises in simulations:

  • Run simulations one time only
  • Run simulations a fixed amount of time (e.g.,10) and take average/mean
  • Run simulations a minimal amount of times (e.g., 3/5) and then use pValue and A12 to decide whether the individuals are different, the same, and which one is better

• Add a penalty to a fitness score when the damage components are not existed in the police report.

• Add a threshold to define whether a damage component deserves our revision.

• Add function to count the number of conditions that the simulations match to the police report and give them a target score.

[alessiogambi]

@harveyvn I struggle a bit to understand these two points:

Add a penalty to a fitness score when the damage components are not existed in the police report. Add a threshold to define whether a damage component deserves our revision.

Can you clarify them a little?

[harveyvn]

@harveyvn I struggle a bit to understand these two points:

Add a penalty to a fitness score when the damage components are not existed in the police report. Add a threshold to define whether a damage component deserves our revision.

Can you clarify them a little?

I noticed that the crash components sometime doesn't exist in the police report. @TriHuynh00 suggests that we might deduct a simulation's fitness score in such cases to make the fitness score become more relevant to the police report.

Regarding to a threshold, the damage value of crash components are sometime extremely small. In the previous AC3R, we didn't consider it as a crash component.

[alessiogambi]

I noticed that the crash components sometime doesn't exist in the police report. @TriHuynh00 suggests that we might deduct a simulation's fitness score in such cases to make the fitness score become more relevant to the police report.

So if the police report does not mention how cars were damaged any crash would work for us or better it does not matter in the fitness function. So your suggestion is to computer an optimal fitness value which is lower that the usual? Sounds good to me. So we probably have cases of different precision: no crashed component, only front/back or side, and very precise descriptions like the front left side.

I'm this case probably we should be able to count the number of conditions that we can match and give them a target score. Then we sum them up only if they are mentioned in the report. Nice

Regarding to a threshold, the damage value of crash components are sometime extremely small. In the previous AC3R, we didn't consider it as a crash component.

OK but what is the solution? A minim value on damage ti be considered a crash? Would it be possible to consider car's bounding boxes and say whether a crash happened?

[harveyvn]

I'm this case probably we should be able to count the number of conditions that we can match and give them a target score. Then we sum them up only if they are mentioned in the report. Nice

I updated cal_fitness to count the number of conditions that one crash scenario matches to a police report. After crashed simulation finishes, there are 2 special situations as below which I need your advices:

• A vehicle has some damage parts. If there are any damage components not found in the police report, the score will be deducted by one.
• A vehicle doesn't have any damages. However, there are some damage components existed in the report, the score will be deducted by the total damaged parts in the police report.
• A function _calculate_score_police_report computes an expected score for the specific scenario. You can check a Case 06 report, its score will be 4 including 1 for crashed scenario and 3 for the total number of damage components.

OK but what is the solution? A minim value on damage ti be considered a crash? Would it be possible to consider car's bounding boxes and say whether a crash happened?

We don't use this value to be considered a crash. At the moment, a value is 0.05 to indicate whether the component should be counted in the cal_fitness or not. I think about this factor because in the previous AC3R's summary, we use the symbol U to mention the case that we cannot measure the level of damage of one vehicle since it is really small. Syntax is given below:

Structure: [caseName]:1-[MatchValue]:2-[MatchValue] Example: Case0:1-PS:2-U MatchValue will be:

1. P: Match a position in a report
2. S: Match a side in a report
3. PS: Match a position and a side
4. U: Cannot measure due to the damage value is extremely small

[alessiogambi]

I am not entirely convinced by this approach to deduct point for unexpected damaged component. I am. In favor to penalize wrong simulations but if the police report is under specified any crash would do. Please put this on hold till the next meeting. Better to discuss it face to face

[alessiogambi]

Below the snapshots for todays meeting.