question about Evaluation

[kairikibear]

Hi again.

I want to know what kind of evaluation you did, specifically on ucsd ped 2

In #1 , you mentioned that figures / metrics shown in the paper are calculated using the windowed approach, which i assumed is what takes place in task 8 in main.py

In function find_max_patch inside utils.py, i see that you basically:

1. Grab the squared distance of flow and rgb channel
2. Window both distance and calculate for each window its respective mean and std
3. Aggregate the computation based on max flow distance mean and max flow distance std

The end result is for a pair of flow and rgb distance matrix is :

the mean of flow distance the mean of rgb distance the std of flow distance the std of rgb distance

for each mean and std aggregation.

then you computed the weights based on training score and get the scores for each frames using eq 10 in your paper

but after that, you didnt only use the combined score, but you concatenated it the the result of find max patch

then you normed and calculate AUC and AUPR for both normed and non-normed data

the big question is, which of these values (there are 12 in total) do you actually use in the paper? do you simply just take the max?

[nguyetn89]

Hi @kairikibear,

The code is ambiguous because I tried different ways of score calculation and did not remove them later. Only the selected way was described in the paper.

In details, the function full_assess_AUC_by_max_patch has a default parameter save_roc_pr_idx=8 that indicates the 8th score in the variable full_scores. This score is calculated according to the weighted combination of 0th and 3th loaded patch scores. Referencing to the function find_max_patch, they correspond to max_val_mean (maximum average score of a motion patch) and mean_appe_1 (average score of appearance patch at the same position). The other scores as well as the non-normed combination scores were just to provide a personal assessment for the selection of score estimation scheme.

In summary, the used score is exactly the one described in the paper. You can simply discard the other unnecessary calculations (because the code is messy as mentioned in the readme), or you can reimplement the function that would not be complicated. By the way, a more-optimized source code with improvements of the method will be published next year (maybe Q1).

[kairikibear]

thanks for the explanation!