About the data of endipointerror avg

[SzuPc]

Thank you for your last reply. I successfully reproduced the endpointerror values of Control and CSRT. The trend of the curve is similar to the figure in the SIRT paper, but the value is 2.2-2.3 times higher than that in the paper. I think Do you know if other processing has been done? Below is my code for numerical processing.

data_csrt = {}
data_mft ={}
for filename in os.listdir(folder_path_csdt):
    if filename.endswith('.json'):
        file_path = os.path.join(folder_path_csdt, filename)
        with open(file_path, 'r') as file:
            json_data = json.load(file)
            data_csrt.update(json_data)

for filename in os.listdir(floder_path_mft):
    if filename.endswith('.json'):
        file_path = os.path.join(floder_path_mft, filename)
        with open(file_path, 'r') as file:
            json_data = json.load(file)
            data_mft.update(json_data)

x = []
y = []
x1 = []
y1 = []
y_csrt = []
y_mft = []
for key, value in data_csrt.items():
    if key != "total":  
        x.append(value["endpointerror_videotime"])
        y.append(value["endpointerror_control"])
        y_csrt.append(value["endpointerror_CSRT"])

for key, value in data_mft.items():
    if key != "total":  
        x1.append(value["endpointerror_videotime"])
        y1.append(value["endpointerror_control"])
        y_mft.append(value["endpointerror_MFT"])

sorted_data = sorted(zip(x, y, y_csrt), key=lambda d: d[0])
sort_mft_data = sorted(zip(x1, y1, y_mft), key=lambda d: d[0])
x_sorted, y_sorted, y_csrt_sorted = zip(*sorted_data)
x_1, y_1, y_mft_sorted = zip(*sort_mft_data)

data_avg_con = []
data_avg_csrt = []
data_avg_mft = []

sum_control = 0 
sum_CSRT = 0
sum_MFT =0

for i in range(len(x_sorted)):
    if x_sorted[i] >=10:
        break
    sum_control = sum_control + (y_sorted[i])
    avg_con = sum_control / ((i+1))

    sum_CSRT = sum_CSRT + (y_csrt_sorted[i])
    avg_csrt = sum_CSRT / ((i+1))

    sum_MFT = sum_MFT + (y_mft_sorted[i])
    avg_mft = sum_MFT / ((i+1))

    data_avg_con.append(avg_con)
    data_avg_csrt.append(avg_csrt)
    data_avg_mft.append(avg_mft)

x_sorted = x_sorted[0:i]

plt.figure(figsize=(18, 6))  
# plt.scatter(x_sorted, data_avg, color='red', label='endpointerror_control', s=5)
# plt.scatter(x_sorted, y_csrt_sorted, color='red', label='endpointerror_MFT', s = 5)
plt.plot(x_sorted, data_avg_con, color='red', label='endpointerror_Control')
plt.plot(x_sorted, data_avg_csrt, color='green', label='endpointerror_CSRT')
plt.plot(x_sorted, data_avg_mft, color='blue', label='endpointerror_MFT')‘

This is the experimental result I ran. I also want to ask if you will disclose the code of the SENDD model. If so, it will be of great significance for us to participate in the challenge.

[athaddius]

At a glance, the code looks good to me. The codebase here is a re-write of that in the paper, so there are bound to be some differences (along with a less filtered dataset). We also average over number of tracks, not videos in the paper from what I remember. Both of these together could add to a factor of difference. That all said, a factor of ~2 is large, so I'll have to dig up the old code and look through it (might be something like a scale change from initial data to that which we released).

For now, the relative ordering from your figure looks good (as in MFT<CSRT<Control), so evaluating comparative performance should be fine.

Regarding SENDD, we will be unable to release the code for this framework, but we note that since SENDD is a framewise tracker (scene flow) without drift correction, we expect methods that re-localize on top of framewise trackers, such as MFTs, to outperform on this challenge.

[athaddius]

I'm going to close this for now since there are other tasks that are taking up my time for the STIR challenge preparation in the meantime.