search

AlbinSou / ocl_survey

Code for "A Comprehensive Empirical Evaluation on Online Continual Learning" ICCVW 2023 VCL Workshop
https://arxiv.org/abs/2308.10328
MIT License
31 stars 9 forks source link

Error in computation of results #6

Closed hiteshvaidya closed 9 months ago

hiteshvaidya commented 9 months ago

I trained the experiment and was analyzing the results with the ipynb files in notebooks folder and following is what I get,

frames = extract_results("/data/hvaidya/ocl_survey/results/er_split_cifar100_20_2000/")
print(frames)
df = frames["training"]

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
[/data/hvaidya/ocl_survey/notebooks/results.ipynb](https://vscode-remote+ssh-002dremote-002bgaivi-002ecse-002eusf-002eedu.vscode-resource.vscode-cdn.net/data/hvaidya/ocl_survey/notebooks/results.ipynb) Cell 3 line 1
----> [1](vscode-notebook-cell://ssh-remote%2Bgaivi.cse.usf.edu/data/hvaidya/ocl_survey/notebooks/results.ipynb#W2sdnNjb2RlLXJlbW90ZQ%3D%3D?line=0) df = compute_average_forgetting(df, 20, base_name="Top1_Acc_Exp/eval_phase/valid_stream/Task000/Exp")
      [2](vscode-notebook-cell://ssh-remote%2Bgaivi.cse.usf.edu/data/hvaidya/ocl_survey/notebooks/results.ipynb#W2sdnNjb2RlLXJlbW90ZQ%3D%3D?line=1) df = compute_AAA(df)

File [/data/hvaidya/ocl_survey/src/toolkit/post_metrics.py:79](https://vscode-remote+ssh-002dremote-002bgaivi-002ecse-002eusf-002eedu.vscode-resource.vscode-cdn.net/data/hvaidya/ocl_survey/src/toolkit/post_metrics.py:79), in compute_average_forgetting(dataframe, num_exp, base_name, name)
     77 for i in range(num_exp):
     78     mname = base_name + f"{i:03d}"
---> 79     dataframe = compute_forgetting(dataframe, mname)
     80     metric_names.append(prefix + mname)
     82 df = compute_average(dataframe, metric_names, name)

File [/data/hvaidya/ocl_survey/src/toolkit/post_metrics.py:28](https://vscode-remote+ssh-002dremote-002bgaivi-002ecse-002eusf-002eedu.vscode-resource.vscode-cdn.net/data/hvaidya/ocl_survey/src/toolkit/post_metrics.py:28), in compute_forgetting(dataframe, metric_name, prefix)
     25 df = dataframe.sort_values("mb_index")
     27 if "valid_stream" in metric_name:
---> 28     raise NotImplementedError(
     29         "The compututation of forgetting on continual metric streams is not supported"
     30     )
     31     df = decorate_with_training_task(
     32         df, base_name="Top1_Acc_Exp/eval_phase/valid_stream/Task000/Exp"
     33     )
     34 else:

NotImplementedError: The compututation of forgetting on continual metric streams is not supported

can you help me with this?

AlbinSou commented 9 months ago

This notebook is not correctly updated, thanks for noting this. It was just a test that I used to display first version of forgetting metric as a curve. However, as you can see from the error, I dropped support for forgetting computation from validation stream. The reason is that I did not use the forgetting metrics on the continual stream but only at task boundaries. So it can be computed from the test stream but not from the validation stream.

In this particular case, replacing "valid_stream", by "test_stream" should work

df = compute_average_forgetting(df, 20, base_name="Top1_Acc_Exp/eval_phase/test_stream/Task000/Exp")

The advised way to get the results is to use the get_results.py script under scripts/ folder.

I don't think I directly included the notebook to create the accuracy curves since they were similar to the example ones that I gave. I can add them if you want, but the notebooks I updated are just meant as a playground to load the results dataframes and plot some metrics based on that.

hiteshvaidya commented 9 months ago

thank you! get_results.py worked for me. Also, I realized my training hadn't finished which is why frames variable was empty