How should I replace my own dataset for the experiment？

[weican1103]

Cool, I find your work to be very fascinating. I am currently in the process of revising a paper and need to add a comparative experiment. My dataset is in the .mat format (train.mat and test.mat), and it consists of 11 categories. For each category being considered as unknown, there is a corresponding set of train and test data; for instance, when the 0th category is unknown, there is one set of train and test data, and when the 1st category is unknown, there is another set of train and test data. I am interested in knowing if I can effectively integrate my own dataset within your framework. Secondly, I would like to understand if it's possible to appropriately customize the labels for known and unknown categories within the code.

[Vanixxz]

Thanks :).

1. Custom Dataset: The SVHN dataset used in our experiments is also in .mat format. You can refer to the 'dataloader' for loading this dataset to load your custom data (_dataset/osrloader.py 95-142).

2. The information about of split (known classes) is recorded in lines 103-226 of the file 'misc/util.py'. You can create the list of known classes in a similar format, and the remaining classes in the dataset will be considered unknown.

Feel free to contact me if you have any further questions.

[weican1103]

I apologize, there's a part that I didn't understand.

Why are split = auroc and split = F1 in the form of matrices? My task assumes one class as unknown while the rest of the classes are known. How can this issue be resolved? Thank you very much. Regarding evaluation metrics, is it possible to print the test labels during output? What I mean is, during the testing phase when an image is input, is it possible to print what the predicted label is after passing through the network? Thank you very much for your help.

[weican1103]

When defining my dataset, how do I define class SVHN_Filter(SVHN):?

[Vanixxz]

Glad to help you!

1. The experiments conducted on split_F1/split_AUROC are averaged over five scenarios. Therefore, there are five different lists of known classes for each dataset, where each list contains label numbers representing the known classes. Taking your task as an example, you can consider the list of known classes to be [0], and the corresponding unknown list would be generated as list(set(list(range(0, 11))) - set(known)).

2. You could print the variables (pred / score) in test.py lines 94-103 to check the performance of model on test samples.

3. If you don't need to conduct experiments on SVHN, you can disregard it. I just thought that if your data format is also .mat, you could refer to this code to create your own dataloader for custom dataset.

[weican1103]

Thank you very much for your assistance; I have completed the task of loading my dataset. However, I still have a question regarding the specific meanings of the evaluation metrics you provided: print('Accuracy: {:.3f}, ' 'AUROC: {:.5f}, ' 'AUPR_IN: {:.5f}, ' 'AUPR_OUT: {:.5f}, ' 'Macro F1-score: {:.5f}'.format(macro_f1), ) Are these metrics calculated based on five experimental scenarios, and if I only need to conduct one experimental scenario, how should I modify it? Furthermore, I need to print out the real and predicted labels for both the test set and the unknown set during the testing phase. Could you please provide me with detailed instructions on how to do this? Once again, I am very, very thankful for your help.

[weican1103]

Hi，我想询问下 1.这个代码里面的labels_open是表示未知集真实标签，pred2是预测的未知集标签，labels_close是已知类真实标签，pred1是预测的已知类标签吗？ 2.我在实验中发现，未知类的真实标签都是-1，但测试出来的未知类是各种类别标签，唯独没有-1，请问，这要如何判断我正确的预测了未知类？

[Vanixxz]

1. The print values represent the results in this scenario. The loop iterating through five scenarios is at the outermost level. If you only need to experiment in one scenario, a single output will serve as the experimental result.

2. During the training phase, unknown classes are not learned. The predicted labels for what you referred to as "unknown prediction" are incorrect. We will reprocess the predicted labels based on values of score function. The sample with score below the threshold will be assigned -1.