Can I do not use the provided training/test setup?

[dby124]

I found there are 6105 audio not used. I am writing to ask if I can split the development by myself to model better. I will make sure that all files having same location id are placed on the same side of the evaluation.

In addition, I also want to ask you how to calculate the model size of SVM model. I build SVM based ASC system by sklearn.svm.SVC() function.

[marmoi]

Hi, Sorry for the late reply. All the files should be used, but of course you can manipulate it as you wish as far as you use the development data for training your model. There is a script"model_size_calulation" where the model size can be calculated

[dby124]

Thanks for your reply. Now I get the split data according to datasets\TAU-urban-acoustic-scenes-2020-mobile-evaluation\evaluation_setup\fold1_train.csv (9775) and fold1_test.csv(4190). Just 13965 files are used. Now, I just want to make sure if the .csv file is correct in evaluation_setup.

In addition, “model_size_calulation” seems only used to calculate the model size of the models built by keras.

I also have another question. I get a unexcepted logloss value (17.) and the accuracy is 48.11% in my own code. The provided data in DCASE website shows logloss: 1.461, accuracy: 46.9. So I am sure that something is wrong in my code. The logloss is calculated by sklearn.metrics.log_loss(). I encode the y_true (The true label) and y_pred (predictions of test data) by one-hot encoding. Can you give me some suggestion.

[dby124]

Thanks for your reply. 

    Now I get the train data and test data from /datasets\TAU-urban-acoustic-scenes-2020-mobile-evaluation\evaluation_setup\fold1_train.csv (9775) and fold1_test.csv(4190). Just 13965 files are used. Now, I just want to make sure if the .csv file is correct in evaluation_setup.

    In addition, “model_size_calulation” seems only used to calculate the model size of the models built by keras. 

    I also have another question. I get a unexcepted logloss value (17.) and the accuracy is 48.11% in my own code. The provided data in DCASE website shows logloss: 1.461, accuracy: 46.9. So I am sure that something is wrong in my code.     The logloss is calculated by sklearn.metrics.log_loss(y_true, y_pred). I encode the y_true (The true label) and y_pred (predictions of test data) by one-hot encoding. Can you give me some suggestion.

      Thank you for you attention again. I am looking forward to your reply soon.

------------------ 原始邮件 ------------------ 发件人: "marmoi/dcase2021_task1a_baseline" @.>; 发送时间: 2021年6月10日(星期四) 凌晨2:39 @.>; @.**@.>; 主题: Re: [marmoi/dcase2021_task1a_baseline] Can I do not use the provided training/test setup? (#2)

Hi, Sorry for the late reply. All the files should be used, but of course you can manipulate it as you wish as far as you use the development data for training your model. There is a script"model_size_calulation" where the model size can be calculated

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[marmoi]

Hi, About the split, if you go to the dcase web page of the task, in the task setup, development dataset there is a table with details of how the split has done and that there are some files that hadn't been used for the train/test split.

Regarding the model calculation, just implement a function that counts all non-zero parameters of your model, regardless of they are trainable or non-trainable.

Finally, about the log_loss, we also use the one from sklearn.metrics. The only thing I can think about is perhaps you are modifying y_pred before using the log_loss function, the predicted probablilities should given to the function as they are returned by the model.

I hope this helps

[dby124]

Hi, marmoi

Thanks so much for your reply. It helps me much. But I still have a question about the log loss.

I know the predicted probablilities should given to the function as they are returned by the model.  However, in my coed, the y_pre is the one-hot encoding of scene classes before using the log_loss() function. Becuase I used a late fusion method to fuse the predicted labels of multiple systems, so we just give the predicted labels instead of probablilities.

From the dcase web page of the task, I found that submission systems will be ranked by macro-average multiclass cross-entropy (Log loss) (average of the class-wise log loss). But the submission package only requirese the predicted labels file (.csv). How do you calculate their log loss?

I want to calculate the log loss of my fusion system using the same way as yours for evaluation dataset. Could you tell me how to calculate the log loss with the predicted labels or give me a function?

 

------------------ 原始邮件 ------------------ 发件人: "marmoi/dcase2021_task1a_baseline" @.>; 发送时间: 2021年6月11日(星期五) 中午1:38 @.>; @.**@.>; 主题: Re: [marmoi/dcase2021_task1a_baseline] Can I do not use the provided training/test setup? (#2)

Hi, About the split, if you go to the dcase web page of the task, in the task setup, development dataset there is a table with details of how the split has done and that there are some files that hadn't been used for the train/test split.

Regarding the model calculation, just implement a function that counts all non-zero parameters of your model, regardless of they are trainable or non-trainable.

Finally, about the log_loss, we also use the one from sklearn.metrics. The only thing I can think about is perhaps you are modifying y_pred before using the log_loss function, the predicted probablilities should given to the function as they are returned by the model.

I hope this helps

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[marmoi]

Hi, Copied from the dcase web page: "System output should be presented as a single text-file (in CSV format, with a header row) containing a classification result for each audio file in the evaluation set. In addition, the results file should contain probabilities for each scene class. Result items can be in any order. Multiple system outputs can be submitted (maximum 4 per participant per subtask)."

[dby124]

Thanks so much. When I use the probabilites, the value of the log loss is correct . It helps me much.  

------------------ 原始邮件 ------------------ 发件人: "marmoi/dcase2021_task1a_baseline" @.>; 发送时间: 2021年6月14日(星期一) 中午1:27 @.>; @.**@.>; 主题: Re: [marmoi/dcase2021_task1a_baseline] Can I do not use the provided training/test setup? (#2)

Hi, Copied from the dcase web page: "System output should be presented as a single text-file (in CSV format, with a header row) containing a classification result for each audio file in the evaluation set. In addition, the results file should contain probabilities for each scene class. Result items can be in any order. Multiple system outputs can be submitted (maximum 4 per participant per subtask)."

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.