Open angoodkind opened 2 years ago
Hi Adam, sorry for that. There was a bug in my code. The save labels included those padding utterances. Now it should work.
Unfortunately it's still off. My test.csv has 4879 utterances. The list of predictions is 4895 items. It's much closer, but there's still a misalignment.
could you provide with me your test.csv?
Let me know if this works. Thanks!
https://drive.google.com/file/d/1zSnPC5pkYZXN0tCex8DLG-M9wZVSy99D/view?usp=sharing
From my side the output length is 4879. I just used the model I shared with you to do the inference.
At column "speaker", the values should be 0s and 1s indicating the speaker turns. So what I did was mapping subject1 and subject2 to 0 and 1. Without doing this the code couldn't run. I am not sure how it got run in your side?
Regarding the utterance ID. The output should be the same length as the input csv file. So I don't see a point of adding that. If there's a mismatch of the length, there must be something wrong. In that case I won't trust the output even it has utterance ids.
Interesting. I saw in some of the changes i the last commit that you changed how the model.pt is named. Did you rename it for your run, as well?
The model I shared with you was named "model_swda.pt". I renamed it to "model.pt" so it can be loaded in the inference() function.
Ok, right, that's what I'd been doing. I'm still getting a result that's the wrong length. Would you mind sending me the correct length list, and I'll try to do more debugging later?
The results don't look good to me. I guess that's because your data is very clean which is very different from the switchboard corpus that has a lot of punctuations and symbols. You can check a few utterances in switchboard. Another researcher who used this repo once faced the same issue. He solved it by training a new model based on the cleaned swithchboard corpus and then using the model to infer on his own dataset.
1) Thanks for the pickled file. It does work.
2) I agree about the results. I don't suppose you have a link to the cleaned corpus, do you? If not, it seems pretty trivial to do on my own.
That's right I don't have a cleaned corpus.
If doing it on your own, 1) please pay attention to the contents in square brackets. 2) In switchboard there are a lot of utterances labeled as interrupted and continued later ('+' tags ,42 in our case), which means the utterance is not a full utterance. It's better to concatenate those since in your dataset every utterance is a full utterance. 3) Utterances labeled as Non-verbal ('x' tags, 26 in our case) should be removed since they're non-existent in your dataset either.
Seem this corpus has already done 2) and 3).
Thanks!
Hi,
I'm running into a similar issue: the dataset I'm running inference on is much cleaner than the swda dataset. As a result a lot of the utterances are coded as 26: nonverbal or 42: other. @angoodkind & @zihaohe123 has it helped to work with a cleaner version of swda? Does any of you have a cleaned up version available? I would very much appreciate the help. Thanks in advance!
@zihaohe123 in the cleaned up version you're referring to in your previous post the conv_id is missing. Is that an issue?
@SebastianSpeer Hi Sebastian,
Thank you for your interest! I'm sorry that the results do not look good on your own dataset. I'm not very sure what the problems might be. But it shouldn't be because of missing conv_id. Please take a closer look at the data in swda and your own data and see if you can find how they are different.
so the conv_id has no influence on the classification accurracy?
@SebastianSpeer Sorry my bad! conv_id is actually very important as it indicates whether utterances are of one conversation. You should add it to your data.
@zihaohe123, thanks for your reply. I have it in my data, but the clened version of the switchboard data you were referring to (https://github.com/NathanDuran/Switchboard-Corpus) does not have the conv id. Do you know of other cleaned versions of the switchboard corpus?
@SebastianSpeer The NathanDuran's corpus does have the conv_id. Go to soda_data/train. Each .txt file is a conversation. The file name is the conv_id.
I am using inference mode. My test.csv file has 4.480 utterances. The output of preds_on_new.pkl is a list that is 12,342 items long. Am I doing something wrong?