Issues when trying different loss functions

[SaniyaAbushakimova]

Hello!

Issue 1: Softmax

The first time when I tried to run the model with Softmax loss, I got the following error message:

File "softmax.py", line 24, in forward
    nloss   = self.criterion(x,label)
ValueError: Expected target size (100, 100), got torch.Size([100])

In the code the problem appeared to be in the areas inside a red box:

As it can be noticed from the comments in the above screenshot, x originally is of size (100, 10, 512), after going through an FC layer its size changes to (100, 10, 100), and then it is passed straight to torch.nn.CrossEntropyLoss() function. However, according to the official PyTorch documentation, this function requires the x to be of size (100, 100).

I have come up with a possible solution and was wondering whether the solution is reasonable and does not have any side effects on the model performance. The solution was to introduce additional transformations that will reduce (100, 10, 100) to (100, 100). It was done with the help of torch.mean(x, dim = 1, keepdim=False). As far as I know, such a method is legal in neural networks, more about the method can be found here.

So, the adjusted code looks like this:

In this way, the model with Softmax loss now runs and works perfectly fine. What do you think about this solution?

---

Issue 2: Amsoftmax, AAmsoftmax

The same issue as with Softmax loss was also detected while using Amsoftmax and AAmsoftmax losses. These errors were resolved in the same manner. Below are the adjusted codes: a) Amsoftmax

b) AAmsoftmax

---

Issue 3: Triplet

The second issue that I encountered was involving the Triplet Loss. There were two main errors with that:

Error 1:

File "/home/sania_abushakimova/sf_pv/loss/triplet.py", line 26, in forward
    assert x.size()[1] == 2 
AssertionError

From the above message, it seems that in the code the error comes from the part highlighted in a red box.

After analyzing other metric-based losses, I came to a conclusion that == sign must be replaced to >=. Please correct me if my assumption was incorrect. (The error was resolved though)

Here are the screenshots from other losses that contain the same assert statement. a) Proto b) AngleProto c) Ge2e

Error 2:

File "SpeakerNet.py", line 112, in train_network
    top1    += prec1.detach().cpu().item();
AttributeError: 'numpy.float64' object has no attribute 'detach'

This problem was caused due to prec1 for some reason being a numpy array instead of a tensor. Below is one piece from SpeakerNet.py:

In order to solve this problem, I manually converted prec1 to numpy array with the help of torch.tensor(prec1). It eventually did solve the problem, but I was wondering whether it was a legal solution.

---

Looking forward to your kind response, it will help me a lot with my research project!

[SaniyaAbushakimova]

Resolved

[speaker-lover]

Excuse me, what is the result of using angleproto loss function? I got eer=2.6 minDCF=0.2. Is this normal? Does it belong to self-supervised learning method or supervised learning method?

[zabir-nabil]

Hi, @SaniyaAbushakimova did you find any discrepancy in results with this mean approach to avoid the assertion error?