Problem in loading the pre-trained model : inconsistent testing results

[muberrayener]

I have a trouble when using pretrained model. I saved model's state_dict after training, then i loaded it, applied it on test data. It gave inconsistent results because i used the same data for visualizing test loss during training.

I use ubuntu 20.04, and latest ME version, Python3.8

==========System========== Linux-5.8.0-50-generic-x86_64-with-glibc2.10 DISTRIB_ID=Ubuntu DISTRIB_RELEASE=20.04 DISTRIB_CODENAME=focal DISTRIB_DESCRIPTION="Ubuntu 20.04.2 LTS" 3.8.8 (default, Apr 13 2021, 19:58:26) [GCC 7.3.0] ==========Pytorch========== 1.7.1 torch.cuda.is_available(): True ==========NVIDIA-SMI========== /usr/bin/nvidia-smi ==========NVCC========== sh: 1: nvcc: not found ==========CC========== /usr/bin/c++ c++ (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0 Copyright (C) 2019 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

==========MinkowskiEngine========== 0.5.2 MinkowskiEngine compiled with CUDA Support: True NVCC version MinkowskiEngine is compiled: 11010 CUDART version MinkowskiEngine is compiled: 11010

[chrischoy]

It seems like the indoor.py example in the MinkowskiEngine example folder is working fine. Are you sure it is the library not your code thats causing the issue?

[muberrayener]

I use imbalanced 3D MRI data with minkowski FCNN network from classification_modelnet40.py in examples folder. Because the data are already in voxelized form (3D tensor), i feed the network with them after converting into sparse tensors (indices, single-channel features).

I trained the model. The weigths of the saved model and loaded model are the same but the results are different. Here is a snippet from testing. I couldn't find what causes this problem.

if name == "main":

config = parser.parse_args()  
model = MinkowskiFCNN(
    in_channel=1, out_channel=2, embedding_channel=1024)
dict_= torch.load(config.weights)
model.load_state_dict(dict_["state_dict"])
print(model)
print(model.state_dict())
y = 14*[0]+ 50* [1]
weights = compute_class_weight('balanced', classes=np.unique(y),y=y)
criterion = nn.CrossEntropyLoss(weight=torch.FloatTensor(weights))
data_loader = make_data_loader(config=config, phase="test")
model.eval()
labels, preds, val_loss = [], [], 0
with torch.no_grad():
    for batch in data_loader:
        input = ME.TensorField(coordinates=batch["coordinates"], 
        features=batch["features"])
        logit = model(input)
        print(logit)
        print(batch["labels"])
        loss = criterion(logit.float(), batch["labels"].long()) 
        print(loss)
        val_loss+=loss.detach().item()
        pred = torch.argmax(logit, 1)
        labels.append(batch["labels"].cpu().numpy())
        preds.append(pred.cpu().numpy())

    print(metrics.confusion_matrix(np.concatenate(labels), np.concatenate(preds)))
    print(f"val_loss: {val_loss/len(data_loader)}")
    print(metrics.classification_report(np.concatenate(labels), 
            np.concatenate(preds),target_names = ['class 0', 'class 1']))

[muberrayener]

If I try to save the whole model using torch.save(model, file_name), it gives the error below: TypeError: cannot pickle 'MinkowskiConvolutionFunction' object

[chrischoy]

This is not a self contained code and I can't help you with debugging. I'm pretty sure the error is in your code. Please try to use just regular pytorch dense 3D convnets first to figure out which part is causing the error.

[muberrayener]

Thank you for your advice, i will try it.

[vgthengane]

@muberrayener How did you solve the error?

[muberrayener]

@muberrayener How did you solve the error?

I tried to solve but nothing worked. I changed the network from MinkowskiFCNN to Minkowski ResNet, the problem disappeared.