Relja
commented
2 years ago Hi,
Sorry but I don't have hands on experience with this and haven't used PyTorch.
Few suggestions:
- There's this tool I know about: https://github.com/albanie/pytorch-mcn and they have already imported NetVLAD apparently.
- This is really more of a question MatConvNet->PyTorch than NetVLAD specific (though you also need to implement the NetVLAD layer which is custom here), so you can try asking the MatConvNet people.
- Maybe going through tensorflow as I imagine tf -> PyTorch could be more standard. I know this project which apparently imported NetVLAD checkpoints to tf: https://github.com/uzh-rpg/netvlad_tf_open
- You should be able to load the .mat file from the opensource and free Octave, so you could use that to 1) export weights in whatever format you want, or 2) double check the imported weights match tha ones in the mat files.
I guess things that could be problematic beyond just importing weights is if there are some slight differences in implementation across frameworks, e.g. maybe there are some tiny padding issues, or I recall reading about some nets being extremely sensitive to the exact method used when resizing images etc.
Hi, I am stuck on an issue to use '.mat' trained model files inside PyTorch module. I want to use these as baseline models for a new dataset and somehow test and find my recall values. The problem with PyTorch is that the weight checkpoints (.pth.tar) are usually saved in the form of Ordered dict with different keys and in .mat files, the weights are save in the form of numpy.ndarray. I am not able to load the model inside my script and use them for inference.
As a normal procedure, I am using scipy.io module and then 'loadmat' function to load the NetVLAD trained model '.mat' files. I suppose we need to check the shape and also possible flipped kernels, since some frameworks flip the conv kernels to use a convolution instead of the correlation. This is suggested inside this thread: https://discuss.pytorch.org/t/how-to-transfer-a-trained-model-from-matlab-to-pytorch/15903
The other option is also to convert to ONNX format and then convert to PyTorch model. The link for that is here: https://stackoverflow.com/questions/66223768/how-to-import-deep-learning-models-from-matlab-to-pytorch The problem with this approach is that you need to have MATLAB first for converting it every time to '.onnx' format and then via script to PyTorch.
Anyone with the better solution and a script to help with this would be really beneficial for future work.