search

ami-iit / matio-cpp

A C++ wrapper of the matio library, with memory ownership handling, to read and write .mat files.
https://ami-iit.github.io/matio-cpp/
BSD 2-Clause "Simplified" License
59 stars 9 forks source link

Cannot read mat files saved from Python with `7.3` format #75

Closed isorrentino closed 10 months ago

isorrentino commented 1 year ago

I saved a .mat file from Python using hdf5storage.savemat with the default format 7.3. In my C++ application, I opened the file, read the structure, and checked the number of fields.

matioCpp::File input(file);
matioCpp::Struct outStruct = input.read("robot").asStruct();
std::vector<std::string> listVar = input.variableNames();
std::cout << "Num variables: " << listVar.size());

However, the output was 0. I verified the content in Matlab, and it was correct.

In summary, the file opening works fine, but the variable returned by the read method is empty https://github.com/ami-iit/matio-cpp/blob/a0daf0691d492b2ed50910ea984f97bc2f945b80/include/matioCpp/File.h#L163

I solved the problem by saving the .mat file with format 5 instead of 7.3.

cc @S-Dafarra

S-Dafarra commented 1 year ago

Thanks @isorrentino, I started opening a PR to print an error when a variable is not found in the file: https://github.com/ami-iit/matio-cpp/pull/74

In any case, I suspect it is an issue of matio. Can you also comment with the actual error message?

isorrentino commented 1 year ago

Sure!

isorrentino@IITICUBLAP263:~/dev/robotology-superbuild/build/src/bipedal-locomotion-framework (master)$ ./bin/RobotDynamicsEstimationUnitTests 
Randomness seeded to: 3227269575
HDF5-DIAG: Error detected in HDF5 (1.10.7) thread 1:
  #000: ../../../src/H5L.c line 1168 in H5Literate(): link iteration failed
    major: Links
    minor: Iteration failed
  #001: ../../../src/H5L.c line 3350 in H5L__iterate(): link iteration failed
    major: Links
    minor: Iteration failed
  #002: ../../../src/H5Gint.c line 853 in H5G_iterate(): error iterating over links
    major: Symbol table
    minor: Iteration failed
  #003: ../../../src/H5Gobj.c line 692 in H5G__obj_iterate(): can't iterate over symbol table
    major: Symbol table
    minor: Iteration failed
  #004: ../../../src/H5Gstab.c line 553 in H5G__stab_iterate(): iteration operator failed
    major: Symbol table
    minor: Can't move to next iterator location
  #005: ../../../src/H5B.c line 1203 in H5B_iterate(): B-tree iteration failed
    major: B-Tree node
    minor: Iteration failed
  #006: ../../../src/H5B.c line 1160 in H5B__iterate_helper(): B-tree iteration failed
    major: B-Tree node
    minor: Iteration failed
  #007: ../../../src/H5Gnode.c line 1000 in H5G__node_iterate(): iteration operator failed
    major: Symbol table
    minor: Can't move to next iterator location
HDF5-DIAG: Error detected in HDF5 (1.10.7) thread 1:
  #000: ../../../src/H5L.c line 1168 in H5Literate(): link iteration failed
    major: Links
    minor: Iteration failed
  #001: ../../../src/H5L.c line 3350 in H5L__iterate(): link iteration failed
    major: Links
    minor: Iteration failed
  #002: ../../../src/H5Gint.c line 853 in H5G_iterate(): error iterating over links
    major: Symbol table
    minor: Iteration failed
  #003: ../../../src/H5Gobj.c line 692 in H5G__obj_iterate(): can't iterate over symbol table
    major: Symbol table
    minor: Iteration failed
  #004: ../../../src/H5Gstab.c line 553 in H5G__stab_iterate(): iteration operator failed
    major: Symbol table
    minor: Can't move to next iterator location
  #005: ../../../src/H5B.c line 1203 in H5B_iterate(): B-tree iteration failed
    major: B-Tree node
    minor: Iteration failed
  #006: ../../../src/H5B.c line 1160 in H5B__iterate_helper(): B-tree iteration failed
    major: B-Tree node
    minor: Iteration failed
  #007: ../../../src/H5Gnode.c line 1000 in H5G__node_iterate(): iteration operator failed
    major: Symbol table
    minor: Can't move to next iterator location
[2023-11-06 19:24:50.150] [thread: 56007] [blf] [info] Num variables --> 0
[2023-11-06 19:24:50.150] [thread: 56007] [blf] [info] Number of fields --> 0
traversaro commented 1 year ago

Just to double check, can you open the file in MATLAB?

isorrentino commented 1 year ago

Just to double check, can you open the file in MATLAB?

I already did it, I wrote it in the first comment. It has the expected content.

traversaro commented 1 year ago

Just to double check, can you open the file in MATLAB?

I already did it, I wrote it in the first comment. It has the expected content.

Ah sorry, I misread the first comment.

S-Dafarra commented 1 year ago

@isorrentino could you also link or upload the file that is not working?

traversaro commented 1 year ago

It would also be interesting to see the output of matdump -g -f whos <filename>.mat .

isorrentino commented 1 year ago

Here is the file

four_joints_two_ft_two_imu_with_noise_bias_friction.zip

I tried to run matdump -g -f whos <filename>.mat, but -g does not exist as option. Did you mean another option?

S-Dafarra commented 1 year ago

I tried with matdump -h -f whos four_joints_two_ft_two_imu_with_noise_bias_friction.mat and I got the same error

HDF5 error #000 in H5Literate2()
      file : H5L.c:1647
      major: Links
      minor: Iteration failed
HDF5 error #001 in H5L__iterate_api_common()
      file : H5L.c:1612
      major: Links
      minor: Iteration failed
HDF5 error #002 in H5VL_link_specific()
      file : H5VLcallback.c:5517
      major: Virtual Object Layer
      minor: Can't operate on object
HDF5 error #003 in H5VL__link_specific()
      file : H5VLcallback.c:5483
      major: Virtual Object Layer
      minor: Can't operate on object
HDF5 error #004 in H5VL__native_link_specific()
      file : H5VLnative_link.c:368
      major: Links
      minor: Iteration failed
HDF5 error #005 in H5L_iterate()
      file : H5Lint.c:2146
      major: Links
      minor: Iteration failed
HDF5 error #006 in H5G_iterate()
      file : H5Gint.c:869
      major: Symbol table
      minor: Iteration failed
HDF5 error #007 in H5G__obj_iterate()
      file : H5Gobj.c:649
      major: Symbol table
      minor: Iteration failed
HDF5 error #008 in H5G__stab_iterate()
      file : H5Gstab.c:507
      major: Symbol table
      minor: Can't move to next iterator location
HDF5 error #009 in H5B_iterate()
      file : H5B.c:1171
      major: B-Tree node
      minor: Iteration failed
HDF5 error #010 in H5B__iterate_helper()
      file : H5B.c:1133
      major: B-Tree node
      minor: Iteration failed
HDF5 error #011 in H5G__node_iterate()
      file : H5Gnode.c:941
      major: Symbol table
      minor: Can't move to next iterator location

In any case, I guess this is expected considering that it is a matio tool: https://github.com/tbeu/matio/blob/da3c5f61154ac000dcc3980fbe939cda3da2447b/tools/matdump.c

traversaro commented 1 year ago

In any case, I guess this is expected considering that it is a matio tool: https://github.com/tbeu/matio/blob/da3c5f61154ac000dcc3980fbe939cda3da2447b/tools/matdump.c

Yes, I wanted to quickly make sure matio-cpp was not involved in any way.

traversaro commented 1 year ago

The same error happens also with hdf5 1.14 : 

(libmatio) traversaro@IITICUBLAP257:~$ matdump -f whos four_joints_two_ft_two_imu_with_noise_bias_friction.mat
HDF5 error #000 in H5Literate2()
      file : H5L.c:1647
      major: Links
      minor: Iteration failed
HDF5 error #001 in H5L__iterate_api_common()
      file : H5L.c:1612
      major: Links
      minor: Iteration failed
HDF5 error #002 in H5VL_link_specific()
      file : H5VLcallback.c:5517
      major: Virtual Object Layer
      minor: Can't operate on object
HDF5 error #003 in H5VL__link_specific()
      file : H5VLcallback.c:5483
      major: Virtual Object Layer
      minor: Can't operate on object
HDF5 error #004 in H5VL__native_link_specific()
      file : H5VLnative_link.c:368
      major: Links
      minor: Iteration failed
HDF5 error #005 in H5L_iterate()
      file : H5Lint.c:2146
      major: Links
      minor: Iteration failed
HDF5 error #006 in H5G_iterate()
      file : H5Gint.c:869
      major: Symbol table
      minor: Iteration failed
HDF5 error #007 in H5G__obj_iterate()
      file : H5Gobj.c:649
      major: Symbol table
      minor: Iteration failed
HDF5 error #008 in H5G__stab_iterate()
      file : H5Gstab.c:507
      major: Symbol table
      minor: Can't move to next iterator location
HDF5 error #009 in H5B_iterate()
      file : H5B.c:1171
      major: B-Tree node
      minor: Iteration failed
HDF5 error #010 in H5B__iterate_helper()
      file : H5B.c:1133
      major: B-Tree node
      minor: Iteration failed
HDF5 error #011 in H5G__node_iterate()
      file : H5Gnode.c:941
      major: Symbol table
      minor: Can't move to next iterator location
(libmatio) traversaro@IITICUBLAP257:~$ mamba list
# packages in environment at /home/traversaro/miniforge3/envs/libmatio:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
c-ares                    1.21.0               hd590300_0    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
hdf5                      1.14.2          nompi_h4f84152_100    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.2               h659d440_0    conda-forge
libaec                    1.1.2                h59595ed_1    conda-forge
libcurl                   8.4.0                hca28451_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libgcc-ng                 13.2.0               h807b86a_2    conda-forge
libgfortran-ng            13.2.0               h69a702a_2    conda-forge
libgfortran5              13.2.0               ha4646dd_2    conda-forge
libgomp                   13.2.0               h807b86a_2    conda-forge
libmatio                  1.5.24               h31675a7_0    conda-forge
libnghttp2                1.55.1               h47da74e_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_2    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
ncurses                   6.4                  h59595ed_2    conda-forge
openssl                   3.1.4                hd590300_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge
traversaro commented 1 year ago

Instead h5dump four_joints_two_ft_two_imu_with_noise_bias_friction.mat works fine, so the problem perhaps is in libmatio ?

S-Dafarra commented 1 year ago

Opened issue upstream: https://github.com/tbeu/matio/issues/221

isorrentino commented 10 months ago

Hi @S-Dafarra, just to let you know, I saw that the issue here https://github.com/tbeu/matio/issues/221 has been closed.

S-Dafarra commented 10 months ago

Yes, the problem should be fixed in the release https://github.com/tbeu/matio/releases/tag/v1.5.25

Closing.