search

SCIInstitute / map3d

Other
3 stars 3 forks source link

Matlab Data Structure Flexibility #40

Open jab0707 opened 4 years ago

jab0707 commented 4 years ago

The map3d data loader does not seem to accept data structures if the basal variable name is not 'ts', even if the fields of the data structure match map3d expected inputs (struct.potvals etc).

I request that the data loader be changed to accept any strcture name and search for the approprate fields (.potvals, etc). Attached is an example where the geom.mat is the geometry, originalDataStruct.mat is the data for that geometry but with a variable name other than 'ts', and correctedDataStruct.mat is the data with the base struct name changed to 'ts'. Currently only the correctedData.mat will load with the geometry.

example.zip

bryanworthen commented 4 years ago

Hi,

So, the code doesn't do anything to specify that the toplevel variable name should be "ts". On opening the correctedData.mat file, it opens fine, but that's because the top node has correctly named children: potvals, etc.

The "originalDataStruct.mat" won't open in map3d, because map3d doesn't recognize it as a valid matlab file. Is there a chance this file is corrupted?

jab0707 commented 4 years ago

Hi Bryan,

The file loads into matlab just fine but I am not sure if there is some part of matlab that could be reading in or fixing something that map3d does not. I can try loading the 'originalDataStruct.mat' into matlab then saving it out again and seeing if that works.

bryanworthen commented 4 years ago

It could also be something about the way map3d reads matlab files (we use a custom library that we (but not I) wrote years ago).

If that's the way you made the "corrected" file, then that might help me get past this hurdle. Bryan

On Mon, Mar 23, 2020 at 10:41 AM Jake Bergquist notifications@github.com wrote:

Hi Bryan,

The file loads into matlab just fine but I am not sure if there is some part of matlab that could be reading in or fixing something that map3d does not. I can try loading the 'originalDataStruct.mat' into matlab then saving it out again and seeing if that works.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SCIInstitute/map3d/issues/40#issuecomment-602717937, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKDF3VUB5KBTM6LHJXCYJTRI6GK3ANCNFSM4LFYK5SQ .

jab0707 commented 4 years ago

I made the "corrected" file by:

load('originalDataStruct.mat') this loads in a struct called ep_peri with fields potvals ts.potvals = ep_peri.potvals; save('correctedDataStruct.mat','ts');

However just now I loaded int he original data struct then saved it back out as is and map3d was able to load it in fine. so clearly as you say it is not a problem with the structure name as long as it has the right fields. It is rather a problem with the original file itself, but I am not sure what because matlab loads it in fine.

bryanworthen commented 4 years ago

Especially since (I assume) you just loaded it and saved it again.

Is there a chance to see if the process that created the original file produces another file that we can't load? It's possible we need map3d to open these files, and to debug whatever problem it is having (could be a rabbit hole). Bryan

On Mon, Mar 23, 2020 at 12:17 PM Jake Bergquist notifications@github.com wrote:

I made the "corrected" file by:

load('originalDataStruct.mat') this loads in a struct called ep_peri with fields potvals ts.potvals = ep_peri.potvals; save('correctedDataStruct.mat','ts');

However just now I loaded int he original data struct then saved it back out as is and map3d was able to load it in fine. so clearly as you say it is not a problem with the structure name as long as it has the right fields. It is rather a problem with the original file itself, but I am not sure what because matlab loads it in fine.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SCIInstitute/map3d/issues/40#issuecomment-602772149, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKDF3WH3HFQQNX75FFTGG3RI6RUTANCNFSM4LFYK5SQ .

jab0707 commented 4 years ago

Attached is another data file for the same geometry generated by the same process. This data is coming from a collaborator so I do not have access tot he code that wrote the files. This one is the same structure etc just different data and it also does not load into map3d ep_peri_res0.mat.zip

rsmacleod commented 4 years ago

Is this one of the subtleties in which MATLAB has some special features that allow it to load data even when the format it a little wrong? You would think there is a strong motivation to build very robust readers into MATLAB. I wonder at what point, we just process the files we need to, essentially filter them with MATLAB, and save the result and "proper" MATLAB files that map3d can read? It seems a function of how many of such files we need to cope with.