[Bug]: "core" file not working with parfor loops

[raphaelguex]

What happened?

I am trying to use the NWB with a parfor loop, to upload the data. And it seems to me that the "core" file (matnwb-master\namespaces\core.mat.) cannot handle the parfor loop. Would you have any suggestion to be able to use parfor with NWB ? Many thanks!

Steps to Reproduce

parfor (amP=1:size(ListFiles,1),amCores)
FunctionToUploadFiles(x,y,z,t)
end

Error Message

Error using spec.loadCache (line 35)
Cannot read file F:\data-archive-1U01NS098981\DABI\scripts\matnwb-master\namespaces\core.mat.

Error in schemes.loadNamespace (line 6)
Cache = spec.loadCache(name, 'savedir', saveDir);

Error in file.writeNamespace (line 3)
Namespace = schemes.loadNamespace(namespaceName, saveDir);

Error in nwbRead>generateSpec (line 132)
        file.writeNamespace(name, saveDir);

Error in nwbRead (line 54)
        generateSpec(filename, h5info(filename, specLocation), 'savedir', saveDir);

Error in LoadTrialsFilesB (line 5)
nwb= nwbRead(char(fileH));

Error in plot_NWB_places_RGb6_Parf (line 15)
parfor (amP=1:size(raw,1),amCores)

Operating System

Windows

Matlab Version

2021b

Code of Conduct

• [X] I agree to follow this project's Code of Conduct
• [X] Have you ensured this bug was not already reported?

[lawrence-mbf]

Hi @raphaelguex,

To generate class files efficiently, MatNWB caches read-in namespaces and generates class files off of these whenever you call generateCore or nwbRead. Using parfor with these calls will cause file locking issues since you're accessing these files in parallel so you would want to disable that file generation when you use nwbRead.

Assuming that the embedded spec is the same for all files you want to read, you should be able to use the ignorecache argument flag when calling nwbRead (i.e. nwbRead('file.nwb', 'ignorecache');). All this flag does is disable embedded that file generation spec. Note that if the embedded spec deviates across these NWB files, you will most likely get read errors instead.

[raphaelguex]

Many thanks for your response and fix! it seems to work well.

May I ask "what could I miss" by using this approach ? which type of infor may be wrong ? I am not familiar with this way to prepare the data and embedded spec is still vague to me. thanks!

[lawrence-mbf]

May I ask "what could I miss" by using this approach?

If you read a file using MatNWB whose schema does not match your environment, you risk running into read errors either due to:

• classes not being defined or defined with incorrect properties.
• classes being in the wrong namespace.

Usually these issues are caught and you should see an error or warning indicating that this is the case, or some kind of conversion error where a class defined in one namespace is being converted to another one (a common one between core namespace versions is the VectorData type which was moved from the core namespace to the hdmf_common namespace).

In such a case, there's no other option other than simply regenerating the class files.

Hopefully that answers your question.

[raphaelguex]

Yes, many thanks for these precisions!