search

JinghaoLu / MIN1PIPE

A MINiscope 1-photon-based Calcium Imaging Signal Extraction PIPEline.
GNU General Public License v3.0
59 stars 27 forks source link

Error using normalize_batch (line 7) Variable 'frame_all' has 2 dimensions in the file, this does not match #21

Open tochenan opened 5 years ago

tochenan commented 5 years ago

Hi Jinghao,

I tried to run the code on a output file of imageJ (tif file of course)after concatenating several tif files together and the following error popped up: Error using normalize_batch (line 7) Variable 'frame_all' has 2 dimensions in the file, this does not match the 3 dimensions in the indexing subscripts.

Error in data_cat (line 261) m = normalize_batch(filename, 'frame_all', imx, imn, idbatch);

Error in min1pipe_HPC (line 87) [m, filename_raw, imaxn, imeanf, pixh, pixw, nf] = data_cat(path_name, file_base{i}, file_fmt{i}, Fsi, Fsi_new, spatialr);

Exception in thread "AWT-EventQueue-0": java.lang.NullPointerException at com.mathworks.widgets.grouptable.transfer.TransferController.getTransfer(TransferController.java:213) at com.mathworks.widgets.grouptable.transfer.TransferController.access$500(TransferController.java:48) at com.mathworks.widgets.grouptable.transfer.TransferController$DropListener.drop(TransferController.java:371) at java.awt.dnd.DropTarget.drop(DropTarget.java:455) at sun.awt.dnd.SunDropTargetContextPeer.processDropMessage(SunDropTargetContextPeer.java:538) at sun.awt.dnd.SunDropTargetContextPeer$EventDispatcher.dispatchDropEvent(SunDropTargetContextPeer.java:852) at sun.awt.dnd.SunDropTargetContextPeer$EventDispatcher.dispatchEvent(SunDropTargetContextPeer.java:776) at sun.awt.dnd.SunDropTargetEvent.dispatch(SunDropTargetEvent.java:48) at java.awt.Component.dispatchEventImpl(Component.java:4744) at java.awt.Container.dispatchEventImpl(Container.java:2297) at java.awt.Component.dispatchEvent(Component.java:4711) at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4904) at java.awt.LightweightDispatcher.processDropTargetEvent(Container.java:4609) at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4471) at java.awt.Container.dispatchEventImpl(Container.java:2283) at java.awt.Window.dispatchEventImpl(Window.java:2746) at java.awt.Component.dispatchEvent(Component.java:4711) at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:760) at java.awt.EventQueue.access$500(EventQueue.java:97) at java.awt.EventQueue$3.run(EventQueue.java:709) at java.awt.EventQueue$3.run(EventQueue.java:703) at java.security.AccessController.doPrivileged(Native Method) at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74) at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:84) at java.awt.EventQueue$4.run(EventQueue.java:733) at java.awt.EventQueue$4.run(EventQueue.java:731) at java.security.AccessController.doPrivileged(Native Method) at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74) at java.awt.EventQueue.dispatchEvent(EventQueue.java:730) at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:205) at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116) at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93) at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)

do you know why this takes place thank you :))

JinghaoLu commented 5 years ago

The problem indicates that frame_all is a matrix instead of a tensor. Why this happen is mysterious to me. I suggest you follow the guide of MIN1PIPE to rename the files and let it concatenate the files for you instead of using imageJ.

tochenan commented 5 years ago

Hello Jinghao,

Sorry that this may sound stupid, I am just wondering how to concatenate the files using Min1pipe. Another problem is that I found some quite obvious signals are not selected as ROIs, I am thinking of changing the Params.pix_select_sigthres parameter, but just not sure how much I should change from the original setting. Thanks a lot.

JinghaoLu commented 5 years ago

Please read the guide for multi-session processing.

The "obvious" signals are most likely bleaching neurons or abnormal ones with no stereotypical calcium events, so the algorithm skips them. That's usually the reason as we also find in our datasets. You can try playing with sigthres parameter, though that's not recommended.

tochenan commented 5 years ago

Do you mean to rename the files according to the inscopix data naming format?

tochenan commented 5 years ago

also for the bleaching problem, I guess if that is the case, the signal will disappear over time? but in the video I saw, I don't think it is the case.

JinghaoLu commented 5 years ago

Based on your video file format, tiff or avi, rename the files and MIN1PIPE will automatically concatenate the files. Test this to see if the original issue is still there.

JinghaoLu commented 5 years ago

I checked the video, and you are right that it is not bleaching, but still it can be the calcium event pattern is not recognized by MIN1PIPE.

Would you mind sharing the ***_reg.mat file or the raw video?

tochenan commented 5 years ago

Sorry, it seems that I also got another error when I tried to concatenate and analyse the videos in MIN1PIPE. I was wondering whether it is because the file is too big or not

Here is the error message: Warning: The following error was caught while executing 'H5ML.id' class destructor: Error using hdf5lib2 The HDF5 library encountered an error and produced the following stack trace information:

H5FD_sec2_truncate              unable to extend file properly
H5FD_truncate                   driver truncate request failed
H5F_dest                        low level truncate failed
H5F_try_close                   problems closing file
H5O_close                       problem attempting file close
H5D_close                       unable to release object header
H5I_dec_app_ref                 can't decrement ID ref count
H5I_dec_app_ref_always_close    can't decrement ID ref count
H5Dclose                        can't decrement count on dataset ID

Error in H5ML.id/close (line 47) H5ML.hdf5lib2(obj.callback, obj.identifier);

Error in H5ML.id/delete (line 36) obj.close();

Error in h5write (line 46) if nargin > 0

Error in savef (line 87) h5write(filename, varname, vars{i}, start, count);

Error in data_cat (line 255) savef(filename, 2, 'frame_all')

Error in min1pipe_HPC (line 87) [m, filename_raw, imaxn, imeanf, pixh, pixw, nf] = data_cat(path_name, file_base{i}, file_fmt{i}, Fsi, Fsi_new, spatialr);

In h5write (line 46) In savef (line 87) In data_cat (line 255) In min1pipe_HPC (line 87) Error using hdf5lib2 The HDF5 library encountered an unknown error.

Error in H5D.write (line 100) H5ML.hdf5lib2('H5Dwrite', varargin{:});

Error in h5write (line 138) H5D.write(dataset_id,'H5ML_DEFAULT',memspace_id,filespace_id,dxpl,Data);

Error in savef (line 87) h5write(filename, varname, vars{i}, start, count);

Error in data_cat (line 255) savef(filename, 2, 'frame_all')

Error in min1pipe_HPC (line 87) [m, filename_raw, imaxn, imeanf, pixh, pixw, nf] = data_cat(path_name, file_base{i}, file_fmt{i}, Fsi, Fsi_new, spatialr);

JinghaoLu commented 5 years ago

How did you rename the files?

JinghaoLu commented 5 years ago

I tested the concatenation using 2 of your previous shared files, and did not replicate the error.

I also looked into the ROI selection issue using the new file, and still did not replicate the error. The parameters I chose was Fsi = 20, Fsi_new = 5, spatialr = 0.25.

tochenan commented 5 years ago

Hi Jinghao, I changed the parameter you chose and didn't replicate the concatenation error. The name of my first tif file is 180621-mouse4_pups.tif and I named the second one 180621-mouse4_pups-1 and the third one 180621-mouse4_pups-2 ...

JinghaoLu commented 5 years ago

Ok I am not sure what happened to the previous parameter selection. One suggestion on renaming: try to keep only one "-" in the end for file indices, and replace the other "-" with "_".

tochenan commented 5 years ago

I am also wondering how can I order the concatenation by renaming the file? I think my previous attempt doesn't really make the concatenation in the order that I want

JinghaoLu commented 5 years ago

It follows the inscopix tradition, the first file is simply the original file name, the second is name-1.tif, the third is name-2.tif ......

Again name should contains no hyphen.