search

pelkmanslab / iBRAIN_BRUTUS

A version of canonical iBRAIN (2015) deployed on BRUTUS cluster
0 stars 0 forks source link

Create jpegs failed from MIPS? #29

Open ewiger opened 9 years ago

ewiger commented 9 years ago

I am posting this into an issue in case it might be of general interest.

Create jpegs failed from MIPS? What could be the reason?

log:

Sender: LSF System <lsfadmin@a9020>
Subject: Job 97029065: <matlab -singleCompThread -nodisplay -nojvm << M_PROG; % Add custom project code support.;brainy.libpath.checkAndAppendLibPath('/BIOL/sonas/biol_uzh_pelkmans_s9/Data/Users/Ilya/151007-organoids-lgr5test-10x_20151007_175154/P1_150901IL004/TIFF/');  create_jpgs('/BIOL/sonas/biol_uzh_pelkmans_s9/Data/Users/Ilya/151007-organoids-lgr5test-10x_20151007_175154/P1_150901IL004/TIFF/','/BIOL/sonas/biol_uzh_pelkmans_s9/Data/Users/Ilya/151007-organoids-lgr5test-10x_20151007_175154/P1_150901IL004/JPG/'); merge_jpgs_per_plate('/BIOL/sonas/biol_uzh_pelkmans_s9/Data/Users/Ilya/151007-organoids-lgr5test-10x_20151007_175154/P1_150901I
L004/JPG/'); M_PROG> Done

Job <matlab -singleCompThread -nodisplay -nojvm << M_PROG; % Add custom project code support.;brainy.libpath.checkAndAppendLibPath('/BIOL/sonas/biol_uzh_pelkmans_s9/Data/Users/Ilya/151007-organoids-lgr5test-10x_20151007_175154/P1_150901IL004/TIFF/');  create_jpgs('/BIOL/sonas/biol_uzh_pelkmans_s9/Data/Users/Ilya/151007-organoids-lgr5test-10x_20151007_175154/P1_150901IL004/TIFF/','/BIOL/sonas/biol_uzh_pelkmans_s9/Data/Users/Ilya/151007-organoids-lgr5test-10x_20151007_175154/P1_150901IL004/JPG/'); merge_jpgs_per_plate('/BIOL/sonas/biol_uzh_pelkmans_s9/Data/Users/Ilya/151007-organoids-lgr5test-10x_20151007_175154/P1_150901IL004/JPG/'); M_PROG> was submitted from host <a9020> by user <ibrain> in cluster <brutus>.
Job was executed on host(s) <a9020>, in queue <pub.8h>, as user <ibrain> in cluster <brutus>.
</cluster/home/biol/ibrain> was used as the home directory.
</cluster/home/biol/ibrain/iBRAIN> was used as the working directory.
Started at Thu Oct  8 10:37:57 2015
Results reported at Thu Oct  8 10:38:12 2015

Your job looked like:

------------------------------------------------------------
# LSBATCH: User input
matlab -singleCompThread -nodisplay -nojvm << M_PROG;
% Add custom project code support.
brainy.libpath.checkAndAppendLibPath('/BIOL/sonas/biol_uzh_pelkmans_s9/Data/Users/Ilya/151007-organoids-lgr5test-10x_20151007_175154/P1_150901IL004/TIFF/');

create_jpgs('/BIOL/sonas/biol_uzh_pelkmans_s9/Data/Users/Ilya/151007-organoids-lgr5test-10x_20151007_175154/P1_150901IL004/TIFF/','/BIOL/sonas/biol_uzh_pelkmans_s9/Data/Users/Ilya/151007-organoids-lgr5test-10x_20151007_175154/P1_150901IL004/JPG/');
merge_jpgs_per_plate('/BIOL/sonas/biol_uzh_pelkmans_s9/Data/Users/Ilya/151007-organoids-lgr5test-10x_20151007_175154/P1_150901IL004/JPG/');
M_PROG
------------------------------------------------------------

Successfully completed.

Resource usage summary:

    CPU time   :      3.75 sec.
    Max Memory :       129 MB
    Max Swap   :       940 MB

    Max Processes  :         3
    Max Threads    :         8

The output (if any) follows:

                            < M A T L A B (R) >
                  Copyright 1984-2013 The MathWorks, Inc.
                    R2013a (8.1.0.604) 64-bit (glnxa64)
                             February 15, 2013

To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.

/cluster/home/biol/ibrain/iBRAIN/startup.m: randomizing rand initial state
/cluster/home/biol/ibrain/iBRAIN/startup.m: 48 CPUs detected, using 1 threads
filepathsep =

/

Ignoring custom project code support. Path does not exist: /BIOL/sonas/biol_uzh_pelkmans_s9/Data/Users/Ilya/151007-organoids-lgr5test-10x_20151007_175154/P1_150901IL004/LIB/MATLAB
create_jpgs: ignoring MIPs, using only z-stacks
create_jpgs: analyzing /BIOL/sonas/biol_uzh_pelkmans_s9/Data/Users/Ilya/151007-organoids-lgr5test-10x_20151007_175154/P1_150901IL004/TIFF/
create_jpgs:  found 0 images
create_jpgs:  parsing channel & position information
all channels were 0 or NaN, setting all to 1
all positions were 0 or NaN, setting all to 1
merge_jpgs_per_plate: processing /BIOL/sonas/biol_uzh_pelkmans_s9/Data/Users/Ilya/151007-organoids-lgr5test-10x_20151007_175154/P1_150901IL004/JPG/
merge_jpgs_per_plate: looking for *.jpg
merge_jpgs_per_plate: found 0 matching files
ewiger commented 9 years ago

If ownMIPs==true then create_jpgs generates own MIPs on the fly. By default ownMIPs is set to true.

Image filenames that I have found in TIFF folder of the project (mentioned in the log above) look as following: P1_150901IL004_D06_T0001F005L01A01C04.png, i.e. there is no part indicating that this is a z-stack image Z00*.

So far, so good.

The problem starts when function check_image_position() is called, which ignores filenames without Z00* substring.

>> check_image_position('P1_150901IL004_D06_T0001F005L01A01C04.png')
Error using check_image_position (line 130)
unknown file name P1_150901IL004_D06_T0001F005L01A01C04.png

>> check_image_position('P1_150901IL004_D06_T0001F005L01A01Z01C04.png')

ans =

     5

Possible solutions to this problem would be:

1) Modify creation of MIPS such that Z00 always added to every Z-stack. 2) Modify regular expression in https://github.com/pelkmanslab/iBRAINShared/blob/master/iBRAIN/CreateJpgsOverview/create_jpgs.m to have Z00 substring optional.

I am in favor of the second option. But this has to be documented in the code so that future developers are aware of difference in naming convention

A MIP image has no Z\d+ substring A Z-plane image has Z\d+ substring

@SaadiaIftikhar what do you think?

SaadiaIftikhar commented 9 years ago

2nd option seems better. Did you work on jpeg and illumination creation at the same time? What would be the logical way to go to fix this bug?

ewiger commented 9 years ago

General approach is:

OK I am following option 2. I placed the fix that should work in my opinion into LIB/MATLAB of Ilya's project.

@SaadiaIftikhar can you test it on the whole project?