IlcorJPGs not created

[ggut]

Potential problem is that IlcorJPG creation is started before illumcorrections are calculated.

Sender: LSF System <lsfadmin@a6000>
Subject: Job 2735074: <matlab -singleCompThread -nodisplay -nojvm << M_PROG; % Add custom project code support.;brainy.libpath.checkAndAppendLibPath('/BIOL/sonas/biol_uzh_pelkmans_s4/Data/Users/Gabriele/Multiplexing/20151202_MCExp12/20151202_MCExp12_stain/TIFF/');  create_jpgs_illumination_corrected('/BIOL/sonas/biol_uzh_pelkmans_s4/Data/Users/Gabriele/Multiplexing/20151202_MCExp12/20151202_MCExp12_stain/TIFF/','/BIOL/sonas/biol_uzh_pelkmans_s4/Data/Users/Gabriele/Multiplexing/20151202_MCExp12/20151202_MCExp12_stain/ILLCORJPG'); merge_jpgs_per_plate('/BIOL/sonas/biol_uzh_pelkmans_s4/Data/Users/Gabriele/Multiplexing/20151202_MCExp12/20151202_MCExp12_stain/ILLCORJPG'); M_PROG> Done

Job <matlab -singleCompThread -nodisplay -nojvm << M_PROG; % Add custom project code support.;brainy.libpath.checkAndAppendLibPath('/BIOL/sonas/biol_uzh_pelkmans_s4/Data/Users/Gabriele/Multiplexing/20151202_MCExp12/20151202_MCExp12_stain/TIFF/');  create_jpgs_illumination_corrected('/BIOL/sonas/biol_uzh_pelkmans_s4/Data/Users/Gabriele/Multiplexing/20151202_MCExp12/20151202_MCExp12_stain/TIFF/','/BIOL/sonas/biol_uzh_pelkmans_s4/Data/Users/Gabriele/Multiplexing/20151202_MCExp12/20151202_MCExp12_stain/ILLCORJPG'); merge_jpgs_per_plate('/BIOL/sonas/biol_uzh_pelkmans_s4/Data/Users/Gabriele/Multiplexing/20151202_MCExp12/20151202_MCExp12_stain/ILLCORJPG'); M_PROG> was submitted from host <a9010> by user <yyauhen> in cluster <brutus>.
Job was executed on host(s) <a6000>, in queue <pub.8h>, as user <yyauhen> in cluster <brutus>.
</cluster/home/biol/yyauhen> was used as the home directory.
</cluster/home/biol/yyauhen/iBRAIN> was used as the working directory.
Started at Wed Dec  2 18:55:52 2015
Results reported at Wed Dec  2 18:56:32 2015

Your job looked like:

------------------------------------------------------------
# LSBATCH: User input
matlab -singleCompThread -nodisplay -nojvm << M_PROG;
% Add custom project code support.
brainy.libpath.checkAndAppendLibPath('/BIOL/sonas/biol_uzh_pelkmans_s4/Data/Users/Gabriele/Multiplexing/20151202_MCExp12/20151202_MCExp12_stain/TIFF/');

create_jpgs_illumination_corrected('/BIOL/sonas/biol_uzh_pelkmans_s4/Data/Users/Gabriele/Multiplexing/20151202_MCExp12/20151202_MCExp12_stain/TIFF/','/BIOL/sonas/biol_uzh_pelkmans_s4/Data/Users/Gabriele/Multiplexing/20151202_MCExp12/20151202_MCExp12_stain/ILLCORJPG');
merge_jpgs_per_plate('/BIOL/sonas/biol_uzh_pelkmans_s4/Data/Users/Gabriele/Multiplexing/20151202_MCExp12/20151202_MCExp12_stain/ILLCORJPG');
M_PROG
------------------------------------------------------------

Successfully completed.

Resource usage summary:

    CPU time   :     23.97 sec.
    Max Memory :       218 MB
    Max Swap   :      1155 MB

    Max Processes  :         3
    Max Threads    :         9

The output (if any) follows:

                            < M A T L A B (R) >
                  Copyright 1984-2013 The MathWorks, Inc.
                    R2013a (8.1.0.604) 64-bit (glnxa64)
                             February 15, 2013

To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.

/cluster/home/biol/yyauhen/iBRAIN/startup.m: randomizing rand initial state
/cluster/home/biol/yyauhen/iBRAIN/startup.m: 16 CPUs detected, using 1 threads
>> >> 
filepathsep =

/

Ignoring custom project code support. Path does not exist: /BIOL/sonas/biol_uzh_pelkmans_s4/Data/Users/Gabriele/Multiplexing/20151202_MCExp12/20151202_MCExp12_stain/LIB/MATLAB
>> >> create_jpgs_illumination_corrected: using all images found
create_jpgs_illumination_corrected: analyzing /BIOL/sonas/biol_uzh_pelkmans_s4/Data/Users/Gabriele/Multiplexing/20151202_MCExp12/20151202_MCExp12_stain/TIFF/
create_jpgs_illumination_corrected:  found 384 images
create_jpgs_illumination_corrected:  parsing channel & position information
get_image_snake: 16 images per well, of type CV7K. cols = 4, rows = 4
create_jpgs_illumination_corrected:  microscope type "CV7K"
create_jpgs_illumination_corrected:  16 images per well
                 channel 1 present
                 channel 2 present
                 channel 3 present
create_jpgs_illumination_corrected: sampling 128 random images from channel 1...Caching illumination correction ... complete 
{Error using imread (line 350)
File
"/BIOL/sonas/biol_uzh_pelkmans_s4/Data/Users/Gabriele/Multiplexing/20151202_MCExp12/20151202_MCExp12_stain/TIFF/20151202_MCExp12_stain_D04_T0001F015L01A03Z01C01.tif"
does not exist.

Error in
create_jpgs_illumination_corrected>imread_shrunken_illumination_corrected (line
374)
    Image = double(shrinkFun(imread(strImage)));

Error in create_jpgs_illumination_corrected (line 191)
        ImageForHeuristics =
        imread_shrunken_illumination_corrected(strImageForHeuristics,
        ShrinkFactorForHeuristics);
} 
>> merge_jpgs_per_plate: processing /BIOL/sonas/biol_uzh_pelkmans_s4/Data/Users/Gabriele/Multiplexing/20151202_MCExp12/20151202_MCExp12_stain/ILLCORJPG
merge_jpgs_per_plate: looking for *.jpg
merge_jpgs_per_plate: found 0 matching files
>> 

[ewiger]

Thank you for reporting. This error can be explained by the case when PNG conversion is running in parallel and removing the tif file. I will check if this is the case.

[tstoeger]

same problem is also with code of create_jpg , if submission does not wait for png conversion (which is a good thing).

Once I had made a version of original create_jpg (I think I have deleted in meantime...) where filename is deconstructed and file extending is saved in a persistent variable, which can be toggled by a error catching. If fileattrib would be used prior to each loading, it would heavily slow down filesystem (and processing) due to repeated listing of all files.

[tstoeger]

While forcing illcorjpgs to wait for completion of png conversion fixes the problem, and enables progression, and while it is not worse than what we had been running in the lab for the last years, it also is a missed chance to speed up the human waiting time by a few hours.

Alternative suggestion: make the code of illcorjpgs (and jpgs) smarter so that it can cope with situations, where the images have been renamed, after illcorjpgs (or jpgs - where outcome should be blank image in case of renaming) had been started.

[ewiger]

Illumination correction does this retry approach. I strongly recommend against such concurrent IO practice. Such architecture leads to strange unpredictable artifacts. Please - no.