Taking a long time to process data.

[monjoybme]

Hi,

I ran ruby -I preprocess_stage1.rband it's taking more than 2 hrs. only for 6 samples (LIDC-IDRI-0001 to LIDC-IDRI-0006]. Could you tell me how much it should take to finish the jon on the whole dataset or, it's in an infinite loop?

[anibali]

Don't use -I. The command should be ruby preprocess_stage1.rb from the same directory as the script. It takes less than a minute to process 6 samples on my machine.

[monjoybme]

Thanks a lot. After stage 1, "scan.dat", "scan_metadata.json" and "nodule_01_metadata.json" have been generated for LIDC-IDRI-0001. In LIDC-IDRI-0003 file nodule-01 to nodule_04 files have been generated along with "scan.dat", "scan_metadata.json" files. Can you please describe the importance of each file?

Moreover, Can you give some insights on stage 2? When I'm running "preprocess_stage2.lua" file I'm getting below error: `lua preprocess_stage2.lua

lua: preprocess_stage2.lua:1: module 'torch' not found:

no field package.preload['torch']

no file './torch.lua'

no file '/usr/share/lua/5.1/torch.lua'

no file '/usr/share/lua/5.1/torch/init.lua'

no file '/usr/lib64/lua/5.1/torch.lua'

no file '/usr/lib64/lua/5.1/torch/init.lua'

no file './torch.so'

no file '/usr/lib64/lua/5.1/torch.so'

no file '/usr/lib64/lua/5.1/loadall.so'

stack traceback:

C: in function 'require'

preprocess_stage2.lua:1: in main chunk

In this code you have mentioned "torch". Is it pytorch? Could you also share the code of your deep learning model where you have used the output of stage 2? Thanks in advance.

[monjoybme]

This is the present issue. Could you please suggest why I'm facing this?

$ lua preprocess_stage2.lua 
   1/   8
lua: /home/usr/torch/install/share/lua/5.1/pl/path.lua:215: argument 1 expected a 'string', got a 'nil'
stack traceback:
    [C]: in function 'error'
    /home/usr/torch/install/share/lua/5.1/pl/utils.lua:429: in function 'assert_arg'
    /home/usr/torch/install/share/lua/5.1/pl/utils.lua:441: in function 'assert_string'
    /home/usr/torch/install/share/lua/5.1/pl/path.lua:215: in function 'basename'
    preprocess_stage2.lua:58: in main chunk
    [C]: ?

[anibali]

Regarding your first question, each scan can have multiple nodule annotations.

Regarding your second question, Torch7 needs to be installed.

Regarding your last question, I believe that the script did not find your stage 1 files. Please make sure that you have edited the paths in the script appropriately (lines 13 and 15 for input/output directories, respectively).

[anibali]

I've added code to the script which checks whether the input directory can be found, so you can run the updated version of the script to confirm whether that's the issue if you like.

[monjoybme]

@anibali Thanks a lot, but still I'm facing same error. After stage 1, "scan.dat", "scan_metadata.json" and "nodule_01_metadata.json" have been saved in the below locations:

for LIDC-IDRI-0001: /home/user/mydir/lidc-idri-preprocessing-master/LIDC-IDRI_stage1/LIDC-IDRI-0001/3000566/

for LIDC-IDRI-0002: /home/user/mydir/lidc-idri-preprocessing-master/LIDC-IDRI_stage1/LIDC-IDRI-0002/3000522/

for LIDC-IDRI-0003: /home/user/mydir/lidc-idri-preprocessing-master/LIDC-IDRI_stage1/LIDC-IDRI-0003/3000611/scan_metadata.json In Stage 2, i have changed input and output directory in "preprocess_stage2.lua" code like below

-- Input directory
local stage1_dir = '/home/user/mydir/lidc-idri-preprocessing-master/LIDC-IDRI_stage1/'
-- Output directory
local stage2_dir = '/home/user/mydir/lidc-idri-preprocessing-master/LIDC-IDRI_stage2/'

N.B: Manually created "LIDC-IDRI_stage2" directory. The command I'm using for this is lua preprocess_stage2.lua . The error I'm getting:

$ lua preprocess_stage2.lua 
   1/   8
lua: /home/user/torch/install/share/lua/5.1/pl/path.lua:215: argument 1 expected a 'string', got a 'nil'
stack traceback:
    [C]: in function 'error'
    /home/user/torch/install/share/lua/5.1/pl/utils.lua:429: in function 'assert_arg'
    /home/user/torch/install/share/lua/5.1/pl/utils.lua:441: in function 'assert_string'
    /home/user/torch/install/share/lua/5.1/pl/path.lua:215: in function 'basename'
    preprocess_stage2.lua:65: in main chunk
    [C]: ?

Please suggest where is my mistake.

[anibali]

Please delete the output folder ("LIDC-IDRI_stage2") and allow the script to create it itself.

[anibali]

It's worth keeping in mind that the stage 2 processing is only really useful if you intend on using Torch7, since it stores data in the .t7 file format.

[monjoybme]

Thanks a lot. Could you share your developed ResNet code?

[anibali]

There are IP restrictions for that particular project, I'm afraid.

Closing this issue since the original problem has been solved.

[monjoybme]

I'm facing this error when running lua preprocess_stage2.lua

lua preprocess_stage2.lua 
   1/   8
lua: Trying to resize storage that is not resizable at /home/User/torch/pkg/torch/lib/TH/generic/THStorage.c:183
stack traceback:
    [C]: ?
    [C]: in function 'IntTensor'
    preprocess_stage2.lua:73: in main chunk
    [C]: ?

Any insights will be highly appreciated. Thanks.

[anibali]

It means that somehow the metadata and pixel data for a nodule do not agree in size, which should not happen.

Please note that stage 2 of the preprocessing is included here as more of a guide/reference than anything. I recommend writing your own code which works from the stage 1 output so that you can customise it to fit your needs.

[monjoybme]

thanks