search

PeterJackNaylor / DRFNS

This repository contains the code necessary in order to reproduce the work contained in the submitted paper: "Segmentation of Nuclei in Histopathology Images by deep regression of the distance map".
MIT License
47 stars 13 forks source link

Error when running the pipeline with the real data #1

Closed QingboKang closed 5 years ago

QingboKang commented 5 years ago

When I try to run the pipeline with the real data, I have already download the image dataset, but an error also occurs, below is the details of the error: N E X T F L O W ~ version 18.10.1 Launching realdataset.nf [magical_fermat] - revision: a123f1f4db [warm up] executor > local [70/e05036] Submitted process > ChangeInput (1) [9a/411b81] Submitted process > ChangeInput (2) [7b/dc403c] Submitted process > BinToDistance [69/dbe9ce] Submitted process > Mean (1) [06/9490fb] Submitted process > CreateRecords (3) [63/4c30bd] Submitted process > CreateRecords (2) [c6/da1e67] Submitted process > CreateRecords (1) [d4/14ea00] Submitted process > Mean (3) [01/2ae9df] Submitted process > CreateRecords (5) [7a/3cc304] Submitted process > Mean (2) [a9/49338d] Submitted process > CreateRecords (4) [20/cb397a] Submitted process > CreateRecords (6) [9c/c07d37] Submitted process > CreateRecords (7) [4a/c6c5e0] Submitted process > CreateRecords (8) [af/e8b5d3] Submitted process > CreateRecords (9) ERROR ~ Error executing process > 'Mean (1)'

Caused by: Process Mean (1) terminated with an error exit status (1)

Command executed:

python MeanCalculation.py --path ImageFolder --output .

Command exit status: 1

It seems that something is wrong in src_ReadData/Data/DataGenRandomT.py file: raise ValueError("sample larger than population") ValueError: sample larger than population

Can anyone help me with this?

Thanks.

PeterJackNaylor commented 5 years ago

Hi, I guess this has something to do with the how the data gets downloaded, could you check that your data is found correctly? For instance, in my local directory if I use this command line: ''' ls datafolder/ForDataGenTrainTestVal''' I get: GT_bladder Slide_bladder ... and so one. Best,

QingboKang commented 5 years ago

Hi, I guess this has something to do with the how the data gets downloaded, could you check that your data is found correctly? For instance, in my local directory if I use this command line: ''' ls datafolder/ForDataGenTrainTestVal''' I get: GT_bladder Slide_bladder ... and so one. Best,

The problem is solved, just as you guessed, it is the data problem. Thanks.

But another error occurred when it run CreateRecords (7) process. The detailed log about the error: Jan-05 15:09:58.221 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 13; name: CreateRecords (7); status: COMPLETED; exit: 137; error: -; workDir: /home/qingbo/Documents/DRFNS-master/work/83/6af659bcbd36b5352100f0374d99a7] Jan-05 15:09:58.369 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'CreateRecords (7)'

Caused by: Process CreateRecords (7) terminated with an error exit status (137)

Command executed:

python TFRecords.py --tf_record train_DIST.tfrecords --split train --path ToAnnotateDistance --crop 16 --UNet --size_train 212 --size_test 0 --seed 42 --epoch 80 --type JUST_READ

Command exit status: 137

Command output: (empty)

Command error: /home/qingbo/.local/lib/python2.7/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters /home/qingbo/miniconda3/envs/nuclei_2.7/lib/python2.7/site-packages/skimage/util/dtype.py:130: UserWarning: Possible precision loss when converting from float64 to uint8 .format(dtypeobj_in, dtypeobj_out)) .command.sh: line 2: 1899 Killed python TFRecords.py --tf_record train_DIST.tfrecords --split train --path ToAnnotateDistance --crop 16 --UNet --size_train 212 --size_test 0 --seed 42 --epoch 80 --type JUST_READ

Could you kindly give me one hint, many thanks.

PeterJackNaylor commented 5 years ago

Hi,

I don't really know, it seems that the process was killed by the system and there could many reasons why this fails. How are you running the nextflow pipeline? On a single compute/cluster? To help debug, don't hesitate to go to the workdir: /home/qingbo/Documents/DRFNS-master/work/83/6af659bcbd36b5352100f0374d99a7 and run the file '.command.sh' or the file '.command.run'.

GekFreeman commented 4 years ago

Hi, I guess this has something to do with the how the data gets downloaded, could you check that your data is found correctly? For instance, in my local directory if I use this command line: ''' ls datafolder/ForDataGenTrainTestVal''' I get: GT_bladder Slide_bladder ... and so one. Best,

The problem is solved, just as you guessed, it is the data problem. Thanks.

But another error occurred when it run CreateRecords (7) process. The detailed log about the error: Jan-05 15:09:58.221 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 13; name: CreateRecords (7); status: COMPLETED; exit: 137; error: -; workDir: /home/qingbo/Documents/DRFNS-master/work/83/6af659bcbd36b5352100f0374d99a7] Jan-05 15:09:58.369 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'CreateRecords (7)'

Caused by: Process CreateRecords (7) terminated with an error exit status (137)

Command executed:

python TFRecords.py --tf_record train_DIST.tfrecords --split train --path ToAnnotateDistance --crop 16 --UNet --size_train 212 --size_test 0 --seed 42 --epoch 80 --type JUST_READ

Command exit status: 137

Command output: (empty)

Command error: /home/qingbo/.local/lib/python2.7/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters /home/qingbo/miniconda3/envs/nuclei_2.7/lib/python2.7/site-packages/skimage/util/dtype.py:130: UserWarning: Possible precision loss when converting from float64 to uint8 .format(dtypeobj_in, dtypeobj_out)) .command.sh: line 2: 1899 Killed python TFRecords.py --tf_record train_DIST.tfrecords --split train --path ToAnnotateDistance --crop 16 --UNet --size_train 212 --size_test 0 --seed 42 --epoch 80 --type JUST_READ

Could you kindly give me one hint, many thanks.

How did you solve this problem? I have the same bug recently, could you help me with this? Thanks a lot!!

wang97zh commented 3 years ago

Hi, I guess this has something to do with the how the data gets downloaded, could you check that your data is found correctly? For instance, in my local directory if I use this command line: ''' ls datafolder/ForDataGenTrainTestVal''' I get: GT_bladder Slide_bladder ... and so one. Best,

The problem is solved, just as you guessed, it is the data problem. Thanks. But another error occurred when it run CreateRecords (7) process. The detailed log about the error: Jan-05 15:09:58.221 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 13; name: CreateRecords (7); status: COMPLETED; exit: 137; error: -; workDir: /home/qingbo/Documents/DRFNS-master/work/83/6af659bcbd36b5352100f0374d99a7] Jan-05 15:09:58.369 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'CreateRecords (7)' Caused by: Process CreateRecords (7) terminated with an error exit status (137) Command executed: python TFRecords.py --tf_record train_DIST.tfrecords --split train --path ToAnnotateDistance --crop 16 --UNet --size_train 212 --size_test 0 --seed 42 --epoch 80 --type JUST_READ Command exit status: 137 Command output: (empty) Command error: /home/qingbo/.local/lib/python2.7/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters /home/qingbo/miniconda3/envs/nuclei_2.7/lib/python2.7/site-packages/skimage/util/dtype.py:130: UserWarning: Possible precision loss when converting from float64 to uint8 .format(dtypeobj_in, dtypeobj_out)) .command.sh: line 2: 1899 Killed python TFRecords.py --tf_record train_DIST.tfrecords --split train --path ToAnnotateDistance --crop 16 --UNet --size_train 212 --size_test 0 --seed 42 --epoch 80 --type JUST_READ Could you kindly give me one hint, many thanks.

How did you solve this problem? I have the same bug recently, could you help me with this? Thanks a lot!!

Did you solve this error? I met the same problem.