search

junqiangchen / BraTS18-Challege

Multimodal Brain Tumor Segmentation Challenge 2018
https://www.med.upenn.edu/sbia/brats2018.html
50 stars 12 forks source link

MemoryError #2

Open Merofine opened 4 years ago

Merofine commented 4 years ago

(brats) F:\Project\brainseg\code1\BraTS18Challegemaster\BraTs18Challege\dataprocess>python data3dprepare.py sub_dirs: ['Brats18_2013_10_1', 'Brats18_2013_11_1', 'Brats18_2013_12_1', 'Brats18_2013_13_1', 'Brats18_2013_14_1', 'Brats18_2013_17_1', 'Brats18_2013_18_1', 'Brats18_2013_19_1', 'Brats18_2013_20_1', 'Brats18_2013_21_1', 'Brats18_2013_22_1', 'Brats18_2013_23_1', 'Brats18_2013_25_1', 'Brats18_2013_26_1', 'Brats18_2013_27_1', 'Brats18_2013_2_1', 'Brats18_2013_3_1', 'Brats18_2013_4_1', 'Brats18_2013_5_1', 'Brats18_2013_7_1', 'Brats18_CBICA_AAB_1', 'Brats18_CBICA_AAG_1', 'Brats18_CBICA_AAL_1', 'Brats18_CBICA_AAP_1', 'Brats18_CBICA_ABB_1', 'Brats18_CBICA_ABE_1', 'Brats18_CBICA_ABM_1', 'Brats18_CBICA_ABN_1', 'Brats18_CBICA_ABO_1', 'Brats18_CBICA_ABY_1', 'Brats18_CBICA_ALN_1', 'Brats18_CBICA_ALU_1', 'Brats18_CBICA_ALX_1', 'Brats18_CBICA_AME_1', 'Brats18_CBICA_AMH_1', 'Brats18_CBICA_ANG_1', 'Brats18_CBICA_ANI_1', 'Brats18_CBICA_ANP_1', 'Brats18_CBICA_ANZ_1', 'Brats18_CBICA_AOD_1', 'Brats18_CBICA_AOH_1', 'Brats18_CBICA_AOO_1', 'Brats18_CBICA_AOP_1', 'Brats18_CBICA_AOZ_1', 'Brats18_CBICA_APR_1', 'Brats18_CBICA_APY_1', 'Brats18_CBICA_APZ_1', 'Brats18_CBICA_AQA_1', 'Brats18_CBICA_AQD_1', 'Brats18_CBICA_AQG_1', 'Brats18_CBICA_AQJ_1', 'Brats18_CBICA_AQN_1', 'Brats18_CBICA_AQO_1', 'Brats18_CBICA_AQP_1', 'Brats18_CBICA_AQQ_1', 'Brats18_CBICA_AQR_1', 'Brats18_CBICA_AQT_1', 'Brats18_CBICA_AQU_1', 'Brats18_CBICA_AQV_1', 'Brats18_CBICA_AQY_1', 'Brats18_CBICA_AQZ_1', 'Brats18_CBICA_ARF_1', 'Brats18_CBICA_ARW_1', 'Brats18_CBICA_ARZ_1', 'Brats18_CBICA_ASA_1', 'Brats18_CBICA_ASE_1', 'Brats18_CBICA_ASG_1', 'Brats18_CBICA_ASH_1', 'Brats18_CBICA_ASK_1', 'Brats18_CBICA_ASN_1', 'Brats18_CBICA_ASO_1', 'Brats18_CBICA_ASU_1', 'Brats18_CBICA_ASV_1', 'Brats18_CBICA_ASW_1', 'Brats18_CBICA_ASY_1', 'Brats18_CBICA_ATB_1', 'Brats18_CBICA_ATD_1', 'Brats18_CBICA_ATF_1', 'Brats18_CBICA_ATP_1', 'Brats18_CBICA_ATV_1', 'Brats18_CBICA_ATX_1', 'Brats18_CBICA_AUN_1', 'Brats18_CBICA_AUQ_1', 'Brats18_CBICA_AUR_1', 'Brats18_CBICA_AVG_1', 'Brats18_CBICA_AVJ_1', 'Brats18_CBICA_AVV_1', 'Brats18_CBICA_AWG_1', 'Brats18_CBICA_AWH_1', 'Brats18_CBICA_AWI_1', 'Brats18_CBICA_AXJ_1', 'Brats18_CBICA_AXL_1', 'Brats18_CBICA_AXM_1', 'Brats18_CBICA_AXN_1', 'Brats18_CBICA_AXO_1', 'Brats18_CBICA_AXQ_1', 'Brats18_CBICA_AXW_1', 'Brats18_CBICA_AYA_1', 'Brats18_CBICA_AYI_1', 'Brats18_CBICA_AYU_1', 'Brats18_CBICA_AYW_1', 'Brats18_CBICA_AZD_1', 'Brats18_CBICA_AZH_1', 'Brats18_CBICA_BFB_1', 'Brats18_CBICA_BFP_1', 'Brats18_CBICA_BHB_1', 'Brats18_CBICA_BHK_1', 'Brats18_CBICA_BHM_1', 'Brats18_TCIA01_131_1', 'Brats18_TCIA01_147_1', 'Brats18_TCIA01_150_1', 'Brats18_TCIA01_180_1', 'Brats18_TCIA01_186_1', 'Brats18_TCIA01_190_1', 'Brats18_TCIA01_201_1', 'Brats18_TCIA01_203_1', 'Brats18_TCIA01_221_1', 'Brats18_TCIA01_231_1', 'Brats18_TCIA01_235_1', 'Brats18_TCIA01_335_1', 'Brats18_TCIA01_378_1', 'Brats18_TCIA01_390_1', 'Brats18_TCIA01_401_1', 'Brats18_TCIA01_411_1', 'Brats18_TCIA01_412_1', 'Brats18_TCIA01_425_1', 'Brats18_TCIA01_429_1', 'Brats18_TCIA01_448_1', 'Brats18_TCIA01_460_1', 'Brats18_TCIA01_499_1', 'Brats18_TCIA02_117_1', 'Brats18_TCIA02_118_1', 'Brats18_TCIA02_135_1', 'Brats18_TCIA02_151_1', 'Brats18_TCIA02_168_1', 'Brats18_TCIA02_171_1', 'Brats18_TCIA02_179_1', 'Brats18_TCIA02_198_1', 'Brats18_TCIA02_208_1', 'Brats18_TCIA02_222_1', 'Brats18_TCIA02_226_1', 'Brats18_TCIA02_274_1', 'Brats18_TCIA02_283_1', 'Brats18_TCIA02_290_1', 'Brats18_TCIA02_300_1', 'Brats18_TCIA02_309_1', 'Brats18_TCIA02_314_1', 'Brats18_TCIA02_321_1', 'Brats18_TCIA02_322_1', 'Brats18_TCIA02_331_1', 'Brats18_TCIA02_368_1', 'Brats18_TCIA02_370_1', 'Brats18_TCIA02_374_1', 'Brats18_TCIA02_377_1', 'Brats18_TCIA02_394_1', 'Brats18_TCIA02_430_1', 'Brats18_TCIA02_455_1', 'Brats18_TCIA02_471_1', 'Brats18_TCIA02_473_1', 'Brats18_TCIA02_491_1', 'Brats18_TCIA02_605_1', 'Brats18_TCIA02_606_1', 'Brats18_TCIA02_607_1', 'Brats18_TCIA02_608_1', 'Brats18_TCIA03_121_1', 'Brats18_TCIA03_133_1', 'Brats18_TCIA03_138_1', 'Brats18_TCIA03_199_1', 'Brats18_TCIA03_257_1', 'Brats18_TCIA03_265_1', 'Brats18_TCIA03_296_1', 'Brats18_TCIA03_338_1', 'Brats18_TCIA03_375_1', 'Brats18_TCIA03_419_1', 'Brats18_TCIA03_474_1', 'Brats18_TCIA03_498_1', 'Brats18_TCIA04_111_1', 'Brats18_TCIA04_149_1', 'Brats18_TCIA04_192_1', 'Brats18_TCIA04_328_1', 'Brats18_TCIA04_343_1', 'Brats18_TCIA04_361_1', 'Brats18_TCIA04_437_1', 'Brats18_TCIA04_479_1', 'Brats18_TCIA05_277_1', 'Brats18_TCIA05_396_1', 'Brats18_TCIA05_444_1', 'Brats18_TCIA05_478_1', 'Brats18_TCIA06_165_1', 'Brats18_TCIA06_184_1', 'Brats18_TCIA06_211_1', 'Brats18_TCIA06_247_1', 'Brats18_TCIA06_332_1', 'Brats18_TCIA06_372_1', 'Brats18_TCIA06_409_1', 'Brats18_TCIA06_603_1', 'Brats18_TCIA08_105_1', 'Brats18_TCIA08_113_1', 'Brats18_TCIA08_162_1', 'Brats18_TCIA08_167_1', 'Brats18_TCIA08_205_1', 'Brats18_TCIA08_218_1', 'Brats18_TCIA08_234_1', 'Brats18_TCIA08_242_1', 'Brats18_TCIA08_278_1', 'Brats18_TCIA08_280_1', 'Brats18_TCIA08_319_1', 'Brats18_TCIA08_406_1', 'Brats18_TCIA08_436_1', 'Brats18_TCIA08_469_1'] sub_dirs: ['Brats18_2013_0_1', 'Brats18_2013_15_1', 'Brats18_2013_16_1', 'Brats18_2013_1_1', 'Brats18_2013_24_1', 'Brats18_2013_28_1', 'Brats18_2013_29_1', 'Brats18_2013_6_1', 'Brats18_2013_8_1', 'Brats18_2013_9_1', 'Brats18_TCIA09_141_1', 'Brats18_TCIA09_177_1', 'Brats18_TCIA09_254_1', 'Brats18_TCIA09_255_1', 'Brats18_TCIA09_312_1', 'Brats18_TCIA09_402_1', 'Brats18_TCIA09_428_1', 'Brats18_TCIA09_451_1', 'Brats18_TCIA09_462_1', 'Brats18_TCIA09_493_1', 'Brats18_TCIA09_620_1', 'Brats18_TCIA10_103_1', 'Brats18_TCIA10_109_1', 'Brats18_TCIA10_130_1', 'Brats18_TCIA10_152_1', 'Brats18_TCIA10_175_1', 'Brats18_TCIA10_202_1', 'Brats18_TCIA10_241_1', 'Brats18_TCIA10_261_1', 'Brats18_TCIA10_266_1', 'Brats18_TCIA10_276_1', 'Brats18_TCIA10_282_1', 'Brats18_TCIA10_299_1', 'Brats18_TCIA10_307_1', 'Brats18_TCIA10_310_1', 'Brats18_TCIA10_325_1', 'Brats18_TCIA10_330_1', 'Brats18_TCIA10_346_1', 'Brats18_TCIA10_351_1', 'Brats18_TCIA10_387_1', 'Brats18_TCIA10_393_1', 'Brats18_TCIA10_408_1', 'Brats18_TCIA10_410_1', 'Brats18_TCIA10_413_1', 'Brats18_TCIA10_420_1', 'Brats18_TCIA10_442_1', 'Brats18_TCIA10_449_1', 'Brats18_TCIA10_490_1', 'Brats18_TCIA10_625_1', 'Brats18_TCIA10_628_1', 'Brats18_TCIA10_629_1', 'Brats18_TCIA10_632_1', 'Brats18_TCIA10_637_1', 'Brats18_TCIA10_639_1', 'Brats18_TCIA10_640_1', 'Brats18_TCIA10_644_1', 'Brats18_TCIA12_101_1', 'Brats18_TCIA12_249_1', 'Brats18_TCIA12_298_1', 'Brats18_TCIA12_466_1', 'Brats18_TCIA12_470_1', 'Brats18_TCIA12_480_1', 'Brats18_TCIA13_615_1', 'Brats18_TCIA13_618_1', 'Brats18_TCIA13_621_1', 'Brats18_TCIA13_623_1', 'Brats18_TCIA13_624_1', 'Brats18_TCIA13_630_1', 'Brats18_TCIA13_633_1', 'Brats18_TCIA13_634_1', 'Brats18_TCIA13_642_1', 'Brats18_TCIA13_645_1', 'Brats18_TCIA13_650_1', 'Brats18_TCIA13_653_1', 'Brats18_TCIA13_654_1'] Traceback (most recent call last): File "data3dprepare.py", line 213, in preparetraindata() File "data3dprepare.py", line 209, in preparetraindata prepare3dtraindata(pathhgg_list, bratshgg_path, trainImage, trainMask, (64, 128, 128), 3, 15, 1) File "data3dprepare.py", line 193, in prepare3dtraindata numberxy=numberxy, numberz=numberz, trainImage=trainImage, trainMask=trainMask, part=part) File "data3dprepare.py", line 94, in gen_image_mask sub_flairimages,sub_maskimages = make_patch(flairimg,segimg, patch_block_size=shape, numberxy=numberxy, numberz=numberz) File "data3dprepare.py", line 72, in make_patch numberxy=numberxy, numberz=numberz) File "data3dprepare.py", line 46, in subimage_generator hr_samples = np.array(hr_samples_list).reshape((len(hr_samples_list), blockz, block_width, block_height))

MemoryError

my env (Anaconda3 64bit): python is 64bit; so i don't down how to deel with it,please tell me thanks.

Merofine commented 4 years ago

i deel with it.

Merofine commented 4 years ago

 Change the size of stride，try bigger

------------------ Original ------------------ From: BIGheqiduo <notifications@github.com> Date: Tue,Dec 24,2019 4:50 PM To: junqiangchen/BraTS18-Challege <BraTS18-Challege@noreply.github.com> Cc: Morofine <704783475@qq.com>, Author <author@noreply.github.com> Subject: Re: [junqiangchen/BraTS18-Challege] MemoryError (#2)

i deel with it.

How to deal with this issue? Could you offer me a useful suggestion ?Lin

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

nehaghatia commented 4 years ago

@Merofine can you tell which stride size work for you to resolve memory error? I am also getting the same memory error.

zhangshuang317 commented 4 years ago

你可以修改这行代码 prepare3dtraindata(pathlgg_list, bratslgg_path, trainImage, trainMask, (64, 128, 128), 3, 15, 2) 把他修改为 prepare3dtraindata(pathlgg_list, bratslgg_path, trainImage, trainMask, (64, 128, 128), 12, 60, 8) 就可以解决这个问题

675492062 commented 4 years ago

你可以修改这行代码 prepare3dtraindata(pathlgg_list, bratslgg_path, trainImage, trainMask, (64, 128, 128), 3, 15, 2) 把他修改为 prepare3dtraindata(pathlgg_list, bratslgg_path, trainImage, trainMask, (64, 128, 128), 12, 60, 8) 就可以解决这个问题

对硬盘容量也是一个考验

zhangshuang317 commented 4 years ago

你可以修改这行代码 prepare3dtraindata(pathlgg_list, bratslgg_path, trainImage, trainMask, (64, 128, 128), 3, 15, 2) 把他修改为 prepare3dtraindata(pathlgg_list, bratslgg_path, trainImage, trainMask, (64, 128, 128), 12, 60, 8) 就可以解决这个问题

对硬盘容量也是一个考验

是的，大概400多g

675492062 commented 4 years ago

你可以修改这行代码 prepare3dtraindata(pathlgg_list, bratslgg_path, trainImage, trainMask, (64, 128, 128), 3, 15, 2) 把他修改为 prepare3dtraindata(pathlgg_list, bratslgg_path, trainImage, trainMask, (64, 128, 128), 12, 60, 8) 就可以解决这个问题

对硬盘容量也是一个考验

是的，大概400多g

So, this sampling and crop operation should be done in the fly

cqlouis commented 3 years ago

你可以修改这行代码 prepare3dtraindata(pathlgg_list, bratslgg_path, trainImage, trainMask, (64, 128, 128), 3, 15, 2) 把他修改为 prepare3dtraindata(pathlgg_list, bratslgg_path, trainImage, trainMask, (64, 128, 128), 12, 60, 8) 就可以解决这个问题

对硬盘容量也是一个考验

是的，大概400多g

So, this sampling and crop operation should be done in the fly

def prepare3dtraindata(pathhgg_list, bratshgg_path, trainImage, trainMask, shape=(16, 256, 256), numberxy=3, numberz=20, part=1):

请问各位大佬，这四个参数是什么意思啊？shape=(16, 256, 256), numberxy=3, numberz=20, part=1 初学，点拨一下吧，谢谢

junqiangchen commented 3 years ago

你可以修改这行代码 prepare3dtraindata(pathlgg_list, bratslgg_path, trainImage, trainMask, (64, 128, 128), 3, 15, 2) 把他修改为 prepare3dtraindata(pathlgg_list, bratslgg_path, trainImage, trainMask, (64, 128, 128), 12, 60, 8) 就可以解决这个问题

这是正解

junqiangchen commented 3 years ago

你可以修改这行代码 prepare3dtraindata(pathlgg_list, bratslgg_path, trainImage, trainMask, (64, 128, 128), 3, 15, 2) 把他修改为 prepare3dtraindata(pathlgg_list, bratslgg_path, trainImage, trainMask, (64, 128, 128), 12, 60, 8) 就可以解决这个问题

对硬盘容量也是一个考验

是的，大概400多g

So, this sampling and crop operation should be done in the fly

def prepare3dtraindata(pathhgg_list, bratshgg_path, trainImage, trainMask, shape=(16, 256, 256), numberxy=3, numberz=20, part=1):

请问各位大佬，这四个参数是什么意思啊？shape=(16, 256, 256), numberxy=3, numberz=20, part=1 初学，点拨一下吧，谢谢

你好，这几个参数是裁切patch的图像大小，以及在裁切过程中按照xyz方向上的步长参数设置，由于当初考虑不周，导致实际使用时对内存和硬盘要求比较高，如果出现内存错误，请将xyz方向上步长参数设置的稍微大一些即可。