_pickle.UnpicklingError: pickle data was truncated

[yehuanran]

When I run train_net I got this.My datasets are coco2014. Traceback (most recent call last): File "tools/train_net.py", line 171, in main() File "tools/train_net.py", line 164, in main model = train(cfg, args.local_rank, args.distributed) File "tools/train_net.py", line 53, in train extra_checkpoint_data = checkpointer.load(cfg.MODEL.WEIGHT) File "/home/huanran/anaconda3/envs/maskrcnn_benchmark/maskscoring_rcnn/maskrcnn_benchmark/utils/checkpoint.py", line 61, in load checkpoint = self._load_file(f) File "/home/huanran/anaconda3/envs/maskrcnn_benchmark/maskscoring_rcnn/maskrcnn_benchmark/utils/checkpoint.py", line 133, in _load_file return load_c2_format(self.cfg, f) File "/home/huanran/anaconda3/envs/maskrcnn_benchmark/maskscoring_rcnn/maskrcnn_benchmark/utils/c2_model_loading.py", line 155, in load_c2_format return C2_FORMAT_LOADER[cfg.MODEL.BACKBONE.CONV_BODY](cfg, f) File "/home/huanran/anaconda3/envs/maskrcnn_benchmark/maskscoring_rcnn/maskrcnn_benchmark/utils/c2_model_loading.py", line 146, in load_resnet_c2_format state_dict = _load_c2_pickled_weights(f) File "/home/huanran/anaconda3/envs/maskrcnn_benchmark/maskscoring_rcnn/maskrcnn_benchmark/utils/c2_model_loading.py", line 124, in _load_c2_pickled_weights data = pickle.load(f, encoding="latin1") _pickle.UnpicklingError: pickle data was truncated. I don't know why.

[zjhuang22]

Hi, have you put the pre-training model in the pretrained_models directory?

[yehuanran]

Hi, have you put the pre-training model in the pretrained_models directory?

yes,I put the pre-training model in the pretrained_models directory.The model is R-50.pkl,is right?

[zjhuang22]

Could you show more details? Such as your running scripts.

[yehuanran]

Could you show more details? Such as your running scripts.

log.txt just like this: 2019-04-18 16:20:52,353 maskrcnn_benchmark INFO: Using 1 GPUs 2019-04-18 16:20:52,353 maskrcnn_benchmark INFO: Namespace(config_file='configs/e2e_ms_rcnn_R_50_FPN_1x.yaml', distributed=False, local_rank=0, opts=['SOLVER.IMS_PER_BATCH', '2', 'SOLVER.BASE_LR', '0.0025', 'SOLVER.MAX_ITER', '720000', 'SOLVER.STEPS', '(480000, 640000)', 'TEST.IMS_PER_BATCH', '1'], skip_test=False) 2019-04-18 16:20:52,353 maskrcnn_benchmark INFO: Collecting env info (might take some time) 2019-04-18 16:20:55,244 maskrcnn_benchmark INFO: PyTorch version: 1.0.1.post2 Is debug build: No CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04 LTS GCC version: (Ubuntu 4.9.3-13ubuntu2) 4.9.3 CMake version: version 3.5.1

Python version: 3.7 Is CUDA available: Yes CUDA runtime version: 9.0.176 GPU models and configuration: GPU 0: GeForce GTX 1070 Ti Nvidia driver version: 390.87 cuDNN version: Probably one of the following: /usr/local/cuda-9.0/lib64/libcudnn.so.7 /usr/local/cuda-9.0/lib64/libcudnn.so.7.1.4 /usr/local/cuda-9.0/lib64/libcudnn_static.a

Versions of relevant libraries: [pip] Could not collect [conda] pytorch 1.0.1 py3.7_cuda9.0.176_cudnn7.4.2_2 pytorch [conda] torchvision 0.2.2 py_3 pytorch Pillow (5.4.1) 2019-04-18 16:20:55,245 maskrcnn_benchmark INFO: Loaded configuration file configs/e2e_ms_rcnn_R_50_FPN_1x.yaml 2019-04-18 16:20:55,245 maskrcnn_benchmark INFO: MODEL: META_ARCHITECTURE: "GeneralizedRCNN" WEIGHT: "catalog://ImageNetPretrained/MSRA/R-50" PRETRAINED_MODELS: 'pretrained_models' BACKBONE: CONV_BODY: "R-50-FPN" OUT_CHANNELS: 256 RPN: USE_FPN: True ANCHOR_STRIDE: (4, 8, 16, 32, 64) PRE_NMS_TOP_N_TRAIN: 2000 PRE_NMS_TOP_N_TEST: 1000 POST_NMS_TOP_N_TEST: 1000 FPN_POST_NMS_TOP_N_TEST: 1000 ROI_HEADS: USE_FPN: True ROI_BOX_HEAD: POOLER_RESOLUTION: 7 POOLER_SCALES: (0.25, 0.125, 0.0625, 0.03125) POOLER_SAMPLING_RATIO: 2 FEATURE_EXTRACTOR: "FPN2MLPFeatureExtractor" PREDICTOR: "FPNPredictor" ROI_MASK_HEAD: POOLER_SCALES: (0.25, 0.125, 0.0625, 0.03125) FEATURE_EXTRACTOR: "MaskRCNNFPNFeatureExtractor" PREDICTOR: "MaskRCNNC4Predictor" POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 2 RESOLUTION: 28 SHARE_BOX_FEATURE_EXTRACTOR: False MASK_ON: True MASKIOU_ON: True DATASETS: TRAIN: ("coco_2014_train", "coco_2014_valminusminival")
TEST: ("coco_2014_minival",) DATALOADER: SIZE_DIVISIBILITY: 32 SOLVER: BASE_LR: 0.02 WEIGHT_DECAY: 0.0001 STEPS: (60000, 80000) MAX_ITER: 90000

2019-04-18 16:20:55,245 maskrcnn_benchmark INFO: Running with config: DATALOADER: ASPECT_RATIO_GROUPING: True NUM_WORKERS: 4 SIZE_DIVISIBILITY: 32 DATASETS: TEST: ('coco_2014_minival',) TRAIN: ('coco_2014_train', 'coco_2014_valminusminival') INPUT: MAX_SIZE_TEST: 1333 MAX_SIZE_TRAIN: 1333 MIN_SIZE_TEST: 800 MIN_SIZE_TRAIN: 800 PIXEL_MEAN: [102.9801, 115.9465, 122.7717] PIXEL_STD: [1.0, 1.0, 1.0] TO_BGR255: True MODEL: BACKBONE: CONV_BODY: R-50-FPN FREEZE_CONV_BODY_AT: 2 OUT_CHANNELS: 256 DEVICE: cuda MASKIOU_LOSS_WEIGHT: 1.0 MASKIOU_ON: True MASK_ON: True META_ARCHITECTURE: GeneralizedRCNN PRETRAINED_MODELS: pretrained_models RESNETS: NUM_GROUPS: 1 RES2_OUT_CHANNELS: 256 RES5_DILATION: 1 STEM_FUNC: StemWithFixedBatchNorm STEM_OUT_CHANNELS: 64 STRIDE_IN_1X1: True TRANS_FUNC: BottleneckWithFixedBatchNorm WIDTH_PER_GROUP: 64 ROI_BOX_HEAD: FEATURE_EXTRACTOR: FPN2MLPFeatureExtractor MLP_HEAD_DIM: 1024 NUM_CLASSES: 81 POOLER_RESOLUTION: 7 POOLER_SAMPLING_RATIO: 2 POOLER_SCALES: (0.25, 0.125, 0.0625, 0.03125) PREDICTOR: FPNPredictor ROI_HEADS: BATCH_SIZE_PER_IMAGE: 512 BBOX_REG_WEIGHTS: (10.0, 10.0, 5.0, 5.0) BG_IOU_THRESHOLD: 0.5 DETECTIONS_PER_IMG: 100 FG_IOU_THRESHOLD: 0.5 NMS: 0.5 POSITIVE_FRACTION: 0.25 SCORE_THRESH: 0.05 USE_FPN: True ROI_MASK_HEAD: CONV_LAYERS: (256, 256, 256, 256) FEATURE_EXTRACTOR: MaskRCNNFPNFeatureExtractor MLP_HEAD_DIM: 1024 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 2 POOLER_SCALES: (0.25, 0.125, 0.0625, 0.03125) POSTPROCESS_MASKS: False POSTPROCESS_MASKS_THRESHOLD: 0.5 PREDICTOR: MaskRCNNC4Predictor RESOLUTION: 28 SHARE_BOX_FEATURE_EXTRACTOR: False RPN: ANCHOR_SIZES: (32, 64, 128, 256, 512) ANCHOR_STRIDE: (4, 8, 16, 32, 64) ASPECT_RATIOS: (0.5, 1.0, 2.0) BATCH_SIZE_PER_IMAGE: 256 BG_IOU_THRESHOLD: 0.3 FG_IOU_THRESHOLD: 0.7 FPN_POST_NMS_TOP_N_TEST: 1000 FPN_POST_NMS_TOP_N_TRAIN: 2000 MIN_SIZE: 0 NMS_THRESH: 0.7 POSITIVE_FRACTION: 0.5 POST_NMS_TOP_N_TEST: 1000 POST_NMS_TOP_N_TRAIN: 2000 PRE_NMS_TOP_N_TEST: 1000 PRE_NMS_TOP_N_TRAIN: 2000 RPN_HEAD: SingleConvRPNHead STRADDLE_THRESH: 0 USE_FPN: True RPN_ONLY: False WEIGHT: catalog://ImageNetPretrained/MSRA/R-50 OUTPUT_DIR: models/ PATHS_CATALOG: /home/huanran/anaconda3/envs/maskrcnn_benchmark/maskscoring_rcnn/maskrcnn_benchmark/config/paths_catalog.py SOLVER: BASE_LR: 0.0025 BIAS_LR_FACTOR: 2 CHECKPOINT_PERIOD: 10000 GAMMA: 0.1 IMS_PER_BATCH: 2 MAX_ITER: 720000 MOMENTUM: 0.9 STEPS: (480000, 640000) WARMUP_FACTOR: 0.3333333333333333 WARMUP_ITERS: 500 WARMUP_METHOD: linear WEIGHT_DECAY: 0.0001 WEIGHT_DECAY_BIAS: 0 TEST: EXPECTED_RESULTS: [] EXPECTED_RESULTS_SIGMA_TOL: 4 IMS_PER_BATCH: 1 2019-04-18 16:20:57,746 maskrcnn_benchmark.utils.checkpoint INFO: Loading checkpoint from catalog://ImageNetPretrained/MSRA/R-50 2019-04-18 16:20:57,747 maskrcnn_benchmark.utils.checkpoint INFO: catalog://ImageNetPretrained/MSRA/R-50 points to https://s3-us-west-2.amazonaws.com/detectron/ImageNetPretrained/MSRA/R-50.pkl 2019-04-18 16:20:57,747 maskrcnn_benchmark.utils.checkpoint INFO: url https://s3-us-west-2.amazonaws.com/detectron/ImageNetPretrained/MSRA/R-50.pkl cached in pretrained_models/R-50.pkl

[yehuanran]

So Do you know how to deal with this problem?

[zjhuang22]

Do you install it according to INSTALL.md?

[anshumankmr]

Did you fix it @yehuanran Edit: Never mind, I got it.

[kongjibai]

@anshumankmr How did you fix it? I met the same problem. Edit: I got it. Because the pretrained model not download completly.