OSError: [Errno 121] Remote I/O error after 12 epochs ??

[rds-itga]

Hi everybody,

I launch a training on 512x512 8 bits .png images, I got this issue after more than 12 epochs training, I really don't understand why, everything was ok before this:

[06/05 04:18:15] d2.engine.train_loop ERROR: Exception during training: Traceback (most recent call last): File "/home/appuser/detectron2_repo/detectron2/engine/train_loop.py", line 132, in train self.run_step() File "/home/appuser/detectron2_repo/detectron2/engine/train_loop.py", line 209, in run_step data = next(self._data_loader_iter) File "/home/appuser/detectron2_repo/detectron2/data/common.py", line 140, in iter for d in self.dataset: File "/home/appuser/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 345, in next data = self._next_data() File "/home/appuser/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 856, in _next_data return self._process_data(data) File "/home/appuser/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data data.reraise() File "/home/appuser/.local/lib/python3.6/site-packages/torch/_utils.py", line 394, in reraise raise self.exc_type(msg) OSError: Caught OSError in DataLoader worker process 0. Original Traceback (most recent call last): File "/home/appuser/.local/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop data = fetcher.fetch(index) File "/home/appuser/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/appuser/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/appuser/detectron2_repo/detectron2/data/common.py", line 41, in getitem data = self._map_func(self._dataset[cur_idx]) File "/home/appuser/detectron2_repo/detectron2/utils/serialize.py", line 23, in call return self._obj(*args, **kwargs) File "/home/appuser/detectron2_repo/detectron2/data/dataset_mapper.py", line 77, in call image = utils.read_image(dataset_dict["file_name"], format=self.img_format) File "/home/appuser/detectron2_repo/detectron2/data/detection_utils.py", line 49, in read_image image = Image.open(f) File "/home/appuser/.local/lib/python3.6/site-packages/PIL/Image.py", line 2818, in open prefix = fp.read(16) OSError: [Errno 121] Remote I/O error

here is the whole log.txt file: log.txt

her is my config.yaml: CUDNN_BENCHMARK: false DATALOADER: ASPECT_RATIO_GROUPING: true FILTER_EMPTY_ANNOTATIONS: true NUM_WORKERS: 4 REPEAT_THRESHOLD: 0.0 SAMPLER_TRAIN: TrainingSampler DATASETS: PRECOMPUTED_PROPOSAL_TOPK_TEST: 1000 PRECOMPUTED_PROPOSAL_TOPK_TRAIN: 2000 PROPOSAL_FILES_TEST: [] PROPOSAL_FILES_TRAIN: [] TEST:

• fibre_unitaire_synt_eval TRAIN:
• fibre_unitaire_synt_train GLOBAL: HACK: 1.0 INPUT: CROP: ENABLED: false SIZE:
  • 0.9
  • 0.9 TYPE: relative_range FORMAT: BGR MASK_FORMAT: polygon MAX_SIZE_TEST: 1024 MAX_SIZE_TRAIN: 1024 MIN_SIZE_TEST: 1024 MIN_SIZE_TRAIN:
• 1024 MIN_SIZE_TRAIN_SAMPLING: choice MODEL: ANCHOR_GENERATOR: ANGLES:
  • • -90
    • 0
    • 90 ASPECT_RATIOS:
  • • 0.5
    • 1.0
    • 2.0 NAME: DefaultAnchorGenerator OFFSET: 0.0 SIZES:
  • • 32
  • • 64
  • • 128
  • • 256
  • • 512 BACKBONE: FREEZE_AT: 2 NAME: build_resnet_fpn_backbone DEVICE: cuda FPN: FUSE_TYPE: sum IN_FEATURES:
  • res2
  • res3
  • res4
  • res5 NORM: '' OUT_CHANNELS: 256 KEYPOINT_ON: false LOAD_PROPOSALS: false MASK_ON: true META_ARCHITECTURE: GeneralizedRCNN PANOPTIC_FPN: COMBINE: ENABLED: true INSTANCES_CONFIDENCE_THRESH: 0.5 OVERLAP_THRESH: 0.5 STUFF_AREA_LIMIT: 4096 INSTANCE_LOSS_WEIGHT: 1.0 PIXEL_MEAN:
• 103.53
• 116.28
• 123.675 PIXEL_STD:
• 57.375
• 57.12
• 58.395 PROPOSAL_GENERATOR: MIN_SIZE: 0 NAME: RPN RESNETS: DEFORM_MODULATED: false DEFORM_NUM_GROUPS: 1 DEFORM_ON_PER_STAGE:
  • false
  • false
  • false
  • false DEPTH: 101 NORM: FrozenBN NUM_GROUPS: 32 OUT_FEATURES:
  • res2
  • res3
  • res4
  • res5 RES2_OUT_CHANNELS: 256 RES5_DILATION: 1 STEM_OUT_CHANNELS: 64 STRIDE_IN_1X1: false WIDTH_PER_GROUP: 8 RETINANET: BBOX_REG_WEIGHTS:
  • 1.0
  • 1.0
  • 1.0
  • 1.0 FOCAL_LOSS_ALPHA: 0.25 FOCAL_LOSS_GAMMA: 2.0 IN_FEATURES:
  • p3
  • p4
  • p5
  • p6
  • p7 IOU_LABELS:
  • 0
  • -1
  • 1 IOU_THRESHOLDS:
  • 0.4
  • 0.5 NMS_THRESH_TEST: 0.5 NUM_CLASSES: 80 NUM_CONVS: 4 PRIOR_PROB: 0.01 SCORE_THRESH_TEST: 0.05 SMOOTH_L1_LOSS_BETA: 0.1 TOPK_CANDIDATES_TEST: 1000 ROI_BOX_CASCADE_HEAD: BBOX_REG_WEIGHTS:
  • • 10.0
    • 10.0
    • 5.0
    • 5.0
  • • 20.0
    • 20.0
    • 10.0
    • 10.0
  • • 30.0
    • 30.0
    • 15.0
    • 15.0 IOUS:
  • 0.5
  • 0.6
  • 0.7 ROI_BOX_HEAD: BBOX_REG_WEIGHTS:
  • 10.0
  • 10.0
  • 5.0
  • 5.0 CLS_AGNOSTIC_BBOX_REG: false CONV_DIM: 256 FC_DIM: 1024 NAME: FastRCNNConvFCHead NORM: '' NUM_CONV: 0 NUM_FC: 2 POOLER_RESOLUTION: 7 POOLER_SAMPLING_RATIO: 0 POOLER_TYPE: ROIAlignV2 SMOOTH_L1_BETA: 0.0 TRAIN_ON_PRED_BOXES: false ROI_HEADS: BATCH_SIZE_PER_IMAGE: 128 IN_FEATURES:
  • p2
  • p3
  • p4
  • p5 IOU_LABELS:
  • 0
  • 1 IOU_THRESHOLDS:
  • 0.5 NAME: StandardROIHeads NMS_THRESH_TEST: 0.5 NUM_CLASSES: 13 POSITIVE_FRACTION: 0.25 PROPOSAL_APPEND_GT: true SCORE_THRESH_TEST: 0.05 ROI_KEYPOINT_HEAD: CONV_DIMS:
  • 512
  • 512
  • 512
  • 512
  • 512
  • 512
  • 512
  • 512 LOSS_WEIGHT: 1.0 MIN_KEYPOINTS_PER_IMAGE: 1 NAME: KRCNNConvDeconvUpsampleHead NORMALIZE_LOSS_BY_VISIBLE_KEYPOINTS: true NUM_KEYPOINTS: 17 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_TYPE: ROIAlignV2 ROI_MASK_HEAD: CLS_AGNOSTIC_MASK: false CONV_DIM: 256 NAME: MaskRCNNConvUpsampleHead NORM: '' NUM_CONV: 4 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_TYPE: ROIAlignV2 RPN: BATCH_SIZE_PER_IMAGE: 256 BBOX_REG_WEIGHTS:
  • 1.0
  • 1.0
  • 1.0
  • 1.0 BOUNDARY_THRESH: -1 HEAD_NAME: StandardRPNHead IN_FEATURES:
  • p2
  • p3
  • p4
  • p5
  • p6 IOU_LABELS:
  • 0
  • -1
  • 1 IOU_THRESHOLDS:
  • 0.3
  • 0.7 LOSS_WEIGHT: 1.0 NMS_THRESH: 0.7 POSITIVE_FRACTION: 0.5 POST_NMS_TOPK_TEST: 1000 POST_NMS_TOPK_TRAIN: 1000 PRE_NMS_TOPK_TEST: 1000 PRE_NMS_TOPK_TRAIN: 2000 SMOOTH_L1_BETA: 0.0 SEM_SEG_HEAD: COMMON_STRIDE: 4 CONVS_DIM: 128 IGNORE_VALUE: 255 IN_FEATURES:
  • p2
  • p3
  • p4
  • p5 LOSS_WEIGHT: 1.0 NAME: SemSegFPNHead NORM: GN NUM_CLASSES: 54 WEIGHTS: detectron2://COCO-InstanceSegmentation/mask_rcnn_X_101_32x8d_FPN_3x/139653917/model_final_2d9806.pkl OUTPUT_DIR: ./output SEED: -1 SOLVER: BASE_LR: 0.00025 BIAS_LR_FACTOR: 1.0 CHECKPOINT_PERIOD: 8670 CLIP_GRADIENTS: CLIP_TYPE: value CLIP_VALUE: 1.0 ENABLED: false NORM_TYPE: 2.0 GAMMA: 0.1 IMS_PER_BATCH: 8 LR_SCHEDULER_NAME: WarmupMultiStepLR MAX_ITER: 173390 MOMENTUM: 0.9 STEPS:
• 43348
• 86695
• 130043 WARMUP_FACTOR: 0.001 WARMUP_ITERS: 1000 WARMUP_METHOD: linear WEIGHT_DECAY: 0.0001 WEIGHT_DECAY_BIAS: 0.0001 WEIGHT_DECAY_NORM: 0.0 TEST: AUG: ENABLED: false FLIP: true MAX_SIZE: 4000 MIN_SIZES:
  • 400
  • 500
  • 600
  • 700
  • 800
  • 900
  • 1000
  • 1100
  • 1200 DETECTIONS_PER_IMAGE: 100 EVAL_PERIOD: 8670 EXPECTED_RESULTS: [] KEYPOINT_OKS_SIGMAS: [] PRECISE_BN: ENABLED: false NUM_ITER: 200 VERSION: 2 VIS_PERIOD: 0

[ppwwyyxx]

According to the stack trace PIL cannot read the image file.