FloatingPointError: Loss became infinite or NaN at iteration=1002!

czsc123 commented 4 years ago

[05/27 09:32:48 detectron2]: Rank of current process: 0. World size: 1 [05/27 09:32:48 detectron2]: Environment info:

sys.platform linux Python 3.7.7 (default, Mar 26 2020, 15:48:22) [GCC 7.3.0] numpy 1.18.1 detectron2 0.1.2 @./detectron2-master/detectron2 detectron2 compiler GCC 7.5 detectron2 CUDA compiler 10.0 detectron2 arch flags sm_75 DETECTRON2_ENV_MODULE PyTorch 1.4.0 @python3.7/site-packages/torch PyTorch debug build False CUDA available True GPU 0 Graphics Device CUDA_HOME /usr/local/cuda NVCC Cuda compilation tools, release 10.0, V10.0.130 Pillow 7.1.2 torchvision 0.5.0 @python3.7/site-packages/torchvision torchvision arch flags sm_35, sm_50, sm_60, sm_70, sm_75 fvcore 0.1.1.post200513 cv2 4.2.0

PyTorch built with:

GCC 7.3
Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v0.21.1 (Git Hash 7d2fd500bc78936d1d648ca713b901012f470dbc)
OpenMP 201511 (a.k.a. OpenMP 4.5)
NNPACK is enabled
CUDA Runtime 10.0
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
CuDNN 7.6.3
Magma 2.5.1
Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,

[05/27 09:32:48 detectron2]: Command line arguments: Namespace(config_file='/meshrcnn/configs/pix3d/meshrcnn_R50_FPN_HD.yaml', dist_url='tcp://127.0.0.1:10112', eval_only=False, machine_rank=0, num_gpus=1, num_machines=1, opts=[], resume=False) [05/27 09:32:48 detectron2]: Contents of args.config_file=/meshrcnn/configs/pix3d/meshrcnn_R50_FPN_HD.yaml: BASE: "MESH-RCNN-FPN_HD.yaml" MODEL: WEIGHTS: "/meshrcnn/R-50.pkl" MASK_ON: True VOXEL_ON: True MESH_ON: True ZPRED_ON: True RESNETS: DEPTH: 50 RPN: IOU_THRESHOLDS: [0.2, 0.5, 0.7] IOU_LABELS: [-1, 0, -1, 1] SMOOTH_L1_BETA: 0.111 ROI_HEADS: NAME: "MeshRCNNROIHeads" BATCH_SIZE_PER_IMAGE: 64 NUM_CLASSES: 2 #9 # Number of foreground classes IOU_THRESHOLDS: [0.2, 0.5] IOU_LABELS: [-1, 0, 1] ROI_BOX_HEAD: SMOOTH_L1_BETA: 1.0 ROI_Z_HEAD: NAME: "FastRCNNFCHead" Z_REG_WEIGHT: 1.0 SMOOTH_L1_BETA: 1.0 ROI_MASK_HEAD: NAME: "MaskRCNNConvUpsampleHead" POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 2 NUM_CONV: 4 ROI_VOXEL_HEAD: NAME: "VoxelRCNNConvUpsampleHead" POOLER_RESOLUTION: 12
POOLER_SAMPLING_RATIO: 2 NUM_CONV: 4 NUM_DEPTH: 24 CLS_AGNOSTIC_VOXEL: True LOSS_WEIGHT: 3.0 CUBIFY_THRESH: 0.2 ROI_MESH_HEAD: NAME: "MeshRCNNGraphConvHead" POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 2 NUM_STAGES: 3 NUM_GRAPH_CONVS: 3 GRAPH_CONV_DIM: 128 GRAPH_CONV_INIT: "normal" GT_COORD_THRESH: 5.0 CHAMFER_LOSS_WEIGHT: 1.0 NORMALS_LOSS_WEIGHT: 0.1 EDGE_LOSS_WEIGHT: 1.0 DATASETS: TRAIN: ("bed_bookcase_s1_train",) TEST: ("bed_bookcase_s1_test",) SOLVER: BASE_LR: 0.0025 #0.02 WEIGHT_DECAY: 0.0001 STEPS: (64000, 80000) #(8000, 10000) MAX_ITER: 88000 # 11000 WARMUP_ITERS: 1000 WARMUP_FACTOR: 0.1 IMS_PER_BATCH: 1

[05/27 09:32:48 detectron2]: Running with full config: CUDNN_BENCHMARK: False DATALOADER: ASPECT_RATIO_GROUPING: False FILTER_EMPTY_ANNOTATIONS: True NUM_WORKERS: 4 REPEAT_THRESHOLD: 0.0 SAMPLER_TRAIN: TrainingSampler DATASETS: PRECOMPUTED_PROPOSAL_TOPK_TEST: 1000 PRECOMPUTED_PROPOSAL_TOPK_TRAIN: 2000 PROPOSAL_FILES_TEST: () PROPOSAL_FILES_TRAIN: () TEST: ('bed_bookcase_s1_test',) TRAIN: ('bed_bookcase_s1_train',) GLOBAL: HACK: 1.0 INPUT: CROP: ENABLED: False SIZE: [0.9, 0.9] TYPE: relative_range FORMAT: BGR MASK_FORMAT: polygon MAX_SIZE_TEST: 1333 MAX_SIZE_TRAIN: 1333 MIN_SIZE_TEST: 800 MIN_SIZE_TRAIN: (800,) MIN_SIZE_TRAIN_SAMPLING: choice MODEL: ANCHOR_GENERATOR: ANGLES: [[-90, 0, 90]] ASPECT_RATIOS: [[0.5, 1.0, 2.0]] NAME: DefaultAnchorGenerator OFFSET: 0.0 SIZES: [[32], [64], [128], [256], [512]] BACKBONE: FREEZE_AT: 2 NAME: build_resnet_fpn_backbone DEVICE: cuda FPN: FUSE_TYPE: sum IN_FEATURES: ['res2', 'res3', 'res4', 'res5'] NORM: OUT_CHANNELS: 256 KEYPOINT_ON: False LOAD_PROPOSALS: False MASK_ON: True MESH_ON: True META_ARCHITECTURE: GeneralizedRCNN PANOPTIC_FPN: COMBINE: ENABLED: True INSTANCES_CONFIDENCE_THRESH: 0.5 OVERLAP_THRESH: 0.5 STUFF_AREA_LIMIT: 4096 INSTANCE_LOSS_WEIGHT: 1.0 PIXEL_MEAN: [103.53, 116.28, 123.675] PIXEL_STD: [1.0, 1.0, 1.0] PROPOSAL_GENERATOR: MIN_SIZE: 0 NAME: RPN RESNETS: DEFORM_MODULATED: False DEFORM_NUM_GROUPS: 1 DEFORM_ON_PER_STAGE: [False, False, False, False] DEPTH: 50 NORM: FrozenBN NUM_GROUPS: 1 OUT_FEATURES: ['res2', 'res3', 'res4', 'res5'] RES2_OUT_CHANNELS: 256 RES5_DILATION: 1 STEM_OUT_CHANNELS: 64 STRIDE_IN_1X1: True WIDTH_PER_GROUP: 64 RETINANET: BBOX_REG_WEIGHTS: (1.0, 1.0, 1.0, 1.0) FOCAL_LOSS_ALPHA: 0.25 FOCAL_LOSS_GAMMA: 2.0 IN_FEATURES: ['p3', 'p4', 'p5', 'p6', 'p7'] IOU_LABELS: [0, -1, 1] IOU_THRESHOLDS: [0.4, 0.5] NMS_THRESH_TEST: 0.5 NUM_CLASSES: 80 NUM_CONVS: 4 PRIOR_PROB: 0.01 SCORE_THRESH_TEST: 0.05 SMOOTH_L1_LOSS_BETA: 0.1 TOPK_CANDIDATES_TEST: 1000 ROI_BOX_CASCADE_HEAD: BBOX_REG_WEIGHTS: ((10.0, 10.0, 5.0, 5.0), (20.0, 20.0, 10.0, 10.0), (30.0, 30.0, 15.0, 15.0)) IOUS: (0.5, 0.6, 0.7) ROI_BOX_HEAD: BBOX_REG_WEIGHTS: (10.0, 10.0, 5.0, 5.0) CLS_AGNOSTIC_BBOX_REG: False CONV_DIM: 256 FC_DIM: 1024 NAME: FastRCNNConvFCHead NORM: NUM_CONV: 0 NUM_FC: 2 POOLER_RESOLUTION: 7 POOLER_SAMPLING_RATIO: 2 POOLER_TYPE: ROIAlign SMOOTH_L1_BETA: 1.0 TRAIN_ON_PRED_BOXES: False ROI_HEADS: BATCH_SIZE_PER_IMAGE: 64 IN_FEATURES: ['p2', 'p3', 'p4', 'p5'] IOU_LABELS: [-1, 0, 1] IOU_THRESHOLDS: [0.2, 0.5] NAME: MeshRCNNROIHeads NMS_THRESH_TEST: 0.5 NUM_CLASSES: 2 POSITIVE_FRACTION: 0.25 PROPOSAL_APPEND_GT: True SCORE_THRESH_TEST: 0.05 ROI_KEYPOINT_HEAD: CONV_DIMS: (512, 512, 512, 512, 512, 512, 512, 512) LOSS_WEIGHT: 1.0 MIN_KEYPOINTS_PER_IMAGE: 1 NAME: KRCNNConvDeconvUpsampleHead NORMALIZE_LOSS_BY_VISIBLE_KEYPOINTS: True NUM_KEYPOINTS: 17 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_TYPE: ROIAlignV2 ROI_MASK_HEAD: CLS_AGNOSTIC_MASK: False CONV_DIM: 256 NAME: MaskRCNNConvUpsampleHead NORM: NUM_CONV: 4 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 2 POOLER_TYPE: ROIAlign ROI_MESH_HEAD: CHAMFER_LOSS_WEIGHT: 1.0 EDGE_LOSS_WEIGHT: 1.0 GRAPH_CONV_DIM: 128 GRAPH_CONV_INIT: normal GT_COORD_THRESH: 5.0 GT_NUM_SAMPLES: 5000 ICO_SPHERE_LEVEL: -1 NAME: MeshRCNNGraphConvHead NORMALS_LOSS_WEIGHT: 0.1 NUM_GRAPH_CONVS: 3 NUM_STAGES: 3 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 2 POOLER_TYPE: ROIAlign PRED_NUM_SAMPLES: 5000 ROI_VOXEL_HEAD: CLS_AGNOSTIC_VOXEL: True CONV_DIM: 256 CUBIFY_THRESH: 0.2 LOSS_WEIGHT: 3.0 NAME: VoxelRCNNConvUpsampleHead NORM: NUM_CONV: 4 NUM_DEPTH: 24 POOLER_RESOLUTION: 12 POOLER_SAMPLING_RATIO: 2 POOLER_TYPE: ROIAlign ROI_Z_HEAD: CLS_AGNOSTIC_Z_REG: False FC_DIM: 1024 NAME: FastRCNNFCHead NUM_FC: 2 POOLER_RESOLUTION: 7 POOLER_SAMPLING_RATIO: 2 POOLER_TYPE: ROIAlign SMOOTH_L1_BETA: 1.0 Z_REG_WEIGHT: 1.0 RPN: BATCH_SIZE_PER_IMAGE: 256 BBOX_REG_WEIGHTS: (1.0, 1.0, 1.0, 1.0) BOUNDARY_THRESH: -1 HEAD_NAME: StandardRPNHead IN_FEATURES: ['p2', 'p3', 'p4', 'p5', 'p6'] IOU_LABELS: [-1, 0, -1, 1] IOU_THRESHOLDS: [0.2, 0.5, 0.7] LOSS_WEIGHT: 1.0 NMS_THRESH: 0.7 POSITIVE_FRACTION: 0.5 POST_NMS_TOPK_TEST: 1000 POST_NMS_TOPK_TRAIN: 1000 PRE_NMS_TOPK_TEST: 1000 PRE_NMS_TOPK_TRAIN: 2000 SMOOTH_L1_BETA: 0.111 SEM_SEG_HEAD: COMMON_STRIDE: 4 CONVS_DIM: 128 IGNORE_VALUE: 255 IN_FEATURES: ['p2', 'p3', 'p4', 'p5'] LOSS_WEIGHT: 1.0 NAME: SemSegFPNHead NORM: GN NUM_CLASSES: 54 VIS_MINIBATCH: False VOXEL_ON: True WEIGHTS: /meshrcnn/R-50.pkl ZPRED_ON: True OUTPUT_DIR: ./output SEED: -1 SOLVER: BASE_LR: 0.0025 BIAS_LR_FACTOR: 1.0 CHECKPOINT_PERIOD: 5000 CLIP_GRADIENTS: CLIP_TYPE: value CLIP_VALUE: 1.0 ENABLED: False NORM_TYPE: 2.0 GAMMA: 0.1 IMS_PER_BATCH: 1 LR_SCHEDULER_NAME: WarmupMultiStepLR MAX_ITER: 88000 MOMENTUM: 0.9 NESTEROV: False STEPS: (64000, 80000) WARMUP_FACTOR: 0.1 WARMUP_ITERS: 1000 WARMUP_METHOD: linear WEIGHT_DECAY: 0.0001 WEIGHT_DECAY_BIAS: 0.0001 WEIGHT_DECAY_NORM: 0.0 TEST: AUG: ENABLED: False FLIP: True MAX_SIZE: 4000 MIN_SIZES: (400, 500, 600, 700, 800, 900, 1000, 1100, 1200) DETECTIONS_PER_IMAGE: 100 EVAL_PERIOD: 0 EXPECTED_RESULTS: [] KEYPOINT_OKS_SIGMAS: [] PRECISE_BN: ENABLED: False NUM_ITER: 200 VERSION: 2 VIS_PERIOD: 0 [05/27 09:32:48 detectron2]: Full config saved to ./output/config.yaml [05/27 09:32:48 d2.utils.env]: Using a generated random seed 48937753

GeneralizedRCNN( (backbone): FPN( (fpn_lateral2): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1)) (fpn_output2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (fpn_lateral3): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1)) (fpn_output3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (fpn_lateral4): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1)) (fpn_output4): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (fpn_lateral5): Conv2d(2048, 256, kernel_size=(1, 1), stride=(1, 1)) (fpn_output5): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (top_block): LastLevelMaxPool() (bottom_up): ResNet( (stem): BasicStem( (conv1): Conv2d( 3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False (norm): FrozenBatchNorm2d(num_features=64, eps=1e-05) ) ) (res2): Sequential( (0): BottleneckBlock( (shortcut): Conv2d( 64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv1): Conv2d( 64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=64, eps=1e-05) ) (conv2): Conv2d( 64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=64, eps=1e-05) ) (conv3): Conv2d( 64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) ) (1): BottleneckBlock( (conv1): Conv2d( 256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=64, eps=1e-05) ) (conv2): Conv2d( 64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=64, eps=1e-05) ) (conv3): Conv2d( 64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) ) (2): BottleneckBlock( (conv1): Conv2d( 256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=64, eps=1e-05) ) (conv2): Conv2d( 64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=64, eps=1e-05) ) (conv3): Conv2d( 64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) ) ) (res3): Sequential( (0): BottleneckBlock( (shortcut): Conv2d( 256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False (norm): FrozenBatchNorm2d(num_features=512, eps=1e-05) ) (conv1): Conv2d( 256, 128, kernel_size=(1, 1), stride=(2, 2), bias=False (norm): FrozenBatchNorm2d(num_features=128, eps=1e-05) ) (conv2): Conv2d( 128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=128, eps=1e-05) ) (conv3): Conv2d( 128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=512, eps=1e-05) ) ) (1): BottleneckBlock( (conv1): Conv2d( 512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=128, eps=1e-05) ) (conv2): Conv2d( 128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=128, eps=1e-05) ) (conv3): Conv2d( 128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=512, eps=1e-05) ) ) (2): BottleneckBlock( (conv1): Conv2d( 512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=128, eps=1e-05) ) (conv2): Conv2d( 128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=128, eps=1e-05) ) (conv3): Conv2d( 128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=512, eps=1e-05) ) ) (3): BottleneckBlock( (conv1): Conv2d( 512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=128, eps=1e-05) ) (conv2): Conv2d( 128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=128, eps=1e-05) ) (conv3): Conv2d( 128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=512, eps=1e-05) ) ) ) (res4): Sequential( (0): BottleneckBlock( (shortcut): Conv2d( 512, 1024, kernel_size=(1, 1), stride=(2, 2), bias=False (norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05) ) (conv1): Conv2d( 512, 256, kernel_size=(1, 1), stride=(2, 2), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv2): Conv2d( 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv3): Conv2d( 256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05) ) ) (1): BottleneckBlock( (conv1): Conv2d( 1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv2): Conv2d( 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv3): Conv2d( 256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05) ) ) (2): BottleneckBlock( (conv1): Conv2d( 1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv2): Conv2d( 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv3): Conv2d( 256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05) ) ) (3): BottleneckBlock( (conv1): Conv2d( 1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv2): Conv2d( 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv3): Conv2d( 256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05) ) ) (4): BottleneckBlock( (conv1): Conv2d( 1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv2): Conv2d( 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv3): Conv2d( 256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05) ) ) (5): BottleneckBlock( (conv1): Conv2d( 1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv2): Conv2d( 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv3): Conv2d( 256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05) ) ) ) (res5): Sequential( (0): BottleneckBlock( (shortcut): Conv2d( 1024, 2048, kernel_size=(1, 1), stride=(2, 2), bias=False (norm): FrozenBatchNorm2d(num_features=2048, eps=1e-05) ) (conv1): Conv2d( 1024, 512, kernel_size=(1, 1), stride=(2, 2), bias=False (norm): FrozenBatchNorm2d(num_features=512, eps=1e-05) ) (conv2): Conv2d( 512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=512, eps=1e-05) ) (conv3): Conv2d( 512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=2048, eps=1e-05) ) ) (1): BottleneckBlock( (conv1): Conv2d( 2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=512, eps=1e-05) ) (conv2): Conv2d( 512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=512, eps=1e-05) ) (conv3): Conv2d( 512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=2048, eps=1e-05) ) ) (2): BottleneckBlock( (conv1): Conv2d( 2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=512, eps=1e-05) ) (conv2): Conv2d( 512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=512, eps=1e-05) ) (conv3): Conv2d( 512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=2048, eps=1e-05) ) ) ) ) ) (proposal_generator): RPN( (anchor_generator): DefaultAnchorGenerator( (cell_anchors): BufferList() ) (rpn_head): StandardRPNHead( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (objectness_logits): Conv2d(256, 3, kernel_size=(1, 1), stride=(1, 1)) (anchor_deltas): Conv2d(256, 12, kernel_size=(1, 1), stride=(1, 1)) ) ) (roi_heads): MeshRCNNROIHeads( (box_pooler): ROIPooler( (level_poolers): ModuleList( (0): ROIAlign(output_size=(7, 7), spatial_scale=0.25, sampling_ratio=2, aligned=False) (1): ROIAlign(output_size=(7, 7), spatial_scale=0.125, sampling_ratio=2, aligned=False) (2): ROIAlign(output_size=(7, 7), spatial_scale=0.0625, sampling_ratio=2, aligned=False) (3): ROIAlign(output_size=(7, 7), spatial_scale=0.03125, sampling_ratio=2, aligned=False) ) ) (box_head): FastRCNNConvFCHead( (fc1): Linear(in_features=12544, out_features=1024, bias=True) (fc2): Linear(in_features=1024, out_features=1024, bias=True) ) (box_predictor): FastRCNNOutputLayers( (cls_score): Linear(in_features=1024, out_features=3, bias=True) (bbox_pred): Linear(in_features=1024, out_features=8, bias=True) ) (mask_pooler): ROIPooler( (level_poolers): ModuleList( (0): ROIAlign(output_size=(14, 14), spatial_scale=0.25, sampling_ratio=2, aligned=False) (1): ROIAlign(output_size=(14, 14), spatial_scale=0.125, sampling_ratio=2, aligned=False) (2): ROIAlign(output_size=(14, 14), spatial_scale=0.0625, sampling_ratio=2, aligned=False) (3): ROIAlign(output_size=(14, 14), spatial_scale=0.03125, sampling_ratio=2, aligned=False) ) ) (mask_head): MaskRCNNConvUpsampleHead( (mask_fcn1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (mask_fcn2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (mask_fcn3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (mask_fcn4): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (deconv): ConvTranspose2d(256, 256, kernel_size=(2, 2), stride=(2, 2)) (predictor): Conv2d(256, 2, kernel_size=(1, 1), stride=(1, 1)) ) (z_pooler): ROIPooler( (level_poolers): ModuleList( (0): ROIAlign(output_size=(7, 7), spatial_scale=0.25, sampling_ratio=2, aligned=False) (1): ROIAlign(output_size=(7, 7), spatial_scale=0.125, sampling_ratio=2, aligned=False) (2): ROIAlign(output_size=(7, 7), spatial_scale=0.0625, sampling_ratio=2, aligned=False) (3): ROIAlign(output_size=(7, 7), spatial_scale=0.03125, sampling_ratio=2, aligned=False) ) ) (z_head): FastRCNNFCHead( (z_fc1): Linear(in_features=12544, out_features=1024, bias=True) (z_fc2): Linear(in_features=1024, out_features=1024, bias=True) (z_pred): Linear(in_features=1024, out_features=2, bias=True) ) (voxel_pooler): ROIPooler( (level_poolers): ModuleList( (0): ROIAlign(output_size=(12, 12), spatial_scale=0.25, sampling_ratio=2, aligned=False) (1): ROIAlign(output_size=(12, 12), spatial_scale=0.125, sampling_ratio=2, aligned=False) (2): ROIAlign(output_size=(12, 12), spatial_scale=0.0625, sampling_ratio=2, aligned=False) (3): ROIAlign(output_size=(12, 12), spatial_scale=0.03125, sampling_ratio=2, aligned=False) ) ) (voxel_head): VoxelRCNNConvUpsampleHead( (voxel_fcn1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (voxel_fcn2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (voxel_fcn3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (voxel_fcn4): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (deconv): ConvTranspose2d(256, 256, kernel_size=(2, 2), stride=(2, 2)) (predictor): Conv2d(256, 24, kernel_size=(1, 1), stride=(1, 1)) ) (mesh_pooler): ROIPooler( (level_poolers): ModuleList( (0): ROIAlign(output_size=(14, 14), spatial_scale=0.25, sampling_ratio=2, aligned=False) (1): ROIAlign(output_size=(14, 14), spatial_scale=0.125, sampling_ratio=2, aligned=False) (2): ROIAlign(output_size=(14, 14), spatial_scale=0.0625, sampling_ratio=2, aligned=False) (3): ROIAlign(output_size=(14, 14), spatial_scale=0.03125, sampling_ratio=2, aligned=False) ) ) (mesh_head): MeshRCNNGraphConvHead( (stages): ModuleList( (0): MeshRefinementStage( (bottleneck): Linear(in_features=256, out_features=128, bias=True) (verts_offset): Linear(in_features=131, out_features=3, bias=True) (gconvs): ModuleList( (0): GraphConv(131 -> 128, directed=False) (1): GraphConv(131 -> 128, directed=False) (2): GraphConv(131 -> 128, directed=False) ) ) (1): MeshRefinementStage( (bottleneck): Linear(in_features=256, out_features=128, bias=True) (verts_offset): Linear(in_features=131, out_features=3, bias=True) (gconvs): ModuleList( (0): GraphConv(259 -> 128, directed=False) (1): GraphConv(131 -> 128, directed=False) (2): GraphConv(131 -> 128, directed=False) ) ) (2): MeshRefinementStage( (bottleneck): Linear(in_features=256, out_features=128, bias=True) (verts_offset): Linear(in_features=131, out_features=3, bias=True) (gconvs): ModuleList( (0): GraphConv(259 -> 128, directed=False) (1): GraphConv(131 -> 128, directed=False) (2): GraphConv(131 -> 128, directed=False) ) ) ) ) ) ) [05/27 09:32:51 d2.data.detection_utils]: TransformGens used in training: [ResizeShortestEdge(short_edge_length=(800,), max_size=1333, sample_style='choice'), RandomFlip()] bed_bookcase_s1_train [05/27 09:32:51 meshrcnn.data.meshrcnn_transforms]: Loading models from bed_bookcase_s1_train... /home/hhd/Documents/paper/pytorch3d-master/pytorch3d/io/obj_io.py:70: UserWarning: Faces have invalid indices warnings.warn("Faces have invalid indices") [05/27 09:32:53 meshrcnn.data.meshrcnn_transforms]: Unique objects loaded: 34 loading annotations into memory... Done (t=0.01s) creating index... index created! thing_classes: ['bed', 'bookcase'] [05/27 09:32:53 meshrcnn.data.datasets.pix3d]: Loaded 1063 images in COCO format from /meshrcnn/datasets/pix3d_hd/bed_bookcase_s1_train.json [05/27 09:32:53 d2.data.build]: Removed 0 images with no usable annotations. 1063 images left. [05/27 09:32:53 d2.data.build]: Distribution of instances among all 2 categories:	category	#instances	category
bed	781	bookcase	282

total	1063

[05/27 09:32:53 d2.data.common]: Serializing 1063 elements to byte tensors and concatenating them all ... [05/27 09:32:53 d2.data.common]: Serialized dataset takes 0.68 MiB [05/27 09:32:53 d2.data.build]: Using training sampler TrainingSampler [05/27 09:32:54 fvcore.common.checkpoint]: Loading checkpoint from /meshrcnn/R-50.pkl [05/27 09:32:55 d2.checkpoint.c2_model_loading]: Remapping C2 weights ...... [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.0.conv1.norm.bias loaded from res2_0_branch2a_bn_beta of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.0.conv1.norm.running_mean loaded from res2_0_branch2a_bn_running_mean of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.0.conv1.norm.running_var loaded from res2_0_branch2a_bn_running_var of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.0.conv1.norm.weight loaded from res2_0_branch2a_bn_gamma of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.0.conv1.weight loaded from res2_0_branch2a_w of shape (64, 64, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.0.conv2.norm.bias loaded from res2_0_branch2b_bn_beta of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.0.conv2.norm.running_mean loaded from res2_0_branch2b_bn_running_mean of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.0.conv2.norm.running_var loaded from res2_0_branch2b_bn_running_var of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.0.conv2.norm.weight loaded from res2_0_branch2b_bn_gamma of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.0.conv2.weight loaded from res2_0_branch2b_w of shape (64, 64, 3, 3) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.0.conv3.norm.bias loaded from res2_0_branch2c_bn_beta of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.0.conv3.norm.running_mean loaded from res2_0_branch2c_bn_running_mean of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.0.conv3.norm.running_var loaded from res2_0_branch2c_bn_running_var of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.0.conv3.norm.weight loaded from res2_0_branch2c_bn_gamma of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.0.conv3.weight loaded from res2_0_branch2c_w of shape (256, 64, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.0.shortcut.norm.bias loaded from res2_0_branch1_bn_beta of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.0.shortcut.norm.running_mean loaded from res2_0_branch1_bn_running_mean of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.0.shortcut.norm.running_var loaded from res2_0_branch1_bn_running_var of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.0.shortcut.norm.weight loaded from res2_0_branch1_bn_gamma of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.0.shortcut.weight loaded from res2_0_branch1_w of shape (256, 64, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.1.conv1.norm.bias loaded from res2_1_branch2a_bn_beta of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.1.conv1.norm.running_mean loaded from res2_1_branch2a_bn_running_mean of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.1.conv1.norm.running_var loaded from res2_1_branch2a_bn_running_var of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.1.conv1.norm.weight loaded from res2_1_branch2a_bn_gamma of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.1.conv1.weight loaded from res2_1_branch2a_w of shape (64, 256, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.1.conv2.norm.bias loaded from res2_1_branch2b_bn_beta of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.1.conv2.norm.running_mean loaded from res2_1_branch2b_bn_running_mean of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.1.conv2.norm.running_var loaded from res2_1_branch2b_bn_running_var of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.1.conv2.norm.weight loaded from res2_1_branch2b_bn_gamma of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.1.conv2.weight loaded from res2_1_branch2b_w of shape (64, 64, 3, 3) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.1.conv3.norm.bias loaded from res2_1_branch2c_bn_beta of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.1.conv3.norm.running_mean loaded from res2_1_branch2c_bn_running_mean of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.1.conv3.norm.running_var loaded from res2_1_branch2c_bn_running_var of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.1.conv3.norm.weight loaded from res2_1_branch2c_bn_gamma of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.1.conv3.weight loaded from res2_1_branch2c_w of shape (256, 64, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.2.conv1.norm.bias loaded from res2_2_branch2a_bn_beta of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.2.conv1.norm.running_mean loaded from res2_2_branch2a_bn_running_mean of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.2.conv1.norm.running_var loaded from res2_2_branch2a_bn_running_var of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.2.conv1.norm.weight loaded from res2_2_branch2a_bn_gamma of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.2.conv1.weight loaded from res2_2_branch2a_w of shape (64, 256, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.2.conv2.norm.bias loaded from res2_2_branch2b_bn_beta of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.2.conv2.norm.running_mean loaded from res2_2_branch2b_bn_running_mean of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.2.conv2.norm.running_var loaded from res2_2_branch2b_bn_running_var of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.2.conv2.norm.weight loaded from res2_2_branch2b_bn_gamma of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.2.conv2.weight loaded from res2_2_branch2b_w of shape (64, 64, 3, 3) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.2.conv3.norm.bias loaded from res2_2_branch2c_bn_beta of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.2.conv3.norm.running_mean loaded from res2_2_branch2c_bn_running_mean of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.2.conv3.norm.running_var loaded from res2_2_branch2c_bn_running_var of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.2.conv3.norm.weight loaded from res2_2_branch2c_bn_gamma of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res2.2.conv3.weight loaded from res2_2_branch2c_w of shape (256, 64, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.0.conv1.norm.bias loaded from res3_0_branch2a_bn_beta of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.0.conv1.norm.running_mean loaded from res3_0_branch2a_bn_running_mean of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.0.conv1.norm.running_var loaded from res3_0_branch2a_bn_running_var of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.0.conv1.norm.weight loaded from res3_0_branch2a_bn_gamma of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.0.conv1.weight loaded from res3_0_branch2a_w of shape (128, 256, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.0.conv2.norm.bias loaded from res3_0_branch2b_bn_beta of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.0.conv2.norm.running_mean loaded from res3_0_branch2b_bn_running_mean of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.0.conv2.norm.running_var loaded from res3_0_branch2b_bn_running_var of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.0.conv2.norm.weight loaded from res3_0_branch2b_bn_gamma of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.0.conv2.weight loaded from res3_0_branch2b_w of shape (128, 128, 3, 3) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.0.conv3.norm.bias loaded from res3_0_branch2c_bn_beta of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.0.conv3.norm.running_mean loaded from res3_0_branch2c_bn_running_mean of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.0.conv3.norm.running_var loaded from res3_0_branch2c_bn_running_var of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.0.conv3.norm.weight loaded from res3_0_branch2c_bn_gamma of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.0.conv3.weight loaded from res3_0_branch2c_w of shape (512, 128, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.0.shortcut.norm.bias loaded from res3_0_branch1_bn_beta of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.0.shortcut.norm.running_mean loaded from res3_0_branch1_bn_running_mean of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.0.shortcut.norm.running_var loaded from res3_0_branch1_bn_running_var of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.0.shortcut.norm.weight loaded from res3_0_branch1_bn_gamma of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.0.shortcut.weight loaded from res3_0_branch1_w of shape (512, 256, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.1.conv1.norm.bias loaded from res3_1_branch2a_bn_beta of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.1.conv1.norm.running_mean loaded from res3_1_branch2a_bn_running_mean of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.1.conv1.norm.running_var loaded from res3_1_branch2a_bn_running_var of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.1.conv1.norm.weight loaded from res3_1_branch2a_bn_gamma of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.1.conv1.weight loaded from res3_1_branch2a_w of shape (128, 512, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.1.conv2.norm.bias loaded from res3_1_branch2b_bn_beta of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.1.conv2.norm.running_mean loaded from res3_1_branch2b_bn_running_mean of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.1.conv2.norm.running_var loaded from res3_1_branch2b_bn_running_var of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.1.conv2.norm.weight loaded from res3_1_branch2b_bn_gamma of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.1.conv2.weight loaded from res3_1_branch2b_w of shape (128, 128, 3, 3) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.1.conv3.norm.bias loaded from res3_1_branch2c_bn_beta of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.1.conv3.norm.running_mean loaded from res3_1_branch2c_bn_running_mean of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.1.conv3.norm.running_var loaded from res3_1_branch2c_bn_running_var of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.1.conv3.norm.weight loaded from res3_1_branch2c_bn_gamma of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.1.conv3.weight loaded from res3_1_branch2c_w of shape (512, 128, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.2.conv1.norm.bias loaded from res3_2_branch2a_bn_beta of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.2.conv1.norm.running_mean loaded from res3_2_branch2a_bn_running_mean of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.2.conv1.norm.running_var loaded from res3_2_branch2a_bn_running_var of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.2.conv1.norm.weight loaded from res3_2_branch2a_bn_gamma of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.2.conv1.weight loaded from res3_2_branch2a_w of shape (128, 512, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.2.conv2.norm.bias loaded from res3_2_branch2b_bn_beta of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.2.conv2.norm.running_mean loaded from res3_2_branch2b_bn_running_mean of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.2.conv2.norm.running_var loaded from res3_2_branch2b_bn_running_var of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.2.conv2.norm.weight loaded from res3_2_branch2b_bn_gamma of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.2.conv2.weight loaded from res3_2_branch2b_w of shape (128, 128, 3, 3) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.2.conv3.norm.bias loaded from res3_2_branch2c_bn_beta of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.2.conv3.norm.running_mean loaded from res3_2_branch2c_bn_running_mean of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.2.conv3.norm.running_var loaded from res3_2_branch2c_bn_running_var of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.2.conv3.norm.weight loaded from res3_2_branch2c_bn_gamma of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.2.conv3.weight loaded from res3_2_branch2c_w of shape (512, 128, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.3.conv1.norm.bias loaded from res3_3_branch2a_bn_beta of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.3.conv1.norm.running_mean loaded from res3_3_branch2a_bn_running_mean of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.3.conv1.norm.running_var loaded from res3_3_branch2a_bn_running_var of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.3.conv1.norm.weight loaded from res3_3_branch2a_bn_gamma of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.3.conv1.weight loaded from res3_3_branch2a_w of shape (128, 512, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.3.conv2.norm.bias loaded from res3_3_branch2b_bn_beta of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.3.conv2.norm.running_mean loaded from res3_3_branch2b_bn_running_mean of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.3.conv2.norm.running_var loaded from res3_3_branch2b_bn_running_var of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.3.conv2.norm.weight loaded from res3_3_branch2b_bn_gamma of shape (128,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.3.conv2.weight loaded from res3_3_branch2b_w of shape (128, 128, 3, 3) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.3.conv3.norm.bias loaded from res3_3_branch2c_bn_beta of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.3.conv3.norm.running_mean loaded from res3_3_branch2c_bn_running_mean of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.3.conv3.norm.running_var loaded from res3_3_branch2c_bn_running_var of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.3.conv3.norm.weight loaded from res3_3_branch2c_bn_gamma of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res3.3.conv3.weight loaded from res3_3_branch2c_w of shape (512, 128, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.0.conv1.norm.bias loaded from res4_0_branch2a_bn_beta of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.0.conv1.norm.running_mean loaded from res4_0_branch2a_bn_running_mean of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.0.conv1.norm.running_var loaded from res4_0_branch2a_bn_running_var of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.0.conv1.norm.weight loaded from res4_0_branch2a_bn_gamma of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.0.conv1.weight loaded from res4_0_branch2a_w of shape (256, 512, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.0.conv2.norm.bias loaded from res4_0_branch2b_bn_beta of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.0.conv2.norm.running_mean loaded from res4_0_branch2b_bn_running_mean of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.0.conv2.norm.running_var loaded from res4_0_branch2b_bn_running_var of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.0.conv2.norm.weight loaded from res4_0_branch2b_bn_gamma of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.0.conv2.weight loaded from res4_0_branch2b_w of shape (256, 256, 3, 3) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.0.conv3.norm.bias loaded from res4_0_branch2c_bn_beta of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.0.conv3.norm.running_mean loaded from res4_0_branch2c_bn_running_mean of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.0.conv3.norm.running_var loaded from res4_0_branch2c_bn_running_var of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.0.conv3.norm.weight loaded from res4_0_branch2c_bn_gamma of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.0.conv3.weight loaded from res4_0_branch2c_w of shape (1024, 256, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.0.shortcut.norm.bias loaded from res4_0_branch1_bn_beta of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.0.shortcut.norm.running_mean loaded from res4_0_branch1_bn_running_mean of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.0.shortcut.norm.running_var loaded from res4_0_branch1_bn_running_var of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.0.shortcut.norm.weight loaded from res4_0_branch1_bn_gamma of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.0.shortcut.weight loaded from res4_0_branch1_w of shape (1024, 512, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.1.conv1.norm.bias loaded from res4_1_branch2a_bn_beta of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.1.conv1.norm.running_mean loaded from res4_1_branch2a_bn_running_mean of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.1.conv1.norm.running_var loaded from res4_1_branch2a_bn_running_var of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.1.conv1.norm.weight loaded from res4_1_branch2a_bn_gamma of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.1.conv1.weight loaded from res4_1_branch2a_w of shape (256, 1024, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.1.conv2.norm.bias loaded from res4_1_branch2b_bn_beta of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.1.conv2.norm.running_mean loaded from res4_1_branch2b_bn_running_mean of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.1.conv2.norm.running_var loaded from res4_1_branch2b_bn_running_var of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.1.conv2.norm.weight loaded from res4_1_branch2b_bn_gamma of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.1.conv2.weight loaded from res4_1_branch2b_w of shape (256, 256, 3, 3) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.1.conv3.norm.bias loaded from res4_1_branch2c_bn_beta of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.1.conv3.norm.running_mean loaded from res4_1_branch2c_bn_running_mean of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.1.conv3.norm.running_var loaded from res4_1_branch2c_bn_running_var of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.1.conv3.norm.weight loaded from res4_1_branch2c_bn_gamma of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.1.conv3.weight loaded from res4_1_branch2c_w of shape (1024, 256, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.2.conv1.norm.bias loaded from res4_2_branch2a_bn_beta of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.2.conv1.norm.running_mean loaded from res4_2_branch2a_bn_running_mean of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.2.conv1.norm.running_var loaded from res4_2_branch2a_bn_running_var of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.2.conv1.norm.weight loaded from res4_2_branch2a_bn_gamma of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.2.conv1.weight loaded from res4_2_branch2a_w of shape (256, 1024, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.2.conv2.norm.bias loaded from res4_2_branch2b_bn_beta of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.2.conv2.norm.running_mean loaded from res4_2_branch2b_bn_running_mean of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.2.conv2.norm.running_var loaded from res4_2_branch2b_bn_running_var of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.2.conv2.norm.weight loaded from res4_2_branch2b_bn_gamma of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.2.conv2.weight loaded from res4_2_branch2b_w of shape (256, 256, 3, 3) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.2.conv3.norm.bias loaded from res4_2_branch2c_bn_beta of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.2.conv3.norm.running_mean loaded from res4_2_branch2c_bn_running_mean of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.2.conv3.norm.running_var loaded from res4_2_branch2c_bn_running_var of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.2.conv3.norm.weight loaded from res4_2_branch2c_bn_gamma of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.2.conv3.weight loaded from res4_2_branch2c_w of shape (1024, 256, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.3.conv1.norm.bias loaded from res4_3_branch2a_bn_beta of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.3.conv1.norm.running_mean loaded from res4_3_branch2a_bn_running_mean of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.3.conv1.norm.running_var loaded from res4_3_branch2a_bn_running_var of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.3.conv1.norm.weight loaded from res4_3_branch2a_bn_gamma of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.3.conv1.weight loaded from res4_3_branch2a_w of shape (256, 1024, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.3.conv2.norm.bias loaded from res4_3_branch2b_bn_beta of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.3.conv2.norm.running_mean loaded from res4_3_branch2b_bn_running_mean of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.3.conv2.norm.running_var loaded from res4_3_branch2b_bn_running_var of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.3.conv2.norm.weight loaded from res4_3_branch2b_bn_gamma of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.3.conv2.weight loaded from res4_3_branch2b_w of shape (256, 256, 3, 3) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.3.conv3.norm.bias loaded from res4_3_branch2c_bn_beta of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.3.conv3.norm.running_mean loaded from res4_3_branch2c_bn_running_mean of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.3.conv3.norm.running_var loaded from res4_3_branch2c_bn_running_var of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.3.conv3.norm.weight loaded from res4_3_branch2c_bn_gamma of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.3.conv3.weight loaded from res4_3_branch2c_w of shape (1024, 256, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.4.conv1.norm.bias loaded from res4_4_branch2a_bn_beta of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.4.conv1.norm.running_mean loaded from res4_4_branch2a_bn_running_mean of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.4.conv1.norm.running_var loaded from res4_4_branch2a_bn_running_var of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.4.conv1.norm.weight loaded from res4_4_branch2a_bn_gamma of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.4.conv1.weight loaded from res4_4_branch2a_w of shape (256, 1024, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.4.conv2.norm.bias loaded from res4_4_branch2b_bn_beta of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.4.conv2.norm.running_mean loaded from res4_4_branch2b_bn_running_mean of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.4.conv2.norm.running_var loaded from res4_4_branch2b_bn_running_var of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.4.conv2.norm.weight loaded from res4_4_branch2b_bn_gamma of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.4.conv2.weight loaded from res4_4_branch2b_w of shape (256, 256, 3, 3) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.4.conv3.norm.bias loaded from res4_4_branch2c_bn_beta of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.4.conv3.norm.running_mean loaded from res4_4_branch2c_bn_running_mean of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.4.conv3.norm.running_var loaded from res4_4_branch2c_bn_running_var of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.4.conv3.norm.weight loaded from res4_4_branch2c_bn_gamma of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.4.conv3.weight loaded from res4_4_branch2c_w of shape (1024, 256, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.5.conv1.norm.bias loaded from res4_5_branch2a_bn_beta of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.5.conv1.norm.running_mean loaded from res4_5_branch2a_bn_running_mean of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.5.conv1.norm.running_var loaded from res4_5_branch2a_bn_running_var of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.5.conv1.norm.weight loaded from res4_5_branch2a_bn_gamma of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.5.conv1.weight loaded from res4_5_branch2a_w of shape (256, 1024, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.5.conv2.norm.bias loaded from res4_5_branch2b_bn_beta of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.5.conv2.norm.running_mean loaded from res4_5_branch2b_bn_running_mean of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.5.conv2.norm.running_var loaded from res4_5_branch2b_bn_running_var of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.5.conv2.norm.weight loaded from res4_5_branch2b_bn_gamma of shape (256,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.5.conv2.weight loaded from res4_5_branch2b_w of shape (256, 256, 3, 3) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.5.conv3.norm.bias loaded from res4_5_branch2c_bn_beta of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.5.conv3.norm.running_mean loaded from res4_5_branch2c_bn_running_mean of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.5.conv3.norm.running_var loaded from res4_5_branch2c_bn_running_var of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.5.conv3.norm.weight loaded from res4_5_branch2c_bn_gamma of shape (1024,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res4.5.conv3.weight loaded from res4_5_branch2c_w of shape (1024, 256, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.0.conv1.norm.bias loaded from res5_0_branch2a_bn_beta of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.0.conv1.norm.running_mean loaded from res5_0_branch2a_bn_running_mean of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.0.conv1.norm.running_var loaded from res5_0_branch2a_bn_running_var of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.0.conv1.norm.weight loaded from res5_0_branch2a_bn_gamma of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.0.conv1.weight loaded from res5_0_branch2a_w of shape (512, 1024, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.0.conv2.norm.bias loaded from res5_0_branch2b_bn_beta of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.0.conv2.norm.running_mean loaded from res5_0_branch2b_bn_running_mean of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.0.conv2.norm.running_var loaded from res5_0_branch2b_bn_running_var of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.0.conv2.norm.weight loaded from res5_0_branch2b_bn_gamma of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.0.conv2.weight loaded from res5_0_branch2b_w of shape (512, 512, 3, 3) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.0.conv3.norm.bias loaded from res5_0_branch2c_bn_beta of shape (2048,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.0.conv3.norm.running_mean loaded from res5_0_branch2c_bn_running_mean of shape (2048,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.0.conv3.norm.running_var loaded from res5_0_branch2c_bn_running_var of shape (2048,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.0.conv3.norm.weight loaded from res5_0_branch2c_bn_gamma of shape (2048,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.0.conv3.weight loaded from res5_0_branch2c_w of shape (2048, 512, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.0.shortcut.norm.bias loaded from res5_0_branch1_bn_beta of shape (2048,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.0.shortcut.norm.running_mean loaded from res5_0_branch1_bn_running_mean of shape (2048,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.0.shortcut.norm.running_var loaded from res5_0_branch1_bn_running_var of shape (2048,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.0.shortcut.norm.weight loaded from res5_0_branch1_bn_gamma of shape (2048,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.0.shortcut.weight loaded from res5_0_branch1_w of shape (2048, 1024, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.1.conv1.norm.bias loaded from res5_1_branch2a_bn_beta of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.1.conv1.norm.running_mean loaded from res5_1_branch2a_bn_running_mean of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.1.conv1.norm.running_var loaded from res5_1_branch2a_bn_running_var of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.1.conv1.norm.weight loaded from res5_1_branch2a_bn_gamma of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.1.conv1.weight loaded from res5_1_branch2a_w of shape (512, 2048, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.1.conv2.norm.bias loaded from res5_1_branch2b_bn_beta of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.1.conv2.norm.running_mean loaded from res5_1_branch2b_bn_running_mean of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.1.conv2.norm.running_var loaded from res5_1_branch2b_bn_running_var of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.1.conv2.norm.weight loaded from res5_1_branch2b_bn_gamma of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.1.conv2.weight loaded from res5_1_branch2b_w of shape (512, 512, 3, 3) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.1.conv3.norm.bias loaded from res5_1_branch2c_bn_beta of shape (2048,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.1.conv3.norm.running_mean loaded from res5_1_branch2c_bn_running_mean of shape (2048,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.1.conv3.norm.running_var loaded from res5_1_branch2c_bn_running_var of shape (2048,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.1.conv3.norm.weight loaded from res5_1_branch2c_bn_gamma of shape (2048,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.1.conv3.weight loaded from res5_1_branch2c_w of shape (2048, 512, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.2.conv1.norm.bias loaded from res5_2_branch2a_bn_beta of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.2.conv1.norm.running_mean loaded from res5_2_branch2a_bn_running_mean of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.2.conv1.norm.running_var loaded from res5_2_branch2a_bn_running_var of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.2.conv1.norm.weight loaded from res5_2_branch2a_bn_gamma of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.2.conv1.weight loaded from res5_2_branch2a_w of shape (512, 2048, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.2.conv2.norm.bias loaded from res5_2_branch2b_bn_beta of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.2.conv2.norm.running_mean loaded from res5_2_branch2b_bn_running_mean of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.2.conv2.norm.running_var loaded from res5_2_branch2b_bn_running_var of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.2.conv2.norm.weight loaded from res5_2_branch2b_bn_gamma of shape (512,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.2.conv2.weight loaded from res5_2_branch2b_w of shape (512, 512, 3, 3) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.2.conv3.norm.bias loaded from res5_2_branch2c_bn_beta of shape (2048,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.2.conv3.norm.running_mean loaded from res5_2_branch2c_bn_running_mean of shape (2048,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.2.conv3.norm.running_var loaded from res5_2_branch2c_bn_running_var of shape (2048,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.2.conv3.norm.weight loaded from res5_2_branch2c_bn_gamma of shape (2048,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.res5.2.conv3.weight loaded from res5_2_branch2c_w of shape (2048, 512, 1, 1) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.stem.conv1.norm.bias loaded from res_conv1_bn_beta of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.stem.conv1.norm.running_mean loaded from res_conv1_bn_running_mean of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.stem.conv1.norm.running_var loaded from res_conv1_bn_running_var of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.stem.conv1.norm.weight loaded from res_conv1_bn_gamma of shape (64,) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: backbone.bottom_up.stem.conv1.weight loaded from conv1_w of shape (64, 3, 7, 7) [05/27 09:32:55 d2.checkpoint.c2_model_loading]: Some model parameters or buffers are not in the checkpoint: backbone.fpn_lateral2.{bias, weight} backbone.fpn_lateral3.{bias, weight} backbone.fpn_lateral4.{bias, weight} backbone.fpn_lateral5.{bias, weight} backbone.fpn_output2.{bias, weight} backbone.fpn_output3.{bias, weight} backbone.fpn_output4.{bias, weight} backbone.fpn_output5.{bias, weight} pixel_mean pixel_std proposal_generator.anchor_generator.cell_anchors.{0, 1, 2, 3, 4} proposal_generator.rpn_head.anchor_deltas.{bias, weight} proposal_generator.rpn_head.conv.{bias, weight} proposal_generator.rpn_head.objectness_logits.{bias, weight} roi_heads.box_head.fc1.{bias, weight} roi_heads.box_head.fc2.{bias, weight} roi_heads.box_predictor.bbox_pred.{bias, weight} roi_heads.box_predictor.cls_score.{bias, weight} roi_heads.mask_head.deconv.{bias, weight} roi_heads.mask_head.mask_fcn1.{bias, weight} roi_heads.mask_head.mask_fcn2.{bias, weight} roi_heads.mask_head.mask_fcn3.{bias, weight} roi_heads.mask_head.mask_fcn4.{bias, weight} roi_heads.mask_head.predictor.{bias, weight} roi_heads.mesh_head.stages.0.bottleneck.{bias, weight} roi_heads.mesh_head.stages.0.gconvs.0.w0.{bias, weight} roi_heads.mesh_head.stages.0.gconvs.0.w1.{bias, weight} roi_heads.mesh_head.stages.0.gconvs.1.w0.{bias, weight} roi_heads.mesh_head.stages.0.gconvs.1.w1.{bias, weight} roi_heads.mesh_head.stages.0.gconvs.2.w0.{bias, weight} roi_heads.mesh_head.stages.0.gconvs.2.w1.{bias, weight} roi_heads.mesh_head.stages.0.verts_offset.{bias, weight} roi_heads.mesh_head.stages.1.bottleneck.{bias, weight} roi_heads.mesh_head.stages.1.gconvs.0.w0.{bias, weight} roi_heads.mesh_head.stages.1.gconvs.0.w1.{bias, weight} roi_heads.mesh_head.stages.1.gconvs.1.w0.{bias, weight} roi_heads.mesh_head.stages.1.gconvs.1.w1.{bias, weight} roi_heads.mesh_head.stages.1.gconvs.2.w0.{bias, weight} roi_heads.mesh_head.stages.1.gconvs.2.w1.{bias, weight} roi_heads.mesh_head.stages.1.verts_offset.{bias, weight} roi_heads.mesh_head.stages.2.bottleneck.{bias, weight} roi_heads.mesh_head.stages.2.gconvs.0.w0.{bias, weight} roi_heads.mesh_head.stages.2.gconvs.0.w1.{bias, weight} roi_heads.mesh_head.stages.2.gconvs.1.w0.{bias, weight} roi_heads.mesh_head.stages.2.gconvs.1.w1.{bias, weight} roi_heads.mesh_head.stages.2.gconvs.2.w0.{bias, weight} roi_heads.mesh_head.stages.2.gconvs.2.w1.{bias, weight} roi_heads.mesh_head.stages.2.verts_offset.{bias, weight} roi_heads.voxel_head.deconv.{bias, weight} roi_heads.voxel_head.predictor.{bias, weight} roi_heads.voxel_head.voxel_fcn1.{bias, weight} roi_heads.voxel_head.voxel_fcn2.{bias, weight} roi_heads.voxel_head.voxel_fcn3.{bias, weight} roi_heads.voxel_head.voxel_fcn4.{bias, weight} roi_heads.z_head.z_fc1.{bias, weight} roi_heads.z_head.z_fc2.{bias, weight} roi_heads.z_head.z_pred.{bias, weight} [05/27 09:32:55 d2.checkpoint.c2_model_loading]: The checkpoint state_dict contains keys that are not used by the model: fc1000_b fc1000_w conv1_b [05/27 09:32:55 d2.engine.train_loop]: Starting training from iteration 0 /python3.7/site-packages/torch/nn/functional.py:2506: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details. "See the documentation of nn.Upsample for details.".format(mode)) [05/27 09:33:07 d2.utils.events]: eta: 8:51:53 iter: 19 total_loss: 6.434 loss_cls: 0.985 loss_box_reg: 0.000 loss_z_reg: 0.025 loss_mask: 0.691 loss_voxel: 2.066 loss_chamfer: 1.425 loss_normals: 0.272 loss_edge: 0.028 loss_rpn_cls: 0.661 loss_rpn_loc: 0.011 time: 0.3622 data_time: 0.0025 lr: 0.000293 max_mem: 1804M [05/27 09:33:14 d2.utils.events]: eta: 8:45:29 iter: 39 total_loss: 5.395 loss_cls: 1.091 loss_box_reg: 0.002 loss_z_reg: 0.041 loss_mask: 0.672 loss_voxel: 2.006 loss_chamfer: 0.879 loss_normals: 0.266 loss_edge: 0.022 loss_rpn_cls: 0.517 loss_rpn_loc: 0.005 time: 0.3551 data_time: 0.0030 lr: 0.000338 max_mem: 1847M [05/27 09:33:21 d2.utils.events]: eta: 8:41:40 iter: 59 total_loss: 4.499 loss_cls: 0.361 loss_box_reg: 0.000 loss_z_reg: 0.031 loss_mask: 0.664 loss_voxel: 1.893 loss_chamfer: 0.706 loss_normals: 0.287 loss_edge: 0.017 loss_rpn_cls: 0.443 loss_rpn_loc: 0.004 time: 0.3552 data_time: 0.0026 lr: 0.000383 max_mem: 1847M [05/27 09:33:28 d2.utils.events]: eta: 8:45:59 iter: 79 total_loss: 4.963 loss_cls: 1.147 loss_box_reg: 0.005 loss_z_reg: 0.069 loss_mask: 0.562 loss_voxel: 1.653 loss_chamfer: 0.490 loss_normals: 0.277 loss_edge: 0.014 loss_rpn_cls: 0.264 loss_rpn_loc: 0.007 time: 0.3564 data_time: 0.0027 lr: 0.000428 max_mem: 1847M [05/27 09:33:36 d2.utils.events]: eta: 8:46:12 iter: 99 total_loss: 4.247 loss_cls: 0.928 loss_box_reg: 0.002 loss_z_reg: 0.125 loss_mask: 0.641 loss_voxel: 1.622 loss_chamfer: 0.423 loss_normals: 0.279 loss_edge: 0.014 loss_rpn_cls: 0.332 loss_rpn_loc: 0.009 time: 0.3570 data_time: 0.0027 lr: 0.000473 max_mem: 1847M [05/27 09:33:43 d2.utils.events]: eta: 8:48:47 iter: 119 total_loss: 3.844 loss_cls: 0.664 loss_box_reg: 0.001 loss_z_reg: 0.042 loss_mask: 0.692 loss_voxel: 1.499 loss_chamfer: 0.379 loss_normals: 0.290 loss_edge: 0.016 loss_rpn_cls: 0.168 loss_rpn_loc: 0.011 time: 0.3586 data_time: 0.0031 lr: 0.000518 max_mem: 1847M [05/27 09:33:51 d2.utils.events]: eta: 8:49:35 iter: 139 total_loss: 3.737 loss_cls: 0.343 loss_box_reg: 0.003 loss_z_reg: 0.037 loss_mask: 0.652 loss_voxel: 1.469 loss_chamfer: 0.377 loss_normals: 0.281 loss_edge: 0.017 loss_rpn_cls: 0.250 loss_rpn_loc: 0.012 time: 0.3595 data_time: 0.0030 lr: 0.000563 max_mem: 1847M [05/27 09:33:58 d2.utils.events]: eta: 8:47:39 iter: 159 total_loss: 3.685 loss_cls: 0.366 loss_box_reg: 0.001 loss_z_reg: 0.051 loss_mask: 0.630 loss_voxel: 1.322 loss_chamfer: 0.336 loss_normals: 0.269 loss_edge: 0.023 loss_rpn_cls: 0.163 loss_rpn_loc: 0.006 time: 0.3582 data_time: 0.0030 lr: 0.000608 max_mem: 1847M [05/27 09:34:05 d2.utils.events]: eta: 8:47:03 iter: 179 total_loss: 3.930 loss_cls: 0.820 loss_box_reg: 0.001 loss_z_reg: 0.073 loss_mask: 0.665 loss_voxel: 1.418 loss_chamfer: 0.390 loss_normals: 0.266 loss_edge: 0.021 loss_rpn_cls: 0.167 loss_rpn_loc: 0.008 time: 0.3576 data_time: 0.0030 lr: 0.000653 max_mem: 1847M [05/27 09:34:13 d2.utils.events]: eta: 8:45:46 iter: 199 total_loss: 3.188 loss_cls: 0.312 loss_box_reg: 0.004 loss_z_reg: 0.036 loss_mask: 0.683 loss_voxel: 1.212 loss_chamfer: 0.412 loss_normals: 0.272 loss_edge: 0.020 loss_rpn_cls: 0.145 loss_rpn_loc: 0.012 time: 0.3573 data_time: 0.0031 lr: 0.000698 max_mem: 1847M [05/27 09:34:20 d2.utils.events]: eta: 8:46:48 iter: 219 total_loss: 3.581 loss_cls: 0.560 loss_box_reg: 0.001 loss_z_reg: 0.031 loss_mask: 0.688 loss_voxel: 1.376 loss_chamfer: 0.358 loss_normals: 0.268 loss_edge: 0.017 loss_rpn_cls: 0.114 loss_rpn_loc: 0.007 time: 0.3578 data_time: 0.0028 lr: 0.000743 max_mem: 1847M [05/27 09:34:27 d2.utils.events]: eta: 8:48:04 iter: 239 total_loss: 3.891 loss_cls: 0.554 loss_box_reg: 0.001 loss_z_reg: 0.076 loss_mask: 0.686 loss_voxel: 1.411 loss_chamfer: 0.391 loss_normals: 0.269 loss_edge: 0.017 loss_rpn_cls: 0.194 loss_rpn_loc: 0.013 time: 0.3582 data_time: 0.0033 lr: 0.000788 max_mem: 1847M [05/27 09:34:35 d2.utils.events]: eta: 8:48:51 iter: 259 total_loss: 3.795 loss_cls: 0.746 loss_box_reg: 0.000 loss_z_reg: 0.033 loss_mask: 0.619 loss_voxel: 1.444 loss_chamfer: 0.384 loss_normals: 0.273 loss_edge: 0.020 loss_rpn_cls: 0.111 loss_rpn_loc: 0.007 time: 0.3583 data_time: 0.0027 lr: 0.000833 max_mem: 1847M [05/27 09:34:42 d2.utils.events]: eta: 8:49:45 iter: 279 total_loss: 3.436 loss_cls: 0.477 loss_box_reg: 0.001 loss_z_reg: 0.025 loss_mask: 0.630 loss_voxel: 1.409 loss_chamfer: 0.299 loss_normals: 0.259 loss_edge: 0.020 loss_rpn_cls: 0.184 loss_rpn_loc: 0.008 time: 0.3593 data_time: 0.0029 lr: 0.000878 max_mem: 1847M [05/27 09:34:50 d2.utils.events]: eta: 8:50:32 iter: 299 total_loss: 3.401 loss_cls: 0.271 loss_box_reg: 0.001 loss_z_reg: 0.021 loss_mask: 0.690 loss_voxel: 1.445 loss_chamfer: 0.308 loss_normals: 0.260 loss_edge: 0.020 loss_rpn_cls: 0.169 loss_rpn_loc: 0.012 time: 0.3602 data_time: 0.0032 lr: 0.000923 max_mem: 1847M [05/27 09:34:57 d2.utils.events]: eta: 8:51:34 iter: 319 total_loss: 3.619 loss_cls: 0.639 loss_box_reg: 0.001 loss_z_reg: 0.036 loss_mask: 0.653 loss_voxel: 1.305 loss_chamfer: 0.389 loss_normals: 0.261 loss_edge: 0.024 loss_rpn_cls: 0.113 loss_rpn_loc: 0.011 time: 0.3608 data_time: 0.0029 lr: 0.000968 max_mem: 1847M [05/27 09:35:05 d2.utils.events]: eta: 8:51:32 iter: 339 total_loss: 3.426 loss_cls: 0.408 loss_box_reg: 0.000 loss_z_reg: 0.035 loss_mask: 0.557 loss_voxel: 1.452 loss_chamfer: 0.365 loss_normals: 0.260 loss_edge: 0.023 loss_rpn_cls: 0.124 loss_rpn_loc: 0.014 time: 0.3614 data_time: 0.0029 lr: 0.001013 max_mem: 1847M [05/27 09:35:12 d2.utils.events]: eta: 8:51:42 iter: 359 total_loss: 3.023 loss_cls: 0.104 loss_box_reg: 0.000 loss_z_reg: 0.027 loss_mask: 0.672 loss_voxel: 1.329 loss_chamfer: 0.295 loss_normals: 0.276 loss_edge: 0.022 loss_rpn_cls: 0.147 loss_rpn_loc: 0.009 time: 0.3620 data_time: 0.0029 lr: 0.001058 max_mem: 1847M [05/27 09:35:20 d2.utils.events]: eta: 8:52:28 iter: 379 total_loss: 3.988 loss_cls: 1.138 loss_box_reg: 0.001 loss_z_reg: 0.028 loss_mask: 0.610 loss_voxel: 1.446 loss_chamfer: 0.364 loss_normals: 0.260 loss_edge: 0.022 loss_rpn_cls: 0.159 loss_rpn_loc: 0.008 time: 0.3627 data_time: 0.0030 lr: 0.001103 max_mem: 1847M [05/27 09:35:28 d2.utils.events]: eta: 8:54:13 iter: 399 total_loss: 4.119 loss_cls: 0.881 loss_box_reg: 0.001 loss_z_reg: 0.023 loss_mask: 0.675 loss_voxel: 1.350 loss_chamfer: 0.516 loss_normals: 0.278 loss_edge: 0.022 loss_rpn_cls: 0.228 loss_rpn_loc: 0.012 time: 0.3649 data_time: 0.0032 lr: 0.001148 max_mem: 2060M [05/27 09:35:36 d2.utils.events]: eta: 8:54:42 iter: 419 total_loss: 3.727 loss_cls: 0.833 loss_box_reg: 0.005 loss_z_reg: 0.016 loss_mask: 0.660 loss_voxel: 1.360 loss_chamfer: 0.434 loss_normals: 0.277 loss_edge: 0.024 loss_rpn_cls: 0.132 loss_rpn_loc: 0.014 time: 0.3656 data_time: 0.0032 lr: 0.001193 max_mem: 2060M [05/27 09:35:44 d2.utils.events]: eta: 8:56:32 iter: 439 total_loss: 3.678 loss_cls: 0.663 loss_box_reg: 0.003 loss_z_reg: 0.034 loss_mask: 0.669 loss_voxel: 1.301 loss_chamfer: 0.388 loss_normals: 0.267 loss_edge: 0.030 loss_rpn_cls: 0.184 loss_rpn_loc: 0.013 time: 0.3672 data_time: 0.0036 lr: 0.001238 max_mem: 2060M [05/27 09:35:52 d2.utils.events]: eta: 8:58:30 iter: 459 total_loss: 3.903 loss_cls: 0.942 loss_box_reg: 0.013 loss_z_reg: 0.092 loss_mask: 0.642 loss_voxel: 1.426 loss_chamfer: 0.403 loss_normals: 0.267 loss_edge: 0.024 loss_rpn_cls: 0.210 loss_rpn_loc: 0.010 time: 0.3685 data_time: 0.0033 lr: 0.001283 max_mem: 2060M [05/27 09:36:00 d2.utils.events]: eta: 9:01:04 iter: 479 total_loss: 4.350 loss_cls: 1.141 loss_box_reg: 0.003 loss_z_reg: 0.040 loss_mask: 0.658 loss_voxel: 1.431 loss_chamfer: 0.448 loss_normals: 0.270 loss_edge: 0.025 loss_rpn_cls: 0.151 loss_rpn_loc: 0.010 time: 0.3693 data_time: 0.0035 lr: 0.001328 max_mem: 2060M [05/27 09:36:08 d2.utils.events]: eta: 9:02:57 iter: 499 total_loss: 4.393 loss_cls: 0.994 loss_box_reg: 0.002 loss_z_reg: 0.230 loss_mask: 0.629 loss_voxel: 1.287 loss_chamfer: 0.409 loss_normals: 0.278 loss_edge: 0.025 loss_rpn_cls: 0.155 loss_rpn_loc: 0.013 time: 0.3699 data_time: 0.0032 lr: 0.001373 max_mem: 2060M [05/27 09:36:16 d2.utils.events]: eta: 9:06:46 iter: 519 total_loss: 4.648 loss_cls: 1.121 loss_box_reg: 0.002 loss_z_reg: 0.080 loss_mask: 0.679 loss_voxel: 1.270 loss_chamfer: 0.443 loss_normals: 0.272 loss_edge: 0.023 loss_rpn_cls: 0.111 loss_rpn_loc: 0.009 time: 0.3713 data_time: 0.0032 lr: 0.001418 max_mem: 2441M [05/27 09:36:24 d2.utils.events]: eta: 9:07:17 iter: 539 total_loss: 4.536 loss_cls: 1.059 loss_box_reg: 0.011 loss_z_reg: 0.059 loss_mask: 0.633 loss_voxel: 1.260 loss_chamfer: 0.482 loss_normals: 0.267 loss_edge: 0.031 loss_rpn_cls: 0.170 loss_rpn_loc: 0.010 time: 0.3722 data_time: 0.0037 lr: 0.001463 max_mem: 2441M [05/27 09:36:32 d2.utils.events]: eta: 9:07:55 iter: 559 total_loss: 3.435 loss_cls: 0.674 loss_box_reg: 0.006 loss_z_reg: 0.052 loss_mask: 0.684 loss_voxel: 1.338 loss_chamfer: 0.487 loss_normals: 0.249 loss_edge: 0.029 loss_rpn_cls: 0.178 loss_rpn_loc: 0.016 time: 0.3737 data_time: 0.0032 lr: 0.001508 max_mem: 2580M [05/27 09:36:40 d2.utils.events]: eta: 9:07:53 iter: 579 total_loss: 3.800 loss_cls: 0.878 loss_box_reg: 0.001 loss_z_reg: 0.120 loss_mask: 0.667 loss_voxel: 1.255 loss_chamfer: 0.383 loss_normals: 0.253 loss_edge: 0.027 loss_rpn_cls: 0.180 loss_rpn_loc: 0.010 time: 0.3738 data_time: 0.0035 lr: 0.001553 max_mem: 2580M [05/27 09:36:47 d2.utils.events]: eta: 9:07:45 iter: 599 total_loss: 3.184 loss_cls: 0.011 loss_box_reg: 0.004 loss_z_reg: 0.058 loss_mask: 0.668 loss_voxel: 1.127 loss_chamfer: 0.310 loss_normals: 0.262 loss_edge: 0.031 loss_rpn_cls: 0.153 loss_rpn_loc: 0.009 time: 0.3738 data_time: 0.0035 lr: 0.001598 max_mem: 2580M [05/27 09:36:55 d2.utils.events]: eta: 9:08:13 iter: 619 total_loss: 4.273 loss_cls: 0.985 loss_box_reg: 0.001 loss_z_reg: 0.042 loss_mask: 0.660 loss_voxel: 1.138 loss_chamfer: 0.309 loss_normals: 0.255 loss_edge: 0.025 loss_rpn_cls: 0.113 loss_rpn_loc: 0.008 time: 0.3742 data_time: 0.0036 lr: 0.001643 max_mem: 2580M [05/27 09:37:04 d2.utils.events]: eta: 9:10:09 iter: 639 total_loss: 4.578 loss_cls: 0.777 loss_box_reg: 0.005 loss_z_reg: 0.104 loss_mask: 0.680 loss_voxel: 1.445 loss_chamfer: 0.689 loss_normals: 0.256 loss_edge: 0.031 loss_rpn_cls: 0.203 loss_rpn_loc: 0.007 time: 0.3760 data_time: 0.0035 lr: 0.001688 max_mem: 2580M [05/27 09:37:11 d2.utils.events]: eta: 9:09:01 iter: 659 total_loss: 3.606 loss_cls: 0.710 loss_box_reg: 0.001 loss_z_reg: 0.024 loss_mask: 0.670 loss_voxel: 1.000 loss_chamfer: 0.402 loss_normals: 0.244 loss_edge: 0.028 loss_rpn_cls: 0.121 loss_rpn_loc: 0.010 time: 0.3758 data_time: 0.0038 lr: 0.001733 max_mem: 2580M [05/27 09:37:20 d2.utils.events]: eta: 9:10:34 iter: 679 total_loss: 3.512 loss_cls: 0.587 loss_box_reg: 0.009 loss_z_reg: 0.016 loss_mask: 0.668 loss_voxel: 1.157 loss_chamfer: 0.407 loss_normals: 0.245 loss_edge: 0.031 loss_rpn_cls: 0.147 loss_rpn_loc: 0.016 time: 0.3764 data_time: 0.0035 lr: 0.001778 max_mem: 2580M [05/27 09:37:28 d2.utils.events]: eta: 9:10:41 iter: 699 total_loss: 3.698 loss_cls: 0.932 loss_box_reg: 0.007 loss_z_reg: 0.029 loss_mask: 0.650 loss_voxel: 1.151 loss_chamfer: 0.413 loss_normals: 0.236 loss_edge: 0.031 loss_rpn_cls: 0.150 loss_rpn_loc: 0.010 time: 0.3771 data_time: 0.0036 lr: 0.001823 max_mem: 2580M [05/27 09:37:36 d2.utils.events]: eta: 9:10:42 iter: 719 total_loss: 3.112 loss_cls: 0.480 loss_box_reg: 0.001 loss_z_reg: 0.012 loss_mask: 0.641 loss_voxel: 1.083 loss_chamfer: 0.370 loss_normals: 0.237 loss_edge: 0.026 loss_rpn_cls: 0.112 loss_rpn_loc: 0.013 time: 0.3775 data_time: 0.0034 lr: 0.001868 max_mem: 2580M [05/27 09:37:44 d2.utils.events]: eta: 9:12:03 iter: 739 total_loss: 4.250 loss_cls: 0.818 loss_box_reg: 0.001 loss_z_reg: 0.034 loss_mask: 0.652 loss_voxel: 1.375 loss_chamfer: 0.423 loss_normals: 0.247 loss_edge: 0.026 loss_rpn_cls: 0.181 loss_rpn_loc: 0.013 time: 0.3790 data_time: 0.0031 lr: 0.001913 max_mem: 2872M [05/27 09:37:52 d2.utils.events]: eta: 9:11:40 iter: 759 total_loss: 3.782 loss_cls: 0.839 loss_box_reg: 0.001 loss_z_reg: 0.038 loss_mask: 0.687 loss_voxel: 1.133 loss_chamfer: 0.523 loss_normals: 0.251 loss_edge: 0.023 loss_rpn_cls: 0.201 loss_rpn_loc: 0.008 time: 0.3790 data_time: 0.0030 lr: 0.001958 max_mem: 2872M [05/27 09:38:00 d2.utils.events]: eta: 9:12:18 iter: 779 total_loss: 3.657 loss_cls: 0.734 loss_box_reg: 0.000 loss_z_reg: 0.028 loss_mask: 0.694 loss_voxel: 1.103 loss_chamfer: 0.543 loss_normals: 0.257 loss_edge: 0.025 loss_rpn_cls: 0.211 loss_rpn_loc: 0.021 time: 0.3794 data_time: 0.0033 lr: 0.002003 max_mem: 2872M [05/27 09:38:08 d2.utils.events]: eta: 9:13:09 iter: 799 total_loss: 3.595 loss_cls: 0.661 loss_box_reg: 0.000 loss_z_reg: 0.037 loss_mask: 0.630 loss_voxel: 1.137 loss_chamfer: 0.413 loss_normals: 0.253 loss_edge: 0.026 loss_rpn_cls: 0.160 loss_rpn_loc: 0.013 time: 0.3804 data_time: 0.0034 lr: 0.002048 max_mem: 2872M [05/27 09:38:17 d2.utils.events]: eta: 9:14:38 iter: 819 total_loss: 3.308 loss_cls: 0.814 loss_box_reg: 0.000 loss_z_reg: 0.013 loss_mask: 0.664 loss_voxel: 1.059 loss_chamfer: 0.360 loss_normals: 0.244 loss_edge: 0.026 loss_rpn_cls: 0.104 loss_rpn_loc: 0.006 time: 0.3810 data_time: 0.0035 lr: 0.002093 max_mem: 2872M [05/27 09:38:25 d2.utils.events]: eta: 9:14:53 iter: 839 total_loss: 3.427 loss_cls: 0.770 loss_box_reg: 0.000 loss_z_reg: 0.017 loss_mask: 0.630 loss_voxel: 1.149 loss_chamfer: 0.326 loss_normals: 0.242 loss_edge: 0.028 loss_rpn_cls: 0.108 loss_rpn_loc: 0.007 time: 0.3814 data_time: 0.0034 lr: 0.002138 max_mem: 2872M [05/27 09:38:33 d2.utils.events]: eta: 9:15:49 iter: 859 total_loss: 3.110 loss_cls: 0.269 loss_box_reg: 0.001 loss_z_reg: 0.035 loss_mask: 0.613 loss_voxel: 1.031 loss_chamfer: 0.250 loss_normals: 0.247 loss_edge: 0.029 loss_rpn_cls: 0.230 loss_rpn_loc: 0.020 time: 0.3825 data_time: 0.0032 lr: 0.002183 max_mem: 2872M [05/27 09:38:42 d2.utils.events]: eta: 9:16:40 iter: 879 total_loss: 3.691 loss_cls: 0.799 loss_box_reg: 0.000 loss_z_reg: 0.012 loss_mask: 0.652 loss_voxel: 1.213 loss_chamfer: 0.532 loss_normals: 0.254 loss_edge: 0.027 loss_rpn_cls: 0.152 loss_rpn_loc: 0.009 time: 0.3832 data_time: 0.0034 lr: 0.002228 max_mem: 2872M [05/27 09:38:50 d2.utils.events]: eta: 9:16:46 iter: 899 total_loss: 3.549 loss_cls: 1.052 loss_box_reg: 0.001 loss_z_reg: 0.008 loss_mask: 0.657 loss_voxel: 1.023 loss_chamfer: 0.428 loss_normals: 0.244 loss_edge: 0.028 loss_rpn_cls: 0.124 loss_rpn_loc: 0.013 time: 0.3836 data_time: 0.0030 lr: 0.002273 max_mem: 2872M [05/27 09:38:58 d2.utils.events]: eta: 9:16:39 iter: 919 total_loss: 3.120 loss_cls: 0.490 loss_box_reg: 0.000 loss_z_reg: 0.038 loss_mask: 0.598 loss_voxel: 1.139 loss_chamfer: 0.336 loss_normals: 0.250 loss_edge: 0.028 loss_rpn_cls: 0.099 loss_rpn_loc: 0.007 time: 0.3835 data_time: 0.0036 lr: 0.002318 max_mem: 2872M [05/27 09:39:06 d2.utils.events]: eta: 9:17:14 iter: 939 total_loss: 3.500 loss_cls: 0.749 loss_box_reg: 0.001 loss_z_reg: 0.026 loss_mask: 0.651 loss_voxel: 1.169 loss_chamfer: 0.371 loss_normals: 0.248 loss_edge: 0.025 loss_rpn_cls: 0.111 loss_rpn_loc: 0.009 time: 0.3842 data_time: 0.0033 lr: 0.002363 max_mem: 2872M [05/27 09:39:15 d2.utils.events]: eta: 9:18:22 iter: 959 total_loss: 3.912 loss_cls: 0.771 loss_box_reg: 0.001 loss_z_reg: 0.091 loss_mask: 0.630 loss_voxel: 1.263 loss_chamfer: 0.414 loss_normals: 0.261 loss_edge: 0.022 loss_rpn_cls: 0.264 loss_rpn_loc: 0.021 time: 0.3853 data_time: 0.0035 lr: 0.002408 max_mem: 2906M [05/27 09:39:24 d2.utils.events]: eta: 9:18:35 iter: 979 total_loss: 3.480 loss_cls: 0.465 loss_box_reg: 0.063 loss_z_reg: 0.047 loss_mask: 0.668 loss_voxel: 1.266 loss_chamfer: 0.454 loss_normals: 0.249 loss_edge: 0.029 loss_rpn_cls: 0.135 loss_rpn_loc: 0.004 time: 0.3868 data_time: 0.0036 lr: 0.002453 max_mem: 2910M [05/27 09:39:33 d2.utils.events]: eta: 9:19:31 iter: 999 total_loss: 4.672 loss_cls: 0.464 loss_box_reg: 0.008 loss_z_reg: 0.265 loss_mask: 0.660 loss_voxel: 1.259 loss_chamfer: 0.359 loss_normals: 0.237 loss_edge: 0.028 loss_rpn_cls: 0.242 loss_rpn_loc: 0.010 time: 0.3879 data_time: 0.0036 lr: 0.002498 max_mem: 2910M Meshes contain nan or inf. Meshes contain nan or inf. Meshes contain nan or inf. Traceback (most recent call last): File "/meshrcnn/tools/train_net.py", line 113, in main(args) File "/meshrcnn/tools/train_net.py", line 100, in main return trainer.train() File "/detectron2-master/detectron2/engine/defaults.py", line 401, in train super().train(self.start_iter, self.max_iter) File "/detectron2-master/detectron2/engine/train_loop.py", line 132, in train self.run_step() File "/detectron2-master/detectron2/engine/train_loop.py", line 217, in run_step self._detect_anomaly(losses, loss_dict) File "/detectron2-master/detectron2/engine/train_loop.py", line 240, in _detect_anomaly self.iter, loss_dict FloatingPointError: Loss became infinite or NaN at iteration=1002! loss_dict = {'loss_cls': tensor(nan, device='cuda:0', grad_fn=), 'loss_box_reg': tensor(nan, device='cuda:0', grad_fn=), 'loss_z_reg': tensor(nan, device='cuda:0', grad_fn=), 'loss_mask': tensor(1.0436e+10, device='cuda:0', grad_fn=), 'loss_voxel': tensor(1.4543e+09, device='cuda:0', grad_fn=), 'loss_chamfer': 0, 'loss_normals': 0, 'loss_edge': 0, 'loss_rpn_cls': tensor(0.6048, device='cuda:0', grad_fn=), 'loss_rpn_loc': tensor(0.0055, device='cuda:0', grad_fn=)} ERROR [05/27 09:39:34 d2.engine.train_loop]: Exception during training: Traceback (most recent call last): File "/detectron2-master/detectron2/engine/train_loop.py", line 132, in train self.run_step() File "/detectron2-master/detectron2/engine/train_loop.py", line 217, in run_step self._detect_anomaly(losses, loss_dict) File "/detectron2-master/detectron2/engine/train_loop.py", line 240, in _detect_anomaly self.iter, loss_dict FloatingPointError: Loss became infinite or NaN at iteration=1002!

loss_dict = {'loss_cls': tensor(nan, device='cuda:0', grad_fn=), 'loss_box_reg': tensor(nan, device='cuda:0', grad_fn=), 'loss_z_reg': tensor(nan, device='cuda:0', grad_fn=), 'loss_mask': tensor(1.0436e+10, device='cuda:0', grad_fn=), 'loss_voxel': tensor(1.4543e+09, device='cuda:0', grad_fn=), 'loss_chamfer': 0, 'loss_normals': 0, 'loss_edge': 0, 'loss_rpn_cls': tensor(0.6048, device='cuda:0', grad_fn=), 'loss_rpn_loc': tensor(0.0055, device='cuda:0', grad_fn=)} [05/27 09:39:34 d2.engine.hooks]: Overall training speed: 1000 iterations in 0:06:28 (0.3883 s / it) [05/27 09:39:34 d2.engine.hooks]: Total training time: 0:06:33 (0:00:05 on hooks)

Hi, thank you for sharing your work！ My computer takes a lot of time loading unique model. So i just choose bed and bookcase to train. I adjust the json file and config file. But when the train is in 1000 iterations, loss became infinite or NaN. Do you konw what's wrong with it? Or can you guide me how to solve this bug?

thanks, czsc

gkioxari commented 4 years ago

Hi @czsc123 I have never tried to train with a subset of categories. The schedule provided is for the whole pix3D dataset for 8 GPU training. Once you deviate from that, many things can go wrong in optimization. I suggest reducing the learning rate to start with. You might find the discussion on https://github.com/facebookresearch/meshrcnn/issues/16 useful.

I am a little concerned when you say your computer takes a long time to load unique models. Why is that? If your machine uses a local disk, loading the models should take very little time. Maybe try to investigate that as well, as I don't see a reason why that should be the case and that will also solve your issue when training with 2 categories.

czsc123 commented 4 years ago

Thank you for your quick reply. It happens when "Loading models from pix3d_s1_train...". It takes about 5~10 minutes to load the whole datasets' mesh models. Maybe I should be more patient and try to train the whlole datasets again. And I will follow the discuss on #16 to see if it works.

albertotono commented 4 years ago

Hi @czsc123, may I ask you what did you change for training only 2 subcategories?

I am trying to do the same task, but I am receiving this

  File "/home/alberto/anaconda3/envs/meshrcnn/lib/python3.8/site-packages/detectron2/data/catalog.py", line 147, in __setattr__
    assert oldval == val, (
AssertionError: Attribute 'thing_classes' in the metadata of 'pix3d_s1_train' cannot be set to a different value!
['bed', 'bookcase', 'chair', 'desk', 'misc', 'sofa', 'table', 'tool', 'wardrobe'] != ['chair', 'desk']

This is what I did. I modified the meshrcnn_R50_FPN.yaml

I changed these

BATCH_SIZE_PER_IMAGE: 6 # reduced for 1 GPU
NUM_CLASSES: 2 # Number of foreground classes
STEPS: (2000, 5000)
BASE_LR: 0.0001
MAX_ITER: 1000
WARMUP_ITERS: 100

I am just trying to run the training avoiding to go out of memory.

Then I modified the .json_s1_train.json deleting the extra categories only from the .json file {"categories": [{"id": 3, "name": "chair"}, {"id": 4, "name": "desk"}] then also in the json_s1_test.json same operation.

But I got this error:


AssertionError: Attribute 'thing_classes' in the metadata of 'pix3d_s1_train' cannot be set to a different value!
['bed', 'bookcase', 'chair', 'desk', 'misc', 'sofa', 'table', 'tool', 'wardrobe'] != ['chair', 'desk']

Any suggestions? I don't know where to find thing_classes I saw there is one in meshrcnn/meshrcnn/data/pix3d.py but I think it references the one in detectron2, as far as I am reading online. https://github.com/facebookresearch/detectron2/issues/174 https://github.com/facebookresearch/detectron2/issues/1139

so I also went in this folder /home/alberto/anaconda3/envs/meshrcnn/lib/python3.8/site-packages/detectron2 to reach catalog.py

Maybe I should register this new 2 categories train in Detecron2 following the procedure.

Any ideas or help regarding how did you passed this error?

ShashwatNigam99 commented 4 years ago

You can find thing classes here : Register pix3d metadata You will have to make appropriate changes to your training JSON files as well. Remove the unwanted categories.

Hope this helps

albertotono commented 4 years ago

Thanks @ShashwatNigam99 I removed the extra categories like this

def get_pix3d_metadata():
    meta = [
        {"name": "chair", "color": [250, 190, 190], "id": 3},  # noqa
        {"name": "desk", "color": [60, 180, 75], "id": 4},  # noqa
    ]
    return meta

and removed also the other on the .json Screenshot from 2020-10-26 19-40-18

I removed only the categories not all the pictures only in the training Screenshot from 2020-10-26 19-42-00

I got this

  File "/home/alberto/Documents/MeshRCNN/meshrcnn/meshrcnn/data/datasets/builtin.py", line 37, in <lambda>
    dataset_name, lambda: load_pix3d_json(json_file, image_root, dataset_name)
  File "/home/alberto/Documents/MeshRCNN/meshrcnn/meshrcnn/data/datasets/pix3d.py", line 143, in load_pix3d_json
    obj["category_id"] = id_map[obj["category_id"]]
KeyError: 7

albertotono commented 4 years ago

I solved that, cleaning the folder also from the extra model, mask and img but now I got this


ne 16, in _open_file
    f = open(f, mode)
FileNotFoundError: [Errno 2] No such file or directory: 'datasets/pix3d/model/table/IKEA_LIATORP_1/model.obj'

albertotono commented 4 years ago

I think that maybe I should change the settings here

https://github.com/facebookresearch/meshrcnn/blob/908ac99e90174dff4d2c7e4032500fed86439b2a/meshrcnn/data/datasets/pix3d.py#L19

ShashwatNigam99 commented 4 years ago

Did you edit your train and test JSON files?

albertotono commented 4 years ago

Yes I did Screenshot from 2020-10-27 06-16-42 Thanks for the help @ShashwatNigam99 much appreciated.

ShashwatNigam99 commented 4 years ago

Screenshot from 2020-10-27 18-50-59 This is from the JSON file you sent. Even though you have removed all other categories apart from 3 and 4 under "categories", you haven't removed from other classes from "annotations". If you see category id 7 and 8 still exist in the screenshot above. This is the cause of your error. You need to edit your train and test JSON files under annotations as well :(

albertotono commented 4 years ago

Thanks @ShashwatNigam99, I went ahead and remove also annotation and images. these are the new .json with only Chairs and Tables

pix3d_s1_t.zip

I got this error.

  File "/home/alberto/Documents/MeshRCNN/meshrcnn/meshrcnn/modeling/roi_heads/roi_heads.py", line 290, in _forward_shape
    raise ValueError("No support for class specific predictions")
ValueError: No support for class specific predictions

I feel now I hit this point https://github.com/facebookresearch/meshrcnn/issues/74 Also in the config there is this flag Screenshot from 2020-10-27 07-43-02

After I changed it to True I received another Runtime Error

RuntimeError: CUDA out of memory. Tried to allocate 56.00 MiB (GPU 0; 7.80 GiB total capacity; 6.06 GiB already allocated; 48.56 MiB free; 6.37 GiB reserved in total by PyTorch)

I think it is impossible to do with 1 gpu

albertotono commented 4 years ago

It is "training" now, I modified other settings.

loss_voxel: 2.062  loss_chamfer: 1.673  loss_normals: 0.286  loss_edge: 0.029  loss_rpn_cls: 0.276  loss_rpn_loc: 0.005  time: 0.6497  data_time: 0.0222  lr: 0.000271  max_mem: 4430M
[10/27 07:53:34 d2.utils.events]:  eta: 0:09:51  iter: 39  total_loss: 5.159  loss_cls: 0.599  loss_box_reg: 0.468  loss_mask: 0.560  loss_voxel: 1.941  loss_chamfer: 1.189  loss_normals: 0.287  loss_edge: 0.022  loss_rpn_cls: 0.111  loss_rpn_loc: 0.008  time: 0.6237  data_time: 0.0035  lr: 0.000451  max_mem: 4430M
[10/27 07:53:46 d2.utils.events]:  eta: 0:09:30  iter: 59  total_loss: 4.189  loss_cls: 0.504  loss_box_reg: 0.459  loss_mask: 0.386  loss_voxel: 1.581  loss_chamfer: 0.838  loss_normals: 0.305  loss_edge: 0.016  loss_rpn_cls: 0.099  loss_rpn_loc: 0.005  time: 0.6214  data_time: 0.0035  lr: 0.000631  max_mem: 4820M
[10/27 07:53:59 d2.utils.events]:  eta: 0:09:24  iter: 79  total_loss: 3.406  loss_cls: 0.454  loss_box_reg: 0.273  loss_mask: 0.385  loss_voxel: 0.940  loss_chamfer: 0.848  loss_normals: 0.301  loss_edge: 0.014  loss_rpn_cls: 0.062  loss_rpn_loc: 0.004  time: 0.6217  data_time: 0.0036  lr: 0.000811  max_mem: 4820M

but after it is still trying to access other classes FileNotFoundError: [Errno 2] No such file or directory: 'datasets/pix3d/model/sofa/IKEA_EKTORP_2/model.obj' maybe I should modify the main pix3d.json file.

KnightOfTheMoonlight commented 3 years ago

quote: "I am just trying to run the training avoiding going out of memory." hi, @albertotono I have encountered the same issue. Even I set BATCH_SIZE_PER_IMAGE:1, NUM_GPU:1. You tried to train one subcategory to reduce the memory cost. Did you assume the whole dataset is trained directly, there is no batch_size setup in the project, so reducing the number of categories will reduce the memory cost. It's still bothering me, there is no "batch_size" parameter. For the training of shapenet, in the "yaml" file, there is the SOLVER: BATCH_SIZE: 64 parameter to set.

And @gkioxari, Can you clarify this confusion?

KnightOfTheMoonlight commented 3 years ago

Follow up on my above question, I found "SOLVER: IMS_PER_BATCH: 2" helps with my problem.

facebookresearch / meshrcnn

FloatingPointError: Loss became infinite or NaN at iteration=1002! #29