There is a gap between the self test results and the official in coco dataset

[wjm202]

环境配置 2022-09-22 07:46:31.451	INFO	cvpods.engine.setup:default_setup:137 - Rank of current process: 0. World size: 8 2022-09-22 07:46:33.296	INFO	cvpods.engine.setup:default_setup:139 - Environment info:

sys.platform linux Python 3.8.5 (default, Sep 4 2020, 07:30:14) [GCC 7.3.0] numpy 1.19.2 cvpods 0.1 @/paddle/SSOD/DenseTeacher/cvpods/cvpods cvpods compiler GCC 8.2 cvpods CUDA compiler 10.1 cvpods arch flags sm_61 cvpods_ENV_MODULE PyTorch 1.7.1+cu101 @/root/anaconda3/lib/python3.8/site-packages/torch PyTorch debug build False CUDA available True GPU 0,1,2,3,4,5,6,7 Tesla P40 CUDA_HOME /usr/local/cuda NVCC Cuda compilation tools, release 10.1, V10.1.243 Pillow 8.0.1 torchvision 0.8.2+cu101 @/root/anaconda3/lib/python3.8/site-packages/torchvision torchvision arch flags sm_35, sm_50, sm_60, sm_70, sm_75 cv2 4.6.0

---

PyTorch built with:

• GCC 7.3
• C++ Version: 201402
• Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v1.6.0 (Git Hash 5ef631a030a6f73131c77892041042805a06064f)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• NNPACK is enabled
• CPU capability usage: AVX2
• CUDA Runtime 10.1
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75
• CuDNN 7.6.3
• Magma 2.5.2
• Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

config，没有改动： `from cvpods.configs.fcos_config import FCOSConfig

from augmentations import WeakAug,StrongAug from dataset import PartialCOCO

_config_dict = dict( DATASETS=dict( SUPERVISED=[ (PartialCOCO,dict( percentage=10, seed=1, supervised=True, sup_file='../COCO_Division/COCO_supervision.txt' )), ], UNSUPERVISED=[ (PartialCOCO,dict( percentage=10, seed=1, supervised=False, sup_file='../COCO_Division/COCO_supervision.txt' )), ], TEST=("coco_2017_val",), ), MODEL=dict( WEIGHTS="/paddle/SSOD/cvpods/cvpods/R-50.pkl", RESNETS=dict(DEPTH=50), FCOS=dict( QUALITY_BRANCH='iou', CENTERNESS_ON_REG=True, NORM_REG_TARGETS=True, NMS_THRESH_TEST=0.6, BBOX_REG_WEIGHTS=(1.0, 1.0, 1.0, 1.0), FOCAL_LOSS_GAMMA=2.0, FOCAL_LOSS_ALPHA=0.25, IOU_LOSS_TYPE="giou", CENTER_SAMPLING_RADIUS=1.5, OBJECT_SIZES_OF_INTEREST=[ [-1, 64], [64, 128], [128, 256], [256, 512], [512, float("inf")], ], NORM_SYNC=True, ), ), DATALOADER=dict( NUM_WORKERS=4, ), SOLVER=dict( LR_SCHEDULER=dict( MAX_ITER=180000, STEPS=(179995, ), WARMUP_ITERS=1000, WARMUP_FACTOR=1.0 / 1000, GAMMA=0.1, ), OPTIMIZER=dict( BASE_LR=0.01, ), IMS_PER_BATCH=16, CHECKPOINT_PERIOD=5000, CLIP_GRADIENTS=dict(ENABLED=True) ), TRAINER=dict( NAME="SemiRunner", EMA=dict( DECAY_FACTOR=0.9996, UPDATE_STEPS=1, START_STEPS=3000, FAKE=False ), SSL=dict( BURN_IN_STEPS=5000, ), DISTILL=dict( RATIO=0.01, SUP_WEIGHT=1, UNSUP_WEIGHT=1, SUPPRESS='linear', WEIGHTS=dict( LOGITS=4., DELTAS=1., QUALITY=1.,
), GAMMA=2. ),

WINDOW_SIZE=1,

),
TEST=dict(
    EVAL_PERIOD=2000,
),
INPUT=dict(
    AUG=dict(
        TRAIN_PIPELINES=dict(
            SUPERVISED=(WeakAug,dict(short_edge_length=(640, 672, 704, 736, 768, 800), max_size=1333, sample_style="choice")),
            UNSUPERVISED=(StrongAug,)
        ),
        TEST_PIPELINES=[
            ("ResizeShortestEdge",
             dict(short_edge_length=800, max_size=1333, sample_style="choice")),
        ],
    )
),
OUTPUT_DIR='outputs',
GLOBAL=dict(
    LOG_INTERVAL=10,
)

)

class CustomFCOSConfig(FCOSConfig): def init(self): super(CustomFCOSConfig, self).init() self._register_configuration(_config_dict)

config = CustomFCOSConfig()` 您好我使用您代码原本的config，环境配置 2022-09-22 07:46:31.451	INFO	cvpods.engine.setup:default_setup:137 - Rank of current process: 0. World size: 8 2022-09-22 07:46:33.296	INFO	cvpods.engine.setup:default_setup:139 - Environment info:

sys.platform linux Python 3.8.5 (default, Sep 4 2020, 07:30:14) [GCC 7.3.0] numpy 1.19.2 cvpods 0.1 @/paddle/SSOD/DenseTeacher/cvpods/cvpods cvpods compiler GCC 8.2 cvpods CUDA compiler 10.1 cvpods arch flags sm_61 cvpods_ENV_MODULE PyTorch 1.7.1+cu101 @/root/anaconda3/lib/python3.8/site-packages/torch PyTorch debug build False CUDA available True GPU 0,1,2,3,4,5,6,7 Tesla P40 CUDA_HOME /usr/local/cuda NVCC Cuda compilation tools, release 10.1, V10.1.243 Pillow 8.0.1 torchvision 0.8.2+cu101 @/root/anaconda3/lib/python3.8/site-packages/torchvision torchvision arch flags sm_35, sm_50, sm_60, sm_70, sm_75 cv2 4.6.0

---

PyTorch built with:

• GCC 7.3
• C++ Version: 201402
• Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v1.6.0 (Git Hash 5ef631a030a6f73131c77892041042805a06064f)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• NNPACK is enabled
• CPU capability usage: AVX2
• CUDA Runtime 10.1
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75
• CuDNN 7.6.3
• Magma 2.5.2
• Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

config，没有改动： `from cvpods.configs.fcos_config import FCOSConfig

from augmentations import WeakAug,StrongAug from dataset import PartialCOCO

_config_dict = dict( DATASETS=dict( SUPERVISED=[ (PartialCOCO,dict( percentage=10, seed=1, supervised=True, sup_file='../COCO_Division/COCO_supervision.txt' )), ], UNSUPERVISED=[ (PartialCOCO,dict( percentage=10, seed=1, supervised=False, sup_file='../COCO_Division/COCO_supervision.txt' )), ], TEST=("coco_2017_val",), ), MODEL=dict( WEIGHTS="/paddle/SSOD/cvpods/cvpods/R-50.pkl", RESNETS=dict(DEPTH=50), FCOS=dict( QUALITY_BRANCH='iou', CENTERNESS_ON_REG=True, NORM_REG_TARGETS=True, NMS_THRESH_TEST=0.6, BBOX_REG_WEIGHTS=(1.0, 1.0, 1.0, 1.0), FOCAL_LOSS_GAMMA=2.0, FOCAL_LOSS_ALPHA=0.25, IOU_LOSS_TYPE="giou", CENTER_SAMPLING_RADIUS=1.5, OBJECT_SIZES_OF_INTEREST=[ [-1, 64], [64, 128], [128, 256], [256, 512], [512, float("inf")], ], NORM_SYNC=True, ), ), DATALOADER=dict( NUM_WORKERS=4, ), SOLVER=dict( LR_SCHEDULER=dict( MAX_ITER=180000, STEPS=(179995, ), WARMUP_ITERS=1000, WARMUP_FACTOR=1.0 / 1000, GAMMA=0.1, ), OPTIMIZER=dict( BASE_LR=0.01, ), IMS_PER_BATCH=16, CHECKPOINT_PERIOD=5000, CLIP_GRADIENTS=dict(ENABLED=True) ), TRAINER=dict( NAME="SemiRunner", EMA=dict( DECAY_FACTOR=0.9996, UPDATE_STEPS=1, START_STEPS=3000, FAKE=False ), SSL=dict( BURN_IN_STEPS=5000, ), DISTILL=dict( RATIO=0.01, SUP_WEIGHT=1, UNSUP_WEIGHT=1, SUPPRESS='linear', WEIGHTS=dict( LOGITS=4., DELTAS=1., QUALITY=1.,
), GAMMA=2. ),

WINDOW_SIZE=1,

),
TEST=dict(
    EVAL_PERIOD=2000,
),
INPUT=dict(
    AUG=dict(
        TRAIN_PIPELINES=dict(
            SUPERVISED=(WeakAug,dict(short_edge_length=(640, 672, 704, 736, 768, 800), max_size=1333, sample_style="choice")),
            UNSUPERVISED=(StrongAug,)
        ),
        TEST_PIPELINES=[
            ("ResizeShortestEdge",
             dict(short_edge_length=800, max_size=1333, sample_style="choice")),
        ],
    )
),
OUTPUT_DIR='outputs',
GLOBAL=dict(
    LOG_INTERVAL=10,
)

)

class CustomFCOSConfig(FCOSConfig): def init(self): super(CustomFCOSConfig, self).init() self._register_configuration(_config_dict)

config = CustomFCOSConfig()` 您好，我在使用您提供的代码训练10%coco数据集，最好结果只有mAP=32.39，对config和代码没有做任何更改，是我的环境配置存在问题吗，使用8张tesla P40训练。除此之外在1%coco上的自测结果也不尽人意，只有mAP=16.2

[ZRandomize]

您好，请测试teacher模型的性能（权重为checkpoint['ema']），而不是student模型(权重为checkpoint['model'])，在DenseTeacher中teacher和student的gap会一直存在，且我们的35.1 mAP 模型，其student性能为32.8%，与您提供的性能接近。 1%的情况应当相同，同时请查看代码是否最新

[wjm202]

好的谢谢

[wjm202]

您好，请测试teacher模型的性能（权重为checkpoint['ema']），而不是student模型(权重为checkpoint['model'])，在DenseTeacher中teacher和student的gap会一直存在，且我们的35.1 mAP 模型，其student性能为32.8%，与您提供的性能接近。 1%的情况应当相同，同时请查看代码是否最新

可是您新发布的代码中确实是采用教师模型推理的，我debug了很多次确实是用的教师模型 我发现每次在val2017推理，都会计算8次mAP，我是应该取这8次的最高值（最高值是符合您给的官方结果的），还是选择最后输出的平均值呢？ 下面两个图分别是8次测试的最高值和最后结果：

[wjm202]

 

我发现每次在val2017推理，都会计算8次mAP，我是应该取这8次的最高值（最高值是符合您给的官方结果的），还是选择最后输出的平均值呢？ 下面两个图分别是8次测试的最高值和最后结果： ------------------ 原始邮件 ------------------ 发件人: "Megvii-BaseDetection/DenseTeacher" @.>; 发送时间: 2022年10月9日(星期天) 下午5:29 @.>; @.**@.>; 主题: Re: [Megvii-BaseDetection/DenseTeacher] There is a gap between the self test results and the official in coco dataset (Issue #14)

您好，请测试teacher模型的性能（权重为checkpoint['ema']），而不是student模型(权重为checkpoint['model'])。 p1的情况应当相同，同时请查看代码是否最新

—

Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[ZRandomize]

理论上cvpods不会计算8次mAP。根据config，cvpods每2000 iter分别计算一次teacher和student的mAP。teacher的性能先输出，随后输出student的性能。注意为了跟踪模型性能曲线，我们设置每2000 iter eval一次，而不是只在训练结束后做一次eval

[wjm202]

ok，thanks

[wjm202]

好的谢谢！

---Original--- From: @.> Date: Sun, Oct 9, 2022 17:29 PM To: @.>; Cc: @.**@.>; Subject: Re: [Megvii-BaseDetection/DenseTeacher] There is a gap between the self test results and the official in coco dataset (Issue #14)

您好，请测试teacher模型的性能（权重为checkpoint['ema']），而不是student模型(权重为checkpoint['model'])。 p1的情况应当相同，同时请查看代码是否最新

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>