search

khurramHashmi / FeatEnHancer

[ICCV 2023] FeatEnHancer: Enhancing Hierarchical Features for Object Detection and Beyond Under Low-Light Vision
https://khurramhashmi.github.io/FeatEnHancer/
Other
15 stars 0 forks source link

Confusing training efficiency #7

Closed 225ceV closed 2 weeks ago

225ceV commented 3 weeks ago

This study is intriguing, and I am reproducing it using the provided training scripts. However, the training efficiency of FQ R-CNN is surprisingly low due to the absence of pre-trained weight files. Below is a log file documenting my training from scratch on the exdark dataset. my environment

-------------------------------  --------------------------------------------------------------------------------------------
sys.platform                     linux
Python                           3.7.16 (default, Jan 17 2023, 22:20:44) [GCC 11.2.0]
numpy                            1.21.6
detectron2                       0.6 @/home/jjk/workspace/REF/detectron2/detectron2
Compiler                         GCC 11.4
CUDA compiler                    not available
DETECTRON2_ENV_MODULE            <not set>
PyTorch                          1.10.0+cu111 @/home/jjk/miniconda3/envs/featenhancer/lib/python3.7/site-packages/torch
PyTorch debug build              False
torch._C._GLIBCXX_USE_CXX11_ABI  False
GPU available                    Yes
GPU 0                            NVIDIA GeForce RTX 3080 Ti Laptop GPU (arch=8.6)
Driver version                   546.80
CUDA_HOME                        None - invalid!
Pillow                           9.5.0
torchvision                      0.11.0+cu111 @/home/jjk/miniconda3/envs/featenhancer/lib/python3.7/site-packages/torchvision
torchvision arch flags           /home/jjk/miniconda3/envs/featenhancer/lib/python3.7/site-packages/torchvision/_C.so
fvcore                           0.1.5.post20221221
iopath                           0.1.9
cv2                              3.4.2
-------------------------------  --------------------------------------------------------------------------------------------
PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
  - CuDNN 8.0.5
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

i use coco format exdark dataset. download from https://github.com/cs-chan/Exclusively-Dark-Image-Dataset after 39999 iter it got

|  AP   |  AP50  |  AP75  |  APs  |  APm  |  APl  |
|:-----:|:------:|:------:|:-----:|:-----:|:-----:|
| 0.004 | 0.019  | 0.000  | 0.000 | 0.000 | 0.005 |
[08/18 22:05:31] d2.evaluation.coco_evaluation INFO: Per-category bbox AP: 
| category   | AP    | category   | AP    | category     | AP    |
|:-----------|:------|:-----------|:------|:-------------|:------|
| person     | 0.016 | bicycle    | 0.000 | car          | 0.000 |
| motorcycle | 0.000 | bus        | 0.000 | boat         | 0.000 |
| cat        | 0.000 | dog        | 0.000 | bottle       | 0.000 |
| cup        | 0.000 | chair      | 0.010 | dining table | 0.019 |

it has low mAP after hours of training. I also tried another dataset and get similar result. I am very interested in the implementation of this paper, it helped me a lot Additionally, I would appreciate it if you could provide the pre-trained weights for exdark.

225ceV commented 3 weeks ago

I commented this line in the config file due to missing weights https://github.com/khurramHashmi/FeatEnHancer/blob/33da74366d95ebb9b9a94ca85db99ea94f844308/low-light-object-detection-detectron2/configs/exdark_config.yaml#L329

khurramHashmi commented 3 weeks ago

Hi, Thanks for your interest in our work.

https://github.com/khurramHashmi/FeatEnHancer/blob/33da74366d95ebb9b9a94ca85db99ea94f844308/low-light-object-detection-detectron2/configs/exdark_config.yaml#L329

Please note that this line should not be commented out. FQ R-CNN with ResNet-101 is a large model, and pre-trained weights on the COCO dataset are necessary to ensure stable training. You can download these weights from Here and fine-tune them on the ExDark dataset.