Closed yxchng closed 1 month ago
The following is my output from running eval_all.py:
(ClearCLIP) cyx@master:/cluster/data7a/cyx/jx/ClearCLIP$ python3 eval_all.py 2>&1 | tee reproduce.log /cluster/home2/cyx/miniconda3/envs/ClearCLIP/lib/python3.10/site-packages/torch/distributed/launch.py:180: FutureWarning: The module torch.distributed.launch is deprecated and will be removed in future. Use torchrun. Note that --use_env is set by default in torchrun. If your script expects `--local_rank` argument to be set, please change it to read from `os.environ['LOCAL_RANK']` instead. See https://pytorch.org/docs/stable/distributed.html#launch-utility for further instructions warnings.warn( WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. ***************************************** Extension horovod.torch has not been built: /cluster/home2/cyx/miniconda3/envs/ClearCLIP/lib/python3.10/site-packages/horovod/torch/mpi_lib_v2.cpython-310-x86_64-linux-gnu.so not found If this is not expected, reinstall Horovod with HOROVOD_WITH_PYTORCH=1 to debug the build error. Warning! MPI libs are missing, but python applications are still available. Extension horovod.torch has not been built: /cluster/home2/cyx/miniconda3/envs/ClearCLIP/lib/python3.10/site-packages/horovod/torch/mpi_lib_v2.cpython-310-x86_64-linux-gnu.so not found If this is not expected, reinstall Horovod with HOROVOD_WITH_PYTORCH=1 to debug the build error. Warning! MPI libs are missing, but python applications are still available. Extension horovod.torch has not been built: /cluster/home2/cyx/miniconda3/envs/ClearCLIP/lib/python3.10/site-packages/horovod/torch/mpi_lib_v2.cpython-310-x86_64-linux-gnu.so not found If this is not expected, reinstall Horovod with HOROVOD_WITH_PYTORCH=1 to debug the build error. Warning! MPI libs are missing, but python applications are still available. Extension horovod.torch has not been built: /cluster/home2/cyx/miniconda3/envs/ClearCLIP/lib/python3.10/site-packages/horovod/torch/mpi_lib_v2.cpython-310-x86_64-linux-gnu.so not found If this is not expected, reinstall Horovod with HOROVOD_WITH_PYTORCH=1 to debug the build error. Warning! MPI libs are missing, but python applications are still available. 09/26 15:32:44 - mmengine - INFO - ------------------------------------------------------------ System environment: sys.platform: linux Python: 3.10.14 (main, May 6 2024, 19:42:50) [GCC 11.2.0] CUDA available: True numpy_random_seed: 295349279 GPU 0,1,2,3,4,5,6,7: NVIDIA GeForce RTX 3090 CUDA_HOME: /usr/local/cuda-11.7 NVCC: Cuda compilation tools, release 11.7, V11.7.64 GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 PyTorch: 1.13.1+cu117 PyTorch compiling details: PyTorch built with: - GCC 9.3 - C++ Version: 201402 - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815) - OpenMP 201511 (a.k.a. OpenMP 4.5) - LAPACK is enabled (usually provided by MKL) - NNPACK is enabled - CPU capability usage: AVX2 - CUDA Runtime 11.7 - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86 - CuDNN 8.5 - Magma 2.6.1 - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.7, CUDNN_VERSION=8.5.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.13.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, TorchVision: 0.14.1+cu117 OpenCV: 4.6.0 MMEngine: 0.8.4 Runtime environment: cudnn_benchmark: True mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0} dist_cfg: {'backend': 'nccl'} seed: 295349279 Distributed launcher: pytorch Distributed training: True GPU number: 4 ------------------------------------------------------------ 09/26 15:32:44 - mmengine - INFO - Config: data_root = './data/VOCdevkit/VOC2012/' dataset_type = 'PascalVOC20Dataset' default_hooks = dict( checkpoint=dict(by_epoch=False, interval=2000, type='CheckpointHook'), logger=dict(interval=50, log_metric_by_epoch=False, type='LoggerHook'), param_scheduler=dict(type='ParamSchedulerHook'), sampler_seed=dict(type='DistSamplerSeedHook'), timer=dict(type='IterTimerHook'), visualization=dict(interval=2, type='SegVisualizationHook')) default_scope = 'mmseg' env_cfg = dict( cudnn_benchmark=True, dist_cfg=dict(backend='nccl'), mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0)) launcher = 'pytorch' load_from = None log_level = 'INFO' log_processor = dict(by_epoch=False) model = dict( clip_type='CLIP', ignore_residual=True, model_type='vanilla', name_path='./configs/cls_voc20.txt', type='ClearCLIPSegmentation', vit_type='ViT-B/16') resume = False test_cfg = dict(type='TestLoop') test_dataloader = dict( batch_size=1, dataset=dict( ann_file='ImageSets/Segmentation/val.txt', data_prefix=dict( img_path='JPEGImages', seg_map_path='SegmentationClass'), data_root='./data/VOCdevkit/VOC2012/', pipeline=[ dict(type='LoadImageFromFile'), dict(keep_ratio=True, scale=( 2048, 448, ), type='Resize'), dict(type='LoadAnnotations'), dict(type='PackSegInputs'), ], type='PascalVOC20Dataset'), num_workers=4, persistent_workers=True, sampler=dict(shuffle=False, type='DefaultSampler')) test_evaluator = dict( iou_metrics=[ 'mIoU', ], type='IoUMetric') test_pipeline = [ dict(type='LoadImageFromFile'), dict(keep_ratio=True, scale=( 2048, 448, ), type='Resize'), dict(type='LoadAnnotations'), dict(type='PackSegInputs'), ] vis_backends = [ dict(type='LocalVisBackend'), ] visualizer = dict( alpha=1.0, name='visualizer', type='SegLocalVisualizer', vis_backends=[ dict(type='LocalVisBackend'), ]) work_dir = './work_logs' 09/26 15:32:48 - mmengine - INFO - Hooks will be executed in the following order: before_run: (VERY_HIGH ) RuntimeInfoHook (BELOW_NORMAL) LoggerHook -------------------- before_train: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (VERY_LOW ) CheckpointHook -------------------- before_train_epoch: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (NORMAL ) DistSamplerSeedHook -------------------- before_train_iter: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook -------------------- after_train_iter: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (NORMAL ) SegVisualizationHook (BELOW_NORMAL) LoggerHook (LOW ) ParamSchedulerHook (VERY_LOW ) CheckpointHook -------------------- after_train_epoch: (NORMAL ) IterTimerHook (LOW ) ParamSchedulerHook (VERY_LOW ) CheckpointHook -------------------- before_val: (VERY_HIGH ) RuntimeInfoHook -------------------- before_val_epoch: (NORMAL ) IterTimerHook -------------------- before_val_iter: (NORMAL ) IterTimerHook -------------------- after_val_iter: (NORMAL ) IterTimerHook (NORMAL ) SegVisualizationHook (BELOW_NORMAL) LoggerHook -------------------- after_val_epoch: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook (LOW ) ParamSchedulerHook (VERY_LOW ) CheckpointHook -------------------- after_val: (VERY_HIGH ) RuntimeInfoHook -------------------- after_train: (VERY_HIGH ) RuntimeInfoHook (VERY_LOW ) CheckpointHook -------------------- before_test: (VERY_HIGH ) RuntimeInfoHook -------------------- before_test_epoch: (NORMAL ) IterTimerHook -------------------- before_test_iter: (NORMAL ) IterTimerHook -------------------- after_test_iter: (NORMAL ) IterTimerHook (NORMAL ) SegVisualizationHook (BELOW_NORMAL) LoggerHook -------------------- after_test_epoch: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook -------------------- after_test: (VERY_HIGH ) RuntimeInfoHook -------------------- after_run: (BELOW_NORMAL) LoggerHook -------------------- 09/26 15:32:49 - mmengine - WARNING - The prefix is not set in metric class IoUMetric. 09/26 15:32:51 - mmengine - INFO - Iter(test) [ 50/363] eta: 0:00:13 time: 0.0255 data_time: 0.0016 memory: 758 09/26 15:32:52 - mmengine - INFO - Iter(test) [100/363] eta: 0:00:08 time: 0.0252 data_time: 0.0016 memory: 745 09/26 15:32:53 - mmengine - INFO - Iter(test) [150/363] eta: 0:00:06 time: 0.0233 data_time: 0.0016 memory: 744 09/26 15:32:54 - mmengine - INFO - Iter(test) [200/363] eta: 0:00:04 time: 0.0274 data_time: 0.0016 memory: 745 09/26 15:32:56 - mmengine - INFO - Iter(test) [250/363] eta: 0:00:03 time: 0.0254 data_time: 0.0016 memory: 745 09/26 15:32:57 - mmengine - INFO - Iter(test) [300/363] eta: 0:00:01 time: 0.0246 data_time: 0.0018 memory: 747 09/26 15:32:58 - mmengine - INFO - Iter(test) [350/363] eta: 0:00:00 time: 0.0225 data_time: 0.0016 memory: 746 09/26 15:33:01 - mmengine - INFO - per class results: 09/26 15:33:01 - mmengine - INFO - +-------------+-------+-------+ | Class | IoU | Acc | +-------------+-------+-------+ | aeroplane | 96.38 | 99.78 | | bicycle | 54.39 | 97.62 | | bird | 93.64 | 94.41 | | boat | 85.71 | 93.59 | | bottle | 85.95 | 91.15 | | bus | 94.22 | 99.5 | | car | 83.22 | 97.37 | | cat | 90.47 | 98.77 | | chair | 28.76 | 33.64 | | cow | 88.73 | 95.18 | | diningtable | 40.57 | 91.74 | | dog | 85.67 | 95.77 | | horse | 85.32 | 96.69 | | motorbike | 75.15 | 92.38 | | person | 48.88 | 49.78 | | pottedplant | 72.72 | 86.97 | | sheep | 81.77 | 97.98 | | sofa | 62.24 | 89.46 | | train | 97.16 | 99.75 | | tvmonitor | 63.97 | 64.89 | +-------------+-------+-------+ 09/26 15:33:01 - mmengine - INFO - Iter(test) [363/363] aAcc: 83.7900 mIoU: 75.7500 mAcc: 88.3200 data_time: 0.0024 time: 0.0267 /cluster/home2/cyx/miniconda3/envs/ClearCLIP/lib/python3.10/site-packages/torch/distributed/launch.py:180: FutureWarning: The module torch.distributed.launch is deprecated and will be removed in future. Use torchrun. Note that --use_env is set by default in torchrun. If your script expects `--local_rank` argument to be set, please change it to read from `os.environ['LOCAL_RANK']` instead. See https://pytorch.org/docs/stable/distributed.html#launch-utility for further instructions warnings.warn( WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. ***************************************** Extension horovod.torch has not been built: /cluster/home2/cyx/miniconda3/envs/ClearCLIP/lib/python3.10/site-packages/horovod/torch/mpi_lib_v2.cpython-310-x86_64-linux-gnu.so not found If this is not expected, reinstall Horovod with HOROVOD_WITH_PYTORCH=1 to debug the build error. Warning! MPI libs are missing, but python applications are still available. Extension horovod.torch has not been built: /cluster/home2/cyx/miniconda3/envs/ClearCLIP/lib/python3.10/site-packages/horovod/torch/mpi_lib_v2.cpython-310-x86_64-linux-gnu.so not found If this is not expected, reinstall Horovod with HOROVOD_WITH_PYTORCH=1 to debug the build error. Warning! MPI libs are missing, but python applications are still available. Extension horovod.torch has not been built: /cluster/home2/cyx/miniconda3/envs/ClearCLIP/lib/python3.10/site-packages/horovod/torch/mpi_lib_v2.cpython-310-x86_64-linux-gnu.so not found If this is not expected, reinstall Horovod with HOROVOD_WITH_PYTORCH=1 to debug the build error. Warning! MPI libs are missing, but python applications are still available. Extension horovod.torch has not been built: /cluster/home2/cyx/miniconda3/envs/ClearCLIP/lib/python3.10/site-packages/horovod/torch/mpi_lib_v2.cpython-310-x86_64-linux-gnu.so not found If this is not expected, reinstall Horovod with HOROVOD_WITH_PYTORCH=1 to debug the build error. Warning! MPI libs are missing, but python applications are still available. 09/26 15:33:16 - mmengine - INFO - ------------------------------------------------------------ System environment: sys.platform: linux Python: 3.10.14 (main, May 6 2024, 19:42:50) [GCC 11.2.0] CUDA available: True numpy_random_seed: 1878372319 GPU 0,1,2,3,4,5,6,7: NVIDIA GeForce RTX 3090 CUDA_HOME: /usr/local/cuda-11.7 NVCC: Cuda compilation tools, release 11.7, V11.7.64 GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 PyTorch: 1.13.1+cu117 PyTorch compiling details: PyTorch built with: - GCC 9.3 - C++ Version: 201402 - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815) - OpenMP 201511 (a.k.a. OpenMP 4.5) - LAPACK is enabled (usually provided by MKL) - NNPACK is enabled - CPU capability usage: AVX2 - CUDA Runtime 11.7 - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86 - CuDNN 8.5 - Magma 2.6.1 - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.7, CUDNN_VERSION=8.5.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.13.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, TorchVision: 0.14.1+cu117 OpenCV: 4.6.0 MMEngine: 0.8.4 Runtime environment: cudnn_benchmark: True mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0} dist_cfg: {'backend': 'nccl'} seed: 1878372319 Distributed launcher: pytorch Distributed training: True GPU number: 4 ------------------------------------------------------------ 09/26 15:33:16 - mmengine - INFO - Config: data_root = './data/cityscapes' dataset_type = 'CityscapesDataset' default_hooks = dict( checkpoint=dict(by_epoch=False, interval=2000, type='CheckpointHook'), logger=dict(interval=50, log_metric_by_epoch=False, type='LoggerHook'), param_scheduler=dict(type='ParamSchedulerHook'), sampler_seed=dict(type='DistSamplerSeedHook'), timer=dict(type='IterTimerHook'), visualization=dict(interval=2, type='SegVisualizationHook')) default_scope = 'mmseg' env_cfg = dict( cudnn_benchmark=True, dist_cfg=dict(backend='nccl'), mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0)) launcher = 'pytorch' load_from = None log_level = 'INFO' log_processor = dict(by_epoch=False) model = dict( clip_type='CLIP', ignore_residual=True, model_type='vanilla', name_path='./configs/cls_city_scapes.txt', slide_crop=224, type='ClearCLIPSegmentation', vit_type='ViT-B/16') resume = False test_cfg = dict(type='TestLoop') test_dataloader = dict( batch_size=1, dataset=dict( data_prefix=dict( img_path='leftImg8bit/val', seg_map_path='gtFine/val'), data_root='./data/cityscapes', pipeline=[ dict(type='LoadImageFromFile'), dict(keep_ratio=True, scale=( 2048, 448, ), type='Resize'), dict(type='LoadAnnotations'), dict(type='PackSegInputs'), ], type='CityscapesDataset'), num_workers=4, persistent_workers=True, sampler=dict(shuffle=False, type='DefaultSampler')) test_evaluator = dict( iou_metrics=[ 'mIoU', ], type='IoUMetric') test_pipeline = [ dict(type='LoadImageFromFile'), dict(keep_ratio=True, scale=( 2048, 448, ), type='Resize'), dict(type='LoadAnnotations'), dict(type='PackSegInputs'), ] vis_backends = [ dict(type='LocalVisBackend'), ] visualizer = dict( alpha=1.0, name='visualizer', type='SegLocalVisualizer', vis_backends=[ dict(type='LocalVisBackend'), ]) work_dir = './work_logs' 09/26 15:33:21 - mmengine - INFO - Hooks will be executed in the following order: before_run: (VERY_HIGH ) RuntimeInfoHook (BELOW_NORMAL) LoggerHook -------------------- before_train: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (VERY_LOW ) CheckpointHook -------------------- before_train_epoch: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (NORMAL ) DistSamplerSeedHook -------------------- before_train_iter: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook -------------------- after_train_iter: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (NORMAL ) SegVisualizationHook (BELOW_NORMAL) LoggerHook (LOW ) ParamSchedulerHook (VERY_LOW ) CheckpointHook -------------------- after_train_epoch: (NORMAL ) IterTimerHook (LOW ) ParamSchedulerHook (VERY_LOW ) CheckpointHook -------------------- before_val: (VERY_HIGH ) RuntimeInfoHook -------------------- before_val_epoch: (NORMAL ) IterTimerHook -------------------- before_val_iter: (NORMAL ) IterTimerHook -------------------- after_val_iter: (NORMAL ) IterTimerHook (NORMAL ) SegVisualizationHook (BELOW_NORMAL) LoggerHook -------------------- after_val_epoch: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook (LOW ) ParamSchedulerHook (VERY_LOW ) CheckpointHook -------------------- after_val: (VERY_HIGH ) RuntimeInfoHook -------------------- after_train: (VERY_HIGH ) RuntimeInfoHook (VERY_LOW ) CheckpointHook -------------------- before_test: (VERY_HIGH ) RuntimeInfoHook -------------------- before_test_epoch: (NORMAL ) IterTimerHook -------------------- before_test_iter: (NORMAL ) IterTimerHook -------------------- after_test_iter: (NORMAL ) IterTimerHook (NORMAL ) SegVisualizationHook (BELOW_NORMAL) LoggerHook -------------------- after_test_epoch: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook -------------------- after_test: (VERY_HIGH ) RuntimeInfoHook -------------------- after_run: (BELOW_NORMAL) LoggerHook -------------------- 09/26 15:33:21 - mmengine - WARNING - The prefix is not set in metric class IoUMetric. 09/26 15:33:27 - mmengine - INFO - Iter(test) [ 50/125] eta: 0:00:08 time: 0.0973 data_time: 0.0043 memory: 927 09/26 15:33:31 - mmengine - INFO - Iter(test) [100/125] eta: 0:00:02 time: 0.0935 data_time: 0.0043 memory: 927 09/26 15:33:34 - mmengine - INFO - per class results: 09/26 15:33:34 - mmengine - INFO - +---------------+-------+-------+ | Class | IoU | Acc | +---------------+-------+-------+ | road | 55.15 | 81.56 | | sidewalk | 12.71 | 37.69 | | building | 35.99 | 38.27 | | wall | 6.19 | 40.47 | | fence | 9.44 | 10.26 | | pole | 1.57 | 1.72 | | traffic light | 4.04 | 61.24 | | traffic sign | 16.69 | 24.49 | | vegetation | 26.09 | 27.2 | | terrain | 0.0 | 0.0 | | sky | 4.75 | 4.77 | | person | 2.98 | 3.05 | | rider | 0.15 | 0.15 | | car | 36.87 | 77.12 | | truck | 23.02 | 72.27 | | bus | 28.06 | 82.78 | | train | 8.88 | 20.39 | | motorcycle | 16.55 | 18.89 | | bicycle | 28.75 | 45.84 | +---------------+-------+-------+ 09/26 15:33:34 - mmengine - INFO - Iter(test) [125/125] aAcc: 52.6700 mIoU: 16.7300 mAcc: 34.1100 data_time: 0.0085 time: 0.1036 /cluster/home2/cyx/miniconda3/envs/ClearCLIP/lib/python3.10/site-packages/torch/distributed/launch.py:180: FutureWarning: The module torch.distributed.launch is deprecated and will be removed in future. Use torchrun. Note that --use_env is set by default in torchrun. If your script expects `--local_rank` argument to be set, please change it to read from `os.environ['LOCAL_RANK']` instead. See https://pytorch.org/docs/stable/distributed.html#launch-utility for further instructions warnings.warn( WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. ***************************************** Extension horovod.torch has not been built: /cluster/home2/cyx/miniconda3/envs/ClearCLIP/lib/python3.10/site-packages/horovod/torch/mpi_lib_v2.cpython-310-x86_64-linux-gnu.so not found If this is not expected, reinstall Horovod with HOROVOD_WITH_PYTORCH=1 to debug the build error. Warning! MPI libs are missing, but python applications are still available. Extension horovod.torch has not been built: /cluster/home2/cyx/miniconda3/envs/ClearCLIP/lib/python3.10/site-packages/horovod/torch/mpi_lib_v2.cpython-310-x86_64-linux-gnu.so not found If this is not expected, reinstall Horovod with HOROVOD_WITH_PYTORCH=1 to debug the build error. Warning! MPI libs are missing, but python applications are still available. Extension horovod.torch has not been built: /cluster/home2/cyx/miniconda3/envs/ClearCLIP/lib/python3.10/site-packages/horovod/torch/mpi_lib_v2.cpython-310-x86_64-linux-gnu.so not found If this is not expected, reinstall Horovod with HOROVOD_WITH_PYTORCH=1 to debug the build error. Warning! MPI libs are missing, but python applications are still available. Extension horovod.torch has not been built: /cluster/home2/cyx/miniconda3/envs/ClearCLIP/lib/python3.10/site-packages/horovod/torch/mpi_lib_v2.cpython-310-x86_64-linux-gnu.so not found If this is not expected, reinstall Horovod with HOROVOD_WITH_PYTORCH=1 to debug the build error. Warning! MPI libs are missing, but python applications are still available. 09/26 15:33:53 - mmengine - INFO - ------------------------------------------------------------ System environment: sys.platform: linux Python: 3.10.14 (main, May 6 2024, 19:42:50) [GCC 11.2.0] CUDA available: True numpy_random_seed: 1388269388 GPU 0,1,2,3,4,5,6,7: NVIDIA GeForce RTX 3090 CUDA_HOME: /usr/local/cuda-11.7 NVCC: Cuda compilation tools, release 11.7, V11.7.64 GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 PyTorch: 1.13.1+cu117 PyTorch compiling details: PyTorch built with: - GCC 9.3 - C++ Version: 201402 - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815) - OpenMP 201511 (a.k.a. OpenMP 4.5) - LAPACK is enabled (usually provided by MKL) - NNPACK is enabled - CPU capability usage: AVX2 - CUDA Runtime 11.7 - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86 - CuDNN 8.5 - Magma 2.6.1 - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.7, CUDNN_VERSION=8.5.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.13.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, TorchVision: 0.14.1+cu117 OpenCV: 4.6.0 MMEngine: 0.8.4 Runtime environment: cudnn_benchmark: True mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0} dist_cfg: {'backend': 'nccl'} seed: 1388269388 Distributed launcher: pytorch Distributed training: True GPU number: 4 ------------------------------------------------------------ 09/26 15:33:53 - mmengine - INFO - Config: data_root = './data/ade/ADEChallengeData2016' dataset_type = 'ADE20KDataset' default_hooks = dict( checkpoint=dict(by_epoch=False, interval=2000, type='CheckpointHook'), logger=dict(interval=50, log_metric_by_epoch=False, type='LoggerHook'), param_scheduler=dict(type='ParamSchedulerHook'), sampler_seed=dict(type='DistSamplerSeedHook'), timer=dict(type='IterTimerHook'), visualization=dict(interval=2, type='SegVisualizationHook')) default_scope = 'mmseg' env_cfg = dict( cudnn_benchmark=True, dist_cfg=dict(backend='nccl'), mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0)) launcher = 'pytorch' load_from = None log_level = 'INFO' log_processor = dict(by_epoch=False) model = dict( clip_type='CLIP', ignore_residual=True, model_type='vanilla', name_path='./configs/cls_ade20k.txt', prob_thd=0.0, type='ClearCLIPSegmentation', vit_type='ViT-B/16') resume = False test_cfg = dict(type='TestLoop') test_dataloader = dict( batch_size=1, dataset=dict( data_prefix=dict( img_path='images/validation', seg_map_path='annotations/validation'), data_root='./data/ade/ADEChallengeData2016', pipeline=[ dict(type='LoadImageFromFile'), dict(keep_ratio=True, scale=( 2048, 448, ), type='Resize'), dict(reduce_zero_label=True, type='LoadAnnotations'), dict(type='PackSegInputs'), ], type='ADE20KDataset'), num_workers=4, persistent_workers=True, sampler=dict(shuffle=False, type='DefaultSampler')) test_evaluator = dict( iou_metrics=[ 'mIoU', ], type='IoUMetric') test_pipeline = [ dict(type='LoadImageFromFile'), dict(keep_ratio=True, scale=( 2048, 448, ), type='Resize'), dict(reduce_zero_label=True, type='LoadAnnotations'), dict(type='PackSegInputs'), ] vis_backends = [ dict(type='LocalVisBackend'), ] visualizer = dict( alpha=1.0, name='visualizer', type='SegLocalVisualizer', vis_backends=[ dict(type='LocalVisBackend'), ]) work_dir = './work_logs' 09/26 15:34:00 - mmengine - INFO - Hooks will be executed in the following order: before_run: (VERY_HIGH ) RuntimeInfoHook (BELOW_NORMAL) LoggerHook -------------------- before_train: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (VERY_LOW ) CheckpointHook -------------------- before_train_epoch: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (NORMAL ) DistSamplerSeedHook -------------------- before_train_iter: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook -------------------- after_train_iter: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (NORMAL ) SegVisualizationHook (BELOW_NORMAL) LoggerHook (LOW ) ParamSchedulerHook (VERY_LOW ) CheckpointHook -------------------- after_train_epoch: (NORMAL ) IterTimerHook (LOW ) ParamSchedulerHook (VERY_LOW ) CheckpointHook -------------------- before_val: (VERY_HIGH ) RuntimeInfoHook -------------------- before_val_epoch: (NORMAL ) IterTimerHook -------------------- before_val_iter: (NORMAL ) IterTimerHook -------------------- after_val_iter: (NORMAL ) IterTimerHook (NORMAL ) SegVisualizationHook (BELOW_NORMAL) LoggerHook -------------------- after_val_epoch: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook (LOW ) ParamSchedulerHook (VERY_LOW ) CheckpointHook -------------------- after_val: (VERY_HIGH ) RuntimeInfoHook -------------------- after_train: (VERY_HIGH ) RuntimeInfoHook (VERY_LOW ) CheckpointHook -------------------- before_test: (VERY_HIGH ) RuntimeInfoHook -------------------- before_test_epoch: (NORMAL ) IterTimerHook -------------------- before_test_iter: (NORMAL ) IterTimerHook -------------------- after_test_iter: (NORMAL ) IterTimerHook (NORMAL ) SegVisualizationHook (BELOW_NORMAL) LoggerHook -------------------- after_test_epoch: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook -------------------- after_test: (VERY_HIGH ) RuntimeInfoHook -------------------- after_run: (BELOW_NORMAL) LoggerHook -------------------- 09/26 15:34:00 - mmengine - WARNING - The prefix is not set in metric class IoUMetric. 09/26 15:34:03 - mmengine - INFO - Iter(test) [ 50/500] eta: 0:00:21 time: 0.0269 data_time: 0.0019 memory: 1187 09/26 15:34:04 - mmengine - INFO - Iter(test) [100/500] eta: 0:00:15 time: 0.0250 data_time: 0.0017 memory: 1236 09/26 15:34:05 - mmengine - INFO - Iter(test) [150/500] eta: 0:00:12 time: 0.0285 data_time: 0.0018 memory: 1023 09/26 15:34:07 - mmengine - INFO - Iter(test) [200/500] eta: 0:00:09 time: 0.0253 data_time: 0.0019 memory: 1047 09/26 15:34:08 - mmengine - INFO - Iter(test) [250/500] eta: 0:00:07 time: 0.0294 data_time: 0.0019 memory: 1077 09/26 15:34:10 - mmengine - INFO - Iter(test) [300/500] eta: 0:00:06 time: 0.0280 data_time: 0.0016 memory: 2896 09/26 15:34:11 - mmengine - INFO - Iter(test) [350/500] eta: 0:00:04 time: 0.0245 data_time: 0.0017 memory: 1022 09/26 15:34:12 - mmengine - INFO - Iter(test) [400/500] eta: 0:00:02 time: 0.0273 data_time: 0.0018 memory: 1076 09/26 15:34:13 - mmengine - INFO - Iter(test) [450/500] eta: 0:00:01 time: 0.0258 data_time: 0.0019 memory: 1027 09/26 15:34:15 - mmengine - INFO - Iter(test) [500/500] eta: 0:00:00 time: 0.0298 data_time: 0.0019 memory: 1051 09/26 15:34:17 - mmengine - INFO - per class results: 09/26 15:34:17 - mmengine - INFO - +---------------------+-------+-------+ | Class | IoU | Acc | +---------------------+-------+-------+ | wall | 4.32 | 4.78 | | building | 11.36 | 11.9 | | sky | 12.42 | 12.6 | | floor | 17.81 | 23.76 | | tree | 28.47 | 39.3 | | ceiling | 7.4 | 8.13 | | road | 26.97 | 58.99 | | bed | 28.3 | 94.03 | | windowpane | 0.76 | 0.77 | | grass | 30.88 | 57.99 | | cabinet | 3.77 | 4.31 | | sidewalk | 13.65 | 52.31 | | person | 5.09 | 5.24 | | earth | 0.0 | 0.0 | | door | 12.47 | 25.36 | | table | 11.68 | 38.28 | | mountain | 27.41 | 35.31 | | plant | 19.5 | 22.7 | | curtain | 8.22 | 9.21 | | chair | 2.49 | 2.69 | | car | 31.43 | 68.27 | | water | 5.4 | 5.8 | | painting | 4.87 | 5.89 | | sofa | 19.35 | 74.45 | | shelf | 4.88 | 5.65 | | house | 4.94 | 73.81 | | sea | 18.65 | 44.23 | | mirror | 0.98 | 0.99 | | rug | 9.16 | 22.3 | | field | 8.21 | 20.91 | | armchair | 7.71 | 20.6 | | seat | 13.1 | 36.45 | | fence | 13.84 | 19.72 | | desk | 8.6 | 44.21 | | rock | 15.69 | 47.98 | | wardrobe | 14.14 | 41.11 | | lamp | 6.27 | 7.5 | | bathtub | 17.91 | 61.02 | | railing | 1.64 | 2.12 | | cushion | 0.58 | 0.59 | | base | 0.0 | 0.0 | | box | 3.53 | 6.92 | | column | 11.46 | 18.28 | | signboard | 3.37 | 4.29 | | chest of drawers | 1.23 | 1.42 | | counter | 2.08 | 14.82 | | sand | 29.25 | 74.41 | | sink | 4.17 | 5.34 | | skyscraper | 36.25 | 89.76 | | fireplace | 21.0 | 49.3 | | refrigerator | 7.27 | 30.13 | | grandstand | 5.54 | 8.59 | | path | 6.29 | 13.21 | | stairs | 2.25 | 4.61 | | runway | 30.16 | 50.1 | | case | 0.0 | 0.0 | | pool table | 35.52 | 48.82 | | pillow | 1.42 | 1.55 | | screen door | 0.0 | 0.0 | | stairway | 11.87 | 32.44 | | river | 6.14 | 31.86 | | bridge | 9.37 | 24.42 | | bookcase | 17.82 | 44.99 | | blind | 2.48 | 2.72 | | coffee table | 1.26 | 2.07 | | toilet | 5.63 | 77.66 | | flower | 11.65 | 22.76 | | book | 5.62 | 5.94 | | hill | 1.68 | 19.52 | | bench | 10.97 | 13.23 | | countertop | 1.16 | 10.3 | | stove | 2.45 | 3.8 | | palm | 19.29 | 46.75 | | kitchen island | 2.3 | 80.29 | | computer | 21.69 | 25.49 | | swivel chair | 2.15 | 4.11 | | boat | 18.87 | 72.92 | | bar | 20.14 | 58.71 | | arcade machine | 2.33 | 2.88 | | hovel | 1.52 | 74.22 | | bus | 21.07 | 80.8 | | towel | 4.83 | 9.57 | | light | 0.0 | 0.0 | | truck | 9.09 | 32.76 | | tower | 8.83 | 89.38 | | chandelier | 12.73 | 21.76 | | awning | 1.81 | 18.88 | | streetlight | 3.26 | 5.54 | | booth | 0.85 | 10.18 | | television receiver | 6.29 | 7.47 | | airplane | 5.05 | 30.15 | | dirt track | 0.0 | 0.0 | | apparel | 0.0 | 0.0 | | pole | 0.02 | 0.03 | | land | 0.26 | 6.82 | | bannister | 2.81 | 6.3 | | escalator | 30.33 | 45.6 | | ottoman | 0.0 | 0.0 | | bottle | 20.33 | 32.7 | | buffet | 0.57 | 6.04 | | poster | 6.53 | 9.2 | | stage | 10.13 | 50.49 | | van | 4.57 | 23.93 | | ship | 2.86 | 3.77 | | fountain | 7.74 | 33.45 | | conveyer belt | 0.0 | 0.0 | | canopy | 0.0 | 0.01 | | washer | 39.83 | 56.11 | | plaything | 2.0 | 2.49 | | swimming pool | 13.25 | 96.53 | | stool | 0.4 | 1.07 | | barrel | 1.26 | 37.36 | | basket | 0.12 | 0.17 | | waterfall | 12.9 | 98.11 | | tent | 19.66 | 96.9 | | bag | 0.78 | 1.03 | | minibike | 13.59 | 13.82 | | cradle | 9.98 | 53.26 | | oven | 0.0 | 0.0 | | ball | 10.96 | 64.46 | | food | 9.69 | 32.69 | | step | 0.07 | 1.18 | | tank | 8.52 | 9.44 | | trade name | 0.77 | 1.63 | | microwave | 37.02 | 41.99 | | pot | 0.12 | 0.13 | | animal | 21.7 | 35.98 | | bicycle | 20.84 | 43.63 | | lake | 5.14 | 85.93 | | dishwasher | 0.0 | 0.0 | | screen | 21.68 | 40.13 | | blanket | 0.05 | 0.05 | | sculpture | 20.81 | 39.61 | | hood | 0.0 | 0.0 | | sconce | 0.38 | 0.49 | | vase | 2.51 | 2.52 | | traffic light | 7.65 | 18.2 | | tray | 1.38 | 2.93 | | ashcan | 0.0 | 0.0 | | fan | 0.0 | 0.0 | | pier | 6.83 | 45.81 | | crt screen | 0.13 | 0.44 | | plate | 28.35 | 39.55 | | monitor | 1.87 | 2.96 | | bulletin board | 0.41 | 1.54 | | shower | 0.24 | 33.5 | | radiator | 0.0 | 0.0 | | glass | 0.0 | 0.0 | | clock | 4.86 | 15.74 | | flag | 25.98 | 36.44 | +---------------------+-------+-------+ 09/26 15:34:17 - mmengine - INFO - Iter(test) [500/500] aAcc: 21.9200 mIoU: 9.1300 mAcc: 25.1200 data_time: 0.0024 time: 0.0292 Running ./configs/cfg_voc20.py Running ./configs/cfg_city_scapes.py Running ./configs/cfg_ade20k.py
I am getting for 75.75 for Context59, 16.73 for Cityscapes and 9.13 for ADE.
which does not seem to match
Please set the parameter _modeltype correctly. In this case, if you would to reproduce ClearCLIP's results, please set it to "ClearCLIP", instead of "vanilla".
The following is my output from running eval_all.py:
I am getting for 75.75 for Context59, 16.73 for Cityscapes and 9.13 for ADE.
which does not seem to match