search

open-mmlab / mmengine

OpenMMLab Foundational Library for Training Deep Learning Models
https://mmengine.readthedocs.io/
Apache License 2.0
1.14k stars 340 forks source link

[Bug] failed to inherit some feature between repository of open-mmlab such as get_model #1158

Open alaa-shubbak opened 1 year ago

alaa-shubbak commented 1 year ago

Prerequisite

Environment

OrderedDict([('sys.platform', 'linux'), ('Python', '3.8.11 (default, Jul 13 2021, 14:00:03) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]'), ('CUDA available', True), ('numpy_random_seed', 2147483648), ('GPU 0', 'Tesla V100-PCIE-32GB'), ('CUDA_HOME', '/usr/local/cuda'), ('NVCC', 'Cuda compilation tools, release 12.1, V12.1.105'), ('GCC', 'gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)'), ('PyTorch', '2.0.0+cu117'), ('PyTorch compiling details', 'PyTorch built with:\n - GCC 9.3\n - C++ Version: 201703\n - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications\n - Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n - OpenMP 201511 (a.k.a. OpenMP 4.5)\n - LAPACK is enabled (usually provided by MKL)\n - NNPACK is enabled\n - CPU capability usage: AVX2\n - CUDA Runtime 11.7\n - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86\n - CuDNN 8.5\n - Magma 2.6.1\n - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.7, CUDNN_VERSION=8.5.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, \n'), ('TorchVision', '0.15.1+cu117'), ('OpenCV', '4.7.0'), ('MMEngine', '0.7.3')])

Reproduces the problem - code sample

i am try to run my custom config of mmpretrain model on my environmnet as bellow

python tools/train.py configs/simclr/simclr_resnet50_8xb32-coslr-200e_all_ACID_Em_att.py --work-dir train_dir/ACID_all/simclr_resnet50_att/

my config is based on the simclr , with addition lines about having the plugin feature of attention mechanism within resnet-50 model.

here my first plog of error and more details : with mmpretrain

and this with mmpose

with mmpose

Reproduces the problem - command or script

it is failed to us both 

model = get_model(
    'mmdet::models/backbones/resnet.py', pretrained=True)

in my config of mmpretrain , although i had but this line from mmengine.hub import get_model

on both train.py file as well as mmpretrain/models/selfsup/simclr.py

Reproduces the problem - error message

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/shubbak/pose_env2/lib/python3.8/site-packages/mmengine/hub/hub.py", line 77, in get_model
    cfg = get_config(cfg_path, pretrained)
  File "/home/shubbak/pose_env2/lib/python3.8/site-packages/mmengine/hub/hub.py", line 51, in get_config
    cfg_path = _get_external_cfg_base_path(package_path, cfg_path)
  File "/home/shubbak/pose_env2/lib/python3.8/site-packages/mmengine/config/utils.py", line 101, in _get_external_cfg_base_path
    check_file_exist(cfg_path)
  File "/home/shubbak/pose_env2/lib/python3.8/site-packages/mmengine/utils/path.py", line 23, in check_file_exist
    raise FileNotFoundError(msg_tmpl.format(filename))
FileNotFoundError: file "/work/shubbak/mmdetection/mmdet/.mim/configs/models/backbones/resnet.py" does not exist

Additional information

I would like to be able to inherit some feature or function from the mmdetction to use it in mmpretrain. not exactly for object detection or downstream task

like this nherit configuration files across repository

i want to use resnet model from mmdet as it has the feeature of plugin to inherit in my mmpretrain model

zhouzaida commented 1 year ago

Hi, does this file (/work/shubbak/mmdetection/mmdet/.mim/configs/models/backbones/resnet.py) exist?

alaa-shubbak commented 1 year ago

No

Hi, does this file (/work/shubbak/mmdetection/mmdet/.mim/configs/models/backbones/resnet.py) exist?

No , i can not find the (.mim) folder in the directory , I only find the normal files as image:

image

i downloaded the mmdetection for my work since long time (5 to 6 months) using the normal download term , like in Case a: If you develop and run mmdet directly, install it from source: although there was not the mmengine there , only with mmcv. I did not use mim for installation as well.

shall i download it again with .mim ? or what? will be lots of changing in the config format , and writing of the structure of the model within using this mmengine?

thanks in advanced

alaa-shubbak commented 1 year ago

/.mim/configs/models/backbones/resnet.py

how can i have such directory in my folder?

HAOCHENYE commented 1 year ago

/.mim/configs/models/backbones/resnet.py

how can i have such directory in my folder?

Hi, you should install mmdet by pip install -e . or pip install mmdet.

alaa-shubbak commented 1 year ago

hello , i have the two directories (mmpretrain and mmdetection ) downloaded separately shall i have mmdetection inside the folder of mmpretrain ,so i can inherit parts from one to another?

as i try to download pip install mmdet., inside my folder of mmpretrian , this gave me error ERROR: Invalid requirement: 'mmdet.'

HAOCHENYE commented 1 year ago

You can put mmpretrain and mmdetection in different directory. The only thing you need to do is to execute pip install -e . in both directories.

alaa-shubbak commented 1 year ago

although i try to install it as you mentioned , i could not find directory .mim in both mmdetcetion and mmpretrain directory , should it be some where in the environment. ? i can find the normal folders of config, mmdet ..etc which the files that i shall inherit , but i still don't know how to inherit correctly