search

facebookresearch / detectron2

Detectron2 is a platform for object detection, segmentation and other visual recognition tasks.
https://detectron2.readthedocs.io/en/latest/
Apache License 2.0
30.21k stars 7.44k forks source link

Use CI to test and release detectron2 for Windows #1712

Open shlpu opened 4 years ago

shlpu commented 4 years ago

Are you willing to release Pre-Built Detectron2 For Windows? Thanks a lot.

ppwwyyxx commented 4 years ago

We don't have such plans, but would appreciate it if anyone is willing to help us set up the CI on windows.

sarthakforwet commented 4 years ago

Hey @shlpu nice thought! @ppwwyyxx I wanna work on this issue can you please tell me what I need to know related to the issue before start working on it. Thanks.

ppwwyyxx commented 4 years ago

We use CircleCI at https://github.com/facebookresearch/detectron2/blob/master/.circleci/config.yml. It provides windows support so it can be configured with automated scripts to install dependencies, build detectron2 and run tests for every commit, on windows. https://github.com/pytorch/vision/blob/master/.circleci/config.yml can be used as a reference.

sarthakforwet commented 4 years ago

@ppwwyyxx I am successful in installing dependencies with windows executor but quite unable to find out how should I install detectron2 on windows.

Here's my sample config file (only for experiments not the complete ready code) - 

version: 2.1

windows-cpu: &windows_cpu
    machine:
      resource_class: windows.medium
      image: windows-server-2019-vs2019:stable
      shell: bash.exe

windows-gpu: &windows_gpu
    machine:
      resource_class: windows.gpu.nvidia.medium
      image: windows-server-2019-nvidia:stable
      shell: bash.exe

cpu: &cpu
  docker:
    - image: circleci/python:3.6.8-stretch
  resource_class: medium

gpu: &gpu
  machine:
    image: ubuntu-1604:201903-01
    docker_layer_caching: true
  resource_class: gpu.small

setup_venv: &setup_venv
  - run:
      name: Setup Virtual Env
      command: |
        python -m venv ~/venv
        echo ". ~/venv/bin/activate" >> $BASH_ENV
        . ~/venv/bin/activate
        python --version
        which python
        which pip
        pip install --upgrade pip

install_dep: &install_dep
  - run:
      name: Install Dependencies
      command: |
        python -m pip install --upgrade pip
        python --version
        python -m pip install 'git+https://github.com/facebookresearch/fvcore'
        python -m pip install opencv-python pytest-xdist
        pip install torch==1.6.0+cpu torchvision==0.7.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

install_detectron2: &install_detectron2
  - run:
      name: Install Detectron2
      command: |
        git clone https://github.com/conansherry/detectron2
        cd detectron2
        python setup.py build develop

jobs:
  windows_cpu_tests:
    <<: *windows_cpu
    steps:
      - checkout
      - <<: *install_dep
      - <<: *install_detectron2

workflows:
  regular:
    jobs:
      - windows_cpu_tests

I am unable to build detectron2 this way. I've found a windows build for detectron2, can we build it like that or any other way? Thanks

sarthakforwet commented 4 years ago

@ppwwyyxx, The only place where I am getting stuck is the installation of pycocotools on windows. As this package is necessary for the installation of detectron2, we need to find a way to install pycocotools on windows.

Related issue: https://github.com/cocodataset/cocoapi/issues/169

PS: I have tried the solution provided in the aforementioned issue but on executing any of the following commands, it exits out with an error.

soad89 commented 4 years ago

Hi. @sarthakforwet what error you get? I am ussualy able to install it on Win10, with some modifications in setup. I can try to assist.

sarthakforwet commented 4 years ago

Hey @soad89 thanks for the support! So basically I tried the following-

and then tried the below approaches to build detectron2:

$ pip install -U --progress-bar off -e .[dev]

This gave the following error-

(detectron_windows_setup) F:\Code\detectron2>pip install -U --progress-bar off -e .[dev]
Obtaining file:///F:/Code/detectron2
Requirement already satisfied, skipping upgrade: termcolor>=1.1 in f:\code\pythonvs\detectron_windows_setup\lib\site-packages (from detectron2==0.2.1) (1.1.0)
Requirement already satisfied, skipping upgrade: Pillow>=7.1 in f:\code\pythonvs\detectron_windows_setup\lib\site-packages (from detectron2==0.2.1) (7.2.0)
Requirement already satisfied, skipping upgrade: yacs>=0.1.6 in f:\code\pythonvs\detectron_windows_setup\lib\site-packages (from detectron2==0.2.1) (0.1.7)
Requirement already satisfied, skipping upgrade: tabulate in f:\code\pythonvs\detectron_windows_setup\lib\site-packages (from detectron2==0.2.1) (0.8.7)
Collecting cloudpickle (from detectron2==0.2.1)
  Using cached https://files.pythonhosted.org/packages/8f/d7/bdb029d3de92c165afe6cdc69a2d01e26189b3bad64a1edf6ef5a5f11d1a/cloudpickle-1.5.0-py3-none-any.whl
....
Installing collected packages: cloudpickle, kiwisolver, python-dateutil, cycler, matplotlib, mock, pyasn1, rsa, pyasn1-modules, cachetools, google-auth, markdown, certifi, chardet, urllib3, idna, requests, werkzeug, tensorboard-plugin-wit, grpcio, wheel, absl-py, oauthlib, requests-oauthlib, google-auth-oauthlib, protobuf, tensorboard, cython, pycocotools, pydot, mccabe, pycodestyle, pyflakes, flake8, isort, appdirs, typed-ast, click, black, flake8-bugbear, flake8-comprehensions, detectron2
  Running setup.py install for absl-py ... done
  Running setup.py install for pycocotools ... error
    ERROR: Command errored out with exit status 1:
     command: 'f:\code\pythonvs\detectron_windows_setup\scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\sarth\\AppData\\Local\\Temp\\pip-install-faz_dy1k\\pycocotools\\setup.py'"'"'; __file__='"'"'C:\\Users\\sarth\\AppData\\Local\\Temp\\pip-install-faz_dy1k\\pycocotools\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\sarth\AppData\Local\Temp\pip-record-3f9iryfy\install-record.txt' --single-version-externally-managed --compile --install-headers 'f:\code\pythonvs\detectron_windows_setup\include\site\python3.8\pycocotools'
         cwd: C:\Users\sarth\AppData\Local\Temp\pip-install-faz_dy1k\pycocotools\
    Complete output (20 lines):
    running install
    running build
    running build_py
    creating build
    creating build\lib.win-amd64-3.8
    creating build\lib.win-amd64-3.8\pycocotools
    copying pycocotools\coco.py -> build\lib.win-amd64-3.8\pycocotools
    copying pycocotools\cocoeval.py -> build\lib.win-amd64-3.8\pycocotools
    copying pycocotools\mask.py -> build\lib.win-amd64-3.8\pycocotools
    copying pycocotools\__init__.py -> build\lib.win-amd64-3.8\pycocotools
    running build_ext
    skipping 'pycocotools\_mask.c' Cython extension (up-to-date)
    building 'pycocotools._mask' extension
    creating build\temp.win-amd64-3.8
    creating build\temp.win-amd64-3.8\Release
    creating build\temp.win-amd64-3.8\Release\common
    creating build\temp.win-amd64-3.8\Release\pycocotools
    C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -If:\code\pythonvs\detectron_windows_setup\lib\site-packages\numpy\core\include -I./common -If:\code\pythonvs\detectron_windows_setup\include -IC:\Python38\include -IC:\Python38\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /Tc./common/maskApi.c /Fobuild\temp.win-amd64-3.8\Release\./common/maskApi.obj -Wno-cpp -Wno-unused-function -std=c99
    cl : Command line error D8021 : invalid numeric argument '/Wno-cpp'
    error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.26.28801\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'f:\code\pythonvs\detectron_windows_setup\scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\sarth\\AppData\\Local\\Temp\\pip-install-faz_dy1k\\pycocotools\\setup.py'"'"'; __file__='"'"'C:\\Users\\sarth\\AppData\\Local\\Temp\\pip-install-faz_dy1k\\pycocotools\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\sarth\AppData\Local\Temp\pip-record-3f9iryfy\install-record.txt' --single-version-externally-managed --compile --install-headers 'f:\code\pythonvs\detectron_windows_setup\include\site\python3.8\pycocotools' Check the logs for full command output.
WARNING: You are using pip version 19.2.3, however version 20.2.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

As can be seen, it errored out at building pycocotools.

After that I tried the following command (in a new environment with aforementioned deps installed).

$ python setup.py build develop

This gave me the following error:

$ python setup.py build install
running build
running build_py
creating build
creating build\lib.win-amd64-3.8
creating build\lib.win-amd64-3.8\detectron2
copying detectron2\__init__.py -> build\lib.win-amd64-3.8\detectron2
creating build\lib.win-amd64-3.8\detectron2\checkpoint
copying detectron2\checkpoint\c2_model_loading.py -> build\lib.win-amd64-3.8\detectron2\checkpoint
copying detectron2\checkpoint\catalog.py -> build\lib.win-amd64-3.8\detectron2\checkpoint
copying detectron2\checkpoint\detection_checkpoint.py -> build\lib.win-amd64-3.8\detectron2\checkpoint
copying detectron2\checkpoint\__init__.py -> build\lib.win-amd64-3.8\detectron2\checkpoint
....
F:\Code\detectron2\detectron2\layers\csrc\cocoeval\cocoeval.cpp(483): error C3861: 'localtime_r': identifier not found
F:\Code\PythonVS\det2\lib\site-packages\torch\utils\cpp_extension.py:335: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
  warnings.warn(msg.format('we could not find ninja.'))
F:\Code\PythonVS\det2\lib\site-packages\torch\utils\cpp_extension.py:270: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified
  warnings.warn('Error checking compiler version for {}: {}'.format(compiler, error))
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.26.28801\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "F:\Code\PythonVS\det2\lib\site-packages\colorama\ansitowin32.py", line 59, in closed
    return stream.closed
ValueError: underlying buffer has been detached

Then I installed deps specified in conansherry's repo (in a new environment)

$ python -m pip install 'git+https://github.com/facebookresearch/fvcore'
$ python -m pip install opencv-python pytest-xdist
$ pip install torch==1.6.0+cpu torchvision==0.7.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
$ pip install cython
$ pip install git+https://github.com/philferriere/cocoapi.git#subdirectory=PythonAPI

and then ran the following command.

$ cd detectron2
$ python setup.py build develop

It gave me the same error mentioned above with directory as original detectron2 repo and gives the following error with directory as conansherry's detectron2 repo:

(det4)
sarth@DESKTOP-5VEVU5R MINGW64 /f/Code/tmp/detectron2 (master)
$ python setup.py build develop
running build
running build_py
running build_ext
running develop
running egg_info
writing detectron2.egg-info\PKG-INFO
writing dependency_links to detectron2.egg-info\dependency_links.txt
writing requirements to detectron2.egg-info\requires.txt
writing top-level names to detectron2.egg-info\top_level.txt
reading manifest file 'detectron2.egg-info\SOURCES.txt'
writing manifest file 'detectron2.egg-info\SOURCES.txt'
running build_ext
copying build\lib.win-amd64-3.8\detectron2\_C.cp38-win_amd64.pyd -> detectron2
Creating f:\code\pythonvs\det4\lib\site-packages\detectron2.egg-link (link to .)
detectron2 0.1 is already the active version in easy-install.pth

Installed f:\code\tmp\detectron2
Processing dependencies for detectron2==0.1
Searching for matplotlib
Reading https://pypi.org/simple/matplotlib/
Downloading https://files.pythonhosted.org/packages/7d/d4/e4c40c62cd8608ca09f0684e64139c56512e195f2351ac41a472d4dc8b38/matplotlib-3.3.0.tar.gz#sha256=24e8db94948019d531ce0bcd637ac24b1c8f6744ac86d2aa0eb6dbaeb1386f82
Best match: matplotlib 3.3.0
Processing matplotlib-3.3.0.tar.gz
Writing C:\Users\sarth\AppData\Local\Temp\easy_install-rypzalv4\matplotlib-3.3.0\setup.cfg
Running matplotlib-3.3.0\setup.py -q bdist_egg --dist-dir C:\Users\sarth\AppData\Local\Temp\easy_install-rypzalv4\matplotlib-3.3.0\egg-dist-tmp-0rl_s2v2
F:\Code\PythonVS\det4\lib\site-packages\torch\utils\cpp_extension.py:335: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
  warnings.warn(msg.format('we could not find ninja.'))
F:\Code\PythonVS\det4\lib\site-packages\torch\utils\cpp_extension.py:270: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified
  warnings.warn('Error checking compiler version for {}: {}'.format(compiler, error))
UPDATING build\lib.win-amd64-3.8\matplotlib\_version.py
set build\lib.win-amd64-3.8\matplotlib\_version.py to '3.3.0'
error: Setup script exited with error: Failed to download FreeType. Please download one of ['https://downloads.sourceforge.net/project/freetype/freetype2/2.6.1/freetype-2.6.1.tar.gz', 'https://download.savannah.gnu.org/releases/freetype/freetype-2.6.1.tar.gz'] and extract it into build\freetype-2.6.1 at the top-level of the source repository.
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "F:\Code\PythonVS\det4\lib\site-packages\colorama\ansitowin32.py", line 59, in closed
    return stream.closed
ValueError: underlying buffer has been detached

PS: I also tried to install latest versions of torch and torchvision but they also errored out at building of pycocotools. Even if I would be able to build from conansherry's repo, the version of detectron2 would still be 0.1.

ppwwyyxx commented 4 years ago

The pycocotools package on PyPI is built from https://github.com/ppwwyyxx/cocoapi. If there is a simple fix to make it support windows, feel free to send a PR there.

F:\Code\detectron2\detectron2\layers\csrc\cocoeval\cocoeval.cpp(483): error C3861: 'localtime_r': identifier not found

Our code is using a time function that's not available on windows. It should be replaced by something else.

sarthakforwet commented 4 years ago

Our code is using a time function that's not available on windows. It should be replaced by something else.

Can you please guide me how can I understand the underlying code and what should be prerequisite for it so that I can work over it. Thanks!

soad89 commented 4 years ago

Hi Guys. So I had a look on https://github.com/ppwwyyxx/cocoapi

On Win10 in my case I have to edit following section in https://github.com/ppwwyyxx/cocoapi/blob/master/PythonAPI/setup.py

ext_modules = [ Extension( 'pycocotools._mask', sources=['./common/maskApi.c', 'pycocotools/_mask.pyx'], include_dirs = [np.get_include(), './common'], extra_compile_args=['-Wno-cpp', '-Wno-unused-function', '-std=c99'], ) ]

to:

ext_modules = [ Extension( 'pycocotools._mask', sources=['../common/maskApi.c', 'pycocotools/_mask.pyx'], include_dirs = [np.get_include(), '../common/'], extra_compile_args=[] ) ]

Please try it if it works also for you. If so one can distinguish in setup by os.name and modify ext_modulesto meet Win10.

sarthakforwet commented 4 years ago

Thanks @soad89! I have changed the code as you suggested and used the platform module to differentiate between Windows and other OS.