Pip install via git link fails

[veenstrajelmer]

Describe the bug Pip installation via git link fails. This is the command and the resulting error message:

(dfm_tools_devenv) c:\DATA\dfm_tools>python -m pip install git+https://github.com/Deltares/hydrolib-core
Collecting git+https://github.com/Deltares/hydrolib-core
  Cloning https://github.com/Deltares/hydrolib-core to c:\snapvolumestemp\mountpoints\{45c63495-0000-0000-0000-100000000000}\{79de0690-9470-4166-b9ee-4548dc416bbd}\svroot\users\veenstra\appdata\local\temp\pip-req-build-sn6e9wbu
  Running command git clone --filter=blob:none --quiet https://github.com/Deltares/hydrolib-core 'C:\SnapVolumesTemp\MountPoints\{45c63495-0000-0000-0000-100000000000}\{79DE0690-9470-4166-B9EE-4548DC416BBD}\SVROOT\Users\veenstra\AppData\Local\Temp\pip-req-build-sn6e9wbu'
  error: unable to create file tests/data/input/e02/c11_korte-woerden-1d/dimr_model/dflowfm/FlowFM_boundaryconditions1d.bc: Filename too long
  error: unable to create file tests/data/input/e02/f101_1D-boundaries/c01_steady-state-flow/CrossSectionDefinitions.ini: Filename too long
  error: unable to create file tests/data/input/e02/f101_1D-boundaries/c01_steady-state-flow/CrossSectionLocations.ini: Filename too long
  error: unable to create file tests/data/input/e02/f101_1D-boundaries/c01_steady-state-flow/ObservationPoints_crs.ini: Filename too long
  fatal: cannot create directory at 'tests/data/input/e02/f152_1d2d_projectmodels_rhu/c04_DHydamo-MGB-initialisation': Filename too long
  warning: Clone succeeded, but checkout failed.
  You can inspect what was checked out with 'git status'
  and retry with 'git restore --source=HEAD :/'

  error: subprocess-exited-with-error

  × git clone --filter=blob:none --quiet https://github.com/Deltares/hydrolib-core 'C:\SnapVolumesTemp\MountPoints\{45c63495-0000-0000-0000-100000000000}\{79DE0690-9470-4166-B9EE-4548DC416BBD}\SVROOT\Users\veenstra\AppData\Local\Temp\pip-req-build-sn6e9wbu' did not run successfully.
  │ exit code: 128
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× git clone --filter=blob:none --quiet https://github.com/Deltares/hydrolib-core 'C:\SnapVolumesTemp\MountPoints\{45c63495-0000-0000-0000-100000000000}\{79DE0690-9470-4166-B9EE-4548DC416BBD}\SVROOT\Users\veenstra\AppData\Local\Temp\pip-req-build-sn6e9wbu' did not run successfully.
│ exit code: 128
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

To Reproduce

python -m pip install git+https://github.com/Deltares/hydrolib-core

Expected behavior A install directly from the github version, which makes it easy to test/use not yet released versions containing bug fixes.

Version info (please complete the following information):

• OS: Windows, Anaconda, Spyder
• Version: 0.3.0

Additional information An alternative would be to check out the hydrolib-core code and install with pip install -e ., but this also crashes (I think since there is no setup.py available).

(dfm_tools_devenv) c:\DATA\hydrolib-core>pip install -e .
Obtaining file:///C:/DATA/hydrolib-core
ERROR: Could not install packages due to an OSError: [WinError 3] The system cannot find the path specified: 'C:\\SnapVolumesTemp\\MountPoints\\{45c63495-0000-0000-0000-100000000000}\\{79DE0690-9470-4166-B9EE-4548DC416BBD}\\SVROOT\\Users\\veenstra\\Anaconda3\\envs\\dfm_tools_devenv\\Lib\\site-packages\\pip\\_internal\\resolution\\resolvelib\\__pycache__\\found_candidates.cpython-38.pyc'

[priscavdsluis]

Hi @veenstrajelmer, could you please try to run the following: git config --system core.longpaths true

The errors that you are getting are a result of, apparently, the too long file paths of soms of our test data files together with your local checkout location.

Hopefully this will resolve the error. Please let us know the result.

I have honestly no idea what the second error is about, and I assume it has something to do with your local environment. I could not reproduce it myself and some of our collegues also use a pip install -e. in hydrolib-core for their development.

[veenstrajelmer]

Unfortunately it does not resolve the issue, I might dive into it later, but for now will stay with the releases:

(dfm_tools_devenv) c:\DATA\dfm_tools>python -m pip install git+https://github.com/Deltares/hydrolib-core
Collecting git+https://github.com/Deltares/hydrolib-core
  Cloning https://github.com/Deltares/hydrolib-core to c:\snapvolumestemp\mountpoints\{45c63495-0000-0000-0000-100000000000}\{79de0690-9470-4166-b9ee-4548dc416bbd}\svroot\users\veenstra\appdata\local\temp\pip-req-build-cnj2ddcg
  Running command git clone --filter=blob:none --quiet https://github.com/Deltares/hydrolib-core 'C:\SnapVolumesTemp\MountPoints\{45c63495-0000-0000-0000-100000000000}\{79DE0690-9470-4166-B9EE-4548DC416BBD}\SVROOT\Users\veenstra\AppData\Local\Temp\pip-req-build-cnj2ddcg'
  Resolved https://github.com/Deltares/hydrolib-core to commit 901c6b4b3ea9425bc5363b4f9bcb907587a745e5
ERROR: Could not install packages due to an OSError: [WinError 3] The system cannot find the path specified: 'C:\\SnapVolumesTemp\\MountPoints\\{45c63495-0000-0000-0000-100000000000}\\{79DE0690-9470-4166-B9EE-4548DC416BBD}\\SVROOT\\Users\\veenstra\\Anaconda3\\envs\\dfm_tools_devenv\\Lib\\site-packages\\pip\\_internal\\resolution\\resolvelib\\__pycache__\\found_candidates.cpython-38.pyc'

Traceback (most recent call last):
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\site-packages\pip\__main__.py", line 31, in <module>
    sys.exit(_main())
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\site-packages\pip\_internal\cli\main.py", line 70, in main
    return command.main(cmd_args)
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\site-packages\pip\_internal\cli\base_command.py", line 101, in main
    return self._main(args)
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\contextlib.py", line 120, in __exit__
    next(self.gen)
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\site-packages\pip\_internal\cli\command_context.py", line 20, in main_context
    yield
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\contextlib.py", line 525, in __exit__
    raise exc_details[1]
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\site-packages\pip\_internal\utils\temp_dir.py", line 70, in tempdir_registry
    yield _tempdir_registry
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\contextlib.py", line 510, in __exit__
    if cb(*exc_details):
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\contextlib.py", line 120, in __exit__
    next(self.gen)
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\site-packages\pip\_internal\utils\temp_dir.py", line 36, in global_tempdir_manager
    _tempdir_manager = old_tempdir_manager
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\contextlib.py", line 525, in __exit__
    raise exc_details[1]
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\contextlib.py", line 510, in __exit__
    if cb(*exc_details):
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\site-packages\pip\_internal\utils\temp_dir.py", line 156, in __exit__
    self.cleanup()
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\site-packages\pip\_internal\utils\temp_dir.py", line 173, in cleanup
    rmtree(self._path)
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\site-packages\pip\_vendor\tenacity\__init__.py", line 326, in wrapped_f
    return self(f, *args, **kw)
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\site-packages\pip\_vendor\tenacity\__init__.py", line 406, in __call__
    do = self.iter(retry_state=retry_state)
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\site-packages\pip\_vendor\tenacity\__init__.py", line 362, in iter
    raise retry_exc.reraise()
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\site-packages\pip\_vendor\tenacity\__init__.py", line 195, in reraise
    raise self.last_attempt.result()
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\concurrent\futures\_base.py", line 437, in result
    return self.__get_result()
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\concurrent\futures\_base.py", line 389, in __get_result
    raise self._exception
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\site-packages\pip\_vendor\tenacity\__init__.py", line 409, in __call__
    result = fn(*args, **kwargs)
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\site-packages\pip\_internal\utils\misc.py", line 128, in rmtree
    shutil.rmtree(dir, ignore_errors=ignore_errors, onerror=rmtree_errorhandler)
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\shutil.py", line 740, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\shutil.py", line 613, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\shutil.py", line 613, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\shutil.py", line 613, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  [Previous line repeated 4 more times]
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\shutil.py", line 622, in _rmtree_unsafe
    onerror(os.rmdir, path, sys.exc_info())
  File "C:\Users\veenstra\Anaconda3\envs\dfm_tools_devenv\lib\shutil.py", line 620, in _rmtree_unsafe
    os.rmdir(path)
OSError: [WinError 145] The directory is not empty: 'C:\\SnapVolumesTemp\\MountPoints\\{45c63495-0000-0000-0000-100000000000}\\{79DE0690-9470-4166-B9EE-4548DC416BBD}\\SVROOT\\Users\\veenstra\\AppData\\Local\\Temp\\pip-req-build-cnj2ddcg\\tests\\data\\input\\e02\\c11_korte-woerden-1d\\dimr_model\\dflowfm'

As for the pip install -e .. I've checked whether the missing path exists, and it does.

[priscavdsluis]

@veenstrajelmer, the last error that you are getting: OSError: [WinError 145] The directory is not empty: 'C:\\SnapVolumesTemp\\MountPoints\\{45c63495-0000-0000-0000-100000000000}\\{79DE0690-9470-4166-B9EE-4548DC416BBD}\\SVROOT\\Users\\veenstra\\AppData\\Local\\Temp\\pip-req-build-cnj2ddcg\\tests\\data\\input\\e02\\c11_korte-woerden-1d\\dimr_model\\dflowfm' if often due to that the folder is open somewhere, do/did you perhaps have it open in your file explorer?

[veenstrajelmer]

It seems to be a WCF issue, so not something related to hydrolib-core. Therefore I will now close this issue.

[veenstrajelmer]

For other users, this issue can be solved by using the solution suggested by Prisca: git config --system core.longpaths true

In combination with enabling the long paths in Windows 10 via the Registry Editor (regedit):

• Browse in the register: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem
• set LongPathsEnabled to 1

This is only necessary when installing a non-release version like this: python -m pip install git+https://github.com/Deltares/HYDROLIB-core It is not issue when doing python -m pip install hydrolib-core