Invalid path handling test_ria_postclonecfg

[mih]

Revealed by datalad/datalad#5080

======================================================================
1541ERROR: Failure: OSError ([WinError 123] The filename, directory name, or volume label syntax is incorrect: '\\C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\datalad_temp_0e20xmuu\\ria-layout-version')
1542----------------------------------------------------------------------
1543Traceback (most recent call last):
1544  File "C:\Miniconda35\envs\test-environment\lib\site-packages\nose\failure.py", line 39, in runTest
1545    raise self.exc_val.with_traceback(self.tb)
1546  File "C:\Miniconda35\envs\test-environment\lib\site-packages\nose\loader.py", line 252, in generate
1547    for test in g():
1548  File "C:\Miniconda35\envs\test-environment\lib\site-packages\datalad\core\distributed\tests\test_clone.py", line 941, in test_ria_postclonecfg
1549    id = _postclonetest_prepare(lcl, store)
1550  File "C:\Miniconda35\envs\test-environment\lib\site-packages\datalad\tests\utils.py", line 750, in _wrap_with_tempfile
1551    return t(*(arg + (filename,)), **kw)
1552  File "C:\Miniconda35\envs\test-environment\lib\site-packages\datalad\core\distributed\tests\test_clone.py", line 904, in _postclonetest_prepare
1553    d.create_sibling_ria(upl_url, "store")
1554  File "C:\Miniconda35\envs\test-environment\lib\site-packages\datalad\distribution\dataset.py", line 503, in apply_func
1555    return f(**kwargs)
1556  File "C:\Miniconda35\envs\test-environment\lib\site-packages\datalad\interface\utils.py", line 495, in eval_func
1557    return return_func(generator_func)(*args, **kwargs)
1558  File "C:\Miniconda35\envs\test-environment\lib\site-packages\datalad\interface\utils.py", line 483, in return_func
1559    results = list(results)
1560  File "C:\Miniconda35\envs\test-environment\lib\site-packages\datalad\interface\utils.py", line 414, in generator_func
1561    allkwargs):
1562  File "C:\Miniconda35\envs\test-environment\lib\site-packages\datalad\interface\utils.py", line 562, in _process_results
1563    for res in results:
1564  File "C:\Miniconda35\envs\test-environment\lib\site-packages\datalad\distributed\create_sibling_ria.py", line 343, in __call__
1565    '1')
1566  File "C:\Miniconda35\envs\test-environment\lib\site-packages\datalad\customremotes\ria_utils.py", line 136, in create_store
1567    if io.exists(version_file):
1568  File "C:\Miniconda35\envs\test-environment\lib\site-packages\datalad\distributed\ora_remote.py", line 203, in exists
1569    return path.exists()
1570  File "C:\Miniconda35\envs\test-environment\lib\pathlib.py", line 1291, in exists
1571    self.stat()
1572  File "C:\Miniconda35\envs\test-environment\lib\pathlib.py", line 1111, in stat
1573    return self._accessor.stat(self)
1574  File "C:\Miniconda35\envs\test-environment\lib\pathlib.py", line 371, in wrapped
1575    return strfunc(str(pathobj), *args)
1576OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: '\\C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\datalad_temp_0e20xmuu\\ria-layout-version'

Possibly same as datalad/datalad#5084.

On a real win10 box this test is skipped with SKIP: Can't create symlinks on both maint and master.

[mih]

I have configured my win10 system to support symlinks and I can now replicate this error

[mih]

This seems to be a problem of the test rather than the underlying tested functionality. The test uses the patter Path(absolute windows path).as_uri() to generate a file URL. However, that doesn't work, hence we have get_local_file_url():

(Pdb) Path(store).as_uri()
'file:///C:/Users/mih/AppData/Local/Temp/datalad_temp_fgdgus62'
(Pdb) get_local_file_url(store)
'file://C/Users/mih/AppData/Local/Temp/datalad_temp_fgdgus62'
(Pdb) get_local_file_url(store, compatibility='git')
'file:///C/Users/mih/AppData/Local/Temp/datalad_temp_fgdgus62'

[mih]

With this change we get the next error:

[{'action': 'create-sibling-ria',
  'message': 'initremote failed.\n'
             'stdout: initremote store-storage \n'
             'failed\n'
             '\n'
             'stderr: git-annex: Non-absolute object tree base path '
             'configuration: '
             '\\C\\Users\\mih\\AppData\\Local\\Temp\\datalad_temp_m0zkr4q8\r\n'
             'git-annex: initremote: 1 failed\n',
  'path': 'C:\\Users\\mih\\AppData\\Local\\Temp\\datalad_temp_g7onyebf\\ds',
  'status': 'error',
  'type': 'dataset'}]

This test in ora_remote.py is wrong. It uses Path(lastsegment of URL).is_absolute(), but Path on Windows is WindowsPath, and the trailing end of a URL will never be a valid absolute windows path.

Using PurePosixPath for the test makes create-sibling-ria work on windows.

[mih]

The previous change wasn't enough to get git annex copy to work, because the store base path is used to derive all relevant locations from it. Hence stuff like an annex key location was a PurePosixPath that does not support .exists(), for example. Explicitly converting to Path when doing LocalIO makes git annex copy work.

[mih]

Next issue is that the RIA store in the test on windows has a functional annex remote, but no git remote...

[mih]

Sooo, turns out that

get_local_file_url(store, compatibility='git')
'file:///C/Users/mih/AppData/Local/Temp/datalad_temp_fgdgus62'

"works" but does do what is desired. It actually points to C:\C\Users\mih..., where as

(Pdb) get_local_file_url(store)
'file://C/Users/mih/AppData/Local/Temp/datalad_temp_fgdgus62'

Really translates to C:\Users\mih....

We have to critically examine each use of compatibility='git'. These wrong path will not lead to any failure in the tests, except when they are tested by other means that the path we generated them from. But they will still be in the wrong place!

[mih]

I will propose a fix for this in a second, but a complete solution is blocked by datalad/datalad#5134 (i.e. the tests won't pass without this problem being solved too).

[mih]

It seems all blockers are removed.

[yarikoptic]

is this still an issue?

[yarikoptic]

the test is marked as known failure

❯ git grep -B3 'def test_ria_postclonecfg'
datalad/core/distributed/tests/test_clone.py-# TODO?: make parametric again on _test_ria_postclonecfg
datalad/core/distributed/tests/test_clone.py-@known_failure_windows  # https://github.com/datalad/datalad/issues/5134
datalad/core/distributed/tests/test_clone.py-@slow  # 14 sec on travis
datalad/core/distributed/tests/test_clone.py:def test_ria_postclonecfg():

so among dozens of other similar ones... is there a point to keep this issue open?