search

datalad / datalad-next

DataLad extension for new functionality and improved user experience
https://datalad.org
Other
9 stars 10 forks source link

Test fail on appveyor after move to hatch #759

Open mih opened 2 months ago

mih commented 2 months ago

It smells like some kind of race condition, but I have no idea yet where it would come from...

test-linux

All these go away, when switching from PY3.9 Ubuntu 2204 to PY3.10 on Ubuntu2004, on a rerun test_initremote_basic_sshurl failure is back.

test-linux-crippled

All these stay, when switching from PY3.9 Ubuntu 2204 to PY3.10 on Ubuntu2004. But on a rerun test_version_check_ssh works.

test-win

All these stay, when switching from PY3.12 Win2022 to PY3.9 on Win2019. And stayed for a rerun.

test-mac

Only the last two stay when switching from PY3.12 sonoma to PY3.9 monterey. All came back after a rerun.

datalad-core-1

This is fragile and goes away on a rerun

datalad-core-4

All these went away, with no change on a rerun. And then came back on another rerun.

adswa commented 2 months ago

the ria failures are very very funky. I spent a while trying to make them fail on my local system. What ultimately did it was to set a pdb traceback in datalad:

(datalad) adina@muninn in ~/repos/datalad on git:maint!
❱ git diff
diff --git a/datalad/customremotes/tests/test_ria_utils.py b/datalad/customremotes/tests/test_ria_utils.py
index 908512316..c58eb6228 100644
--- a/datalad/customremotes/tests/test_ria_utils.py
+++ b/datalad/customremotes/tests/test_ria_utils.py
@@ -121,7 +121,7 @@ def _test_setup_ds_in_store(io_cls, io_args, store=None):

 def test_setup_ds_in_store():
-
+    import pdb; pdb.set_trace()
     _test_setup_ds_in_store(LocalIO, [])

     if on_windows:

when I then run DATALAD_TESTS_SSH=1 hatch run tests.py3.10:run --reruns 5 -v -s datalad_next/patches/tests/test_patched_ria_ora.py (having commented out the testremote invocation in this test suite, because it takes a while), the first failure is

AILED datalad_next/patches/tests/test_patched_ria_ora.py::test_version_check_ssh - ValueError: Conflicting version found at target: 2
===================================================== 1 failed, 19 passed, 5 rerun in 162.23s (0:02:42) =====================================================

When I then run the same command again, I get

FAILED datalad_next/patches/tests/test_patched_ria_ora.py::test_obtain_permission_ssh - ValueError: Conflicting version found at target: 2
FAILED datalad_next/patches/tests/test_patched_ria_ora.py::test_remote_layout_ssh - ValueError: Conflicting version found at target: 2
FAILED datalad_next/patches/tests/test_patched_ria_ora.py::test_initremote_basic_sshurl - ValueError: Conflicting version found at target: 2
FAILED datalad_next/patches/tests/test_patched_ria_ora.py::test_version_check_ssh - ValueError: Conflicting version found at target: 2
==================================================== 4 failed, 16 passed, 20 rerun in 179.22s (0:02:59) ==================================

and when I run the same command a third time, I get

FAILED datalad_next/patches/tests/test_patched_ria_ora.py::test_obtain_permission_ssh - ValueError: Conflicting version found at target: 2
FAILED datalad_next/patches/tests/test_patched_ria_ora.py::test_remote_layout_ssh - ValueError: Conflicting version found at target: 2
FAILED datalad_next/patches/tests/test_patched_ria_ora.py::test_version_check_ssh - ValueError: Conflicting version found at target: 2
==================================================== 3 failed, 17 passed, 15 rerun in 155.11s (0:02:35) ====================================================

and after running a fourth time I get

FAILED datalad_next/patches/tests/test_patched_ria_ora.py::test_obtain_permission_ssh - ValueError: Conflicting version found at target: 2
FAILED datalad_next/patches/tests/test_patched_ria_ora.py::test_setup_ds_in_store_ssh - ValueError: Conflicting version found at target: 2
FAILED datalad_next/patches/tests/test_patched_ria_ora.py::test_initremote_basic_sshurl - ValueError: Conflicting version found at target: 2
==================================================== 3 failed, 17 passed, 15 rerun in 169.63s (0:02:49) =====================================================

and the fifth run gives

FAILED datalad_next/patches/tests/test_patched_ria_ora.py::test_initremote_basic_sshurl - ValueError: Conflicting version found at target: 2
FAILED datalad_next/patches/tests/test_patched_ria_ora.py::test_version_check_ssh - ValueError: Conflicting version found at target: 2
FAILED datalad_next/patches/tests/test_patched_ria_ora.py::test_setup_ds_in_store_ssh - ValueError: Conflicting version found at target: 2
==================================================== 3 failed, 17 passed, 15 rerun in 170.48s (0:02:50) =====================================================

absolutely no idea what is happening.

Its probably worthwhile to note that the test suite doesn't end by itself but needs a CTRL-C after the summary (likely because of the breakpoint in datalad, although that breakpoint never brings me into a debugger during the test execution)

mih commented 2 months ago

I suspect that these tests are not cleanly implemented and suffer from and/or cause residuals in the environment that makes them order-dependent. This is, from my POV, the main change from the previous test setup to the new one. The tests run in random order now (each time). I think this is a good thing in and of itself.