search

DataLab-Platform / DataLab

Open-source Platform for Scientific and Technical Data Processing and Visualization
https://datalab-platform.com
BSD 3-Clause "New" or "Revised" License
36 stars 4 forks source link

Debian packaging for V0.16.2 #84

Closed rolandmas closed 2 months ago

rolandmas commented 2 months ago

Describe the bug The "signal" symbol seems to be conflicting with the standard library's "signal" module.

To Reproduce Running the testsuite in a clean Debian unstable chroot, as part of building the .deb package.

Expected behavior Testsuite proceeds and succeeds :-)

Screenshots Here's the part where the package build runs the testsuite:

make[1]: Entering directory '/build/datalab-0.16.1'
set -e; for v in $(py3versions -vs) ; do HOME=$(mktemp -d);trap "rm -rf $HOME" EXIT;cd /build/datalab-0.16.1/.pybuild/cpython3_${v}_datalab/build ; PYTHONPATH=. xvfb-run python$v ./cdl/tests/all_tests.py && rm -r $HOME ; done
QStandardPaths: error creating runtime directory '/run/user/1001' (No such file or directory)
QStandardPaths: error creating runtime directory '/run/user/1001' (No such file or directory)
Qt: Session management error: None of the authentication protocols specified are supported
Traceback (most recent call last):
  File "/build/datalab-0.16.1/.pybuild/cpython3_3.11_datalab/build/cdl/tests/features/tour_unit_test.py", line 22, in <module>
    test_tour()
  File "/build/datalab-0.16.1/.pybuild/cpython3_3.11_datalab/build/cdl/tests/features/tour_unit_test.py", line 17, in test_tour
    with cdltest_app_context() as win:
  File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
   File "/build/datalab-0.16.1/.pybuild/cpython3_3.11_datalab/build/cdl/tests/__init__.py", line 56, in cdltest_app_context
    win = CDLMainWindow(console=console)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/build/datalab-0.16.1/.pybuild/cpython3_3.11_datalab/build/cdl/core/gui/main.py", line 193, in __init__
    self.setup(console)
  File "/build/datalab-0.16.1/.pybuild/cpython3_3.11_datalab/build/cdl/core/gui/main.py", line 699, in setup
    self.__add_signal_image_panels()
  File "/build/datalab-0.16.1/.pybuild/cpython3_3.11_datalab/build/cdl/core/gui/main.py", line 909, in __add_signal_image_panels
    cdock = self.__add_dockwidget(self.__add_signal_panel(), title=_("Signal View"))
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/build/datalab-0.16.1/.pybuild/cpython3_3.11_datalab/build/cdl/core/gui/main.py", line 863, in __add_signal_panel
    self.signalpanel = signal.SignalPanel(self, dpw, self.signalpanel_toolbar)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/build/datalab-0.16.1/.pybuild/cpython3_3.11_datalab/build/cdl/core/gui/panel/signal.py", line 75, in __init__
    self.processor = SignalProcessor(self, dockableplotwidget.plotwidget)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/build/datalab-0.16.1/.pybuild/cpython3_3.11_datalab/build/cdl/core/gui/processor/base.py", line 176, in __init__
    self.set_process_isolation_enabled(Conf.main.process_isolation_enabled.get())
  File "/build/datalab-0.16.1/.pybuild/cpython3_3.11_datalab/build/cdl/core/gui/processor/base.py", line 193, in set_process_isolation_enabled
    self.worker.create_pool()
  File "/build/datalab-0.16.1/.pybuild/cpython3_3.11_datalab/build/cdl/core/gui/processor/base.py", line 95, in create_pool
    POOL = Pool(processes=1)  # pylint: disable=not-callable,consider-using-with
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 191, in __init__
    self._setup_queues()
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 346, in _setup_queues
    self._inqueue = self._ctx.SimpleQueue()
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/context.py", line 113, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/queues.py", line 341, in __init__
    self._rlock = ctx.Lock()
                  ^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/synchronize.py", line 169, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/usr/lib/python3.11/multiprocessing/synchronize.py", line 79, in __init__
    from .resource_tracker import register
  File "/usr/lib/python3.11/multiprocessing/resource_tracker.py", line 30, in <module>
    _IGNORED_SIGNALS = (signal.SIGINT, signal.SIGTERM)
                        ^^^^^^^^^^^^^
AttributeError: module 'signal' has no attribute 'SIGINT'

Installation information Building from the 0.16.1 release, no patches.

Additional context I suspect that the "signal" module that "multiprocessing" tries to use is actually one provided by Datalab rather than the one in the Python standard library. So either something's not tight enough when isolating the scope of the imports, or maybe it would be better to rename the "signal" submodules in Datalab.

PierreRaybaut commented 2 months ago

Hi @rolandmas, thanks for the feedback!

I suspect that this AttributeError: module 'signal' has no attribute 'SIGINT' is only a side effect: this is not what's causing the crash. Indeed, I never saw such name conflict on any of the test platforms (Python 3.8 to 3.12, Centos, Ubuntu, Windows - so quite a lot).

I think that we should begin by fixing this part of the issue:

QStandardPaths: error creating runtime directory '/run/user/1001' (No such file or directory)
QStandardPaths: error creating runtime directory '/run/user/1001' (No such file or directory)
Qt: Session management error: None of the authentication protocols specified are supported

Apparently, this is caused by the fact that the XDG_RUNTIME_DIR environment variable is set to a problematic directory (here /run/user/1001): nonexistent or with wrong permissions?

PierreRaybaut commented 2 months ago

BTW, the "cdl\tests\features\tour_unit_test.py" test is not the first test of the DataLab test suite. Would it be possible to copy/paste here the full log to better understand what's going on?

Otherwise, I will need to be able to reproduce the build to debug, like we did on the PlotPy Issue.

PierreRaybaut commented 2 months ago

I'm ready to rename the module into signals to avoid this conflict because I think that it's masking the real issue here. But I'm quite convinced that this won't solve the issue - but, it may help understanding it and solving it, so...

PierreRaybaut commented 2 months ago

However, the impact of the renaming is quite significant (for consistency, a lot of modules would be renamed as well), so if you would help me diagnose the problem, it would be great.

rolandmas commented 2 months ago

Thanks for the XDG_RUNTIME_DIR idea, I'll investigate it before any other and keep you informed.

PierreRaybaut commented 2 months ago

Ok, thanks @rolandmas .

BTW, I think that the "signal" module that should be renamed just for testing purpose is this one: "cdl/tests/features/signal". You may rename this folder without precaution (because modules inside of it are not called from elsewjere in the code base), just to see what happens.

PierreRaybaut commented 2 months ago

My previous comment may still be relevant.

However, in the meantime, I though about one thing that might be the cause. The test suite is configured for pytest thanks to the conftest.py file which is in the cdl/tests directory. If this configuration is not taken into account, then it could lead to ignore the fact that it forces pytest to reset the working directory after executing each test. Then, if the working directory is cdl/tests/features when executing tour_unit_test.py, it would explain the signal module name conflict because one of the subpackage in this folder is precisely named signal.

To check if conftest.py has been taken into account, there is an easy way:

  1. If conftest.py is loaded by pytest, you would see this header:
============================================================================================= test session starts ==============================================================================================
platform win32 -- Python 3.11.6, pytest-8.0.2, pluggy-1.4.0
DataLab 0.16.2 [Plugins: Données de test 1.0.0]
guidata 3.5.0, PlotPy 2.3.2
PythonQwt 0.12.1, PyQt5 5.15.10 [Qt version: 5.15.2]
NumPy 1.26.2, SciPy 1.11.4, h5py 3.10.0, scikit-image 0.22.0, OpenCV 4.8.1
rootdir: C:\Dev\DataLab
configfile: pyproject.toml
collected 206 items
  1. If it is not taken into account, you would see the standard header:
============================================================================================= test session starts ==============================================================================================
platform win32 -- Python 3.11.6, pytest-8.0.2, pluggy-1.4.0
rootdir: C:\Dev\DataLab
configfile: pyproject.toml
collected 206 items

If what you see is described in point 2., then you should try to add the folder name after the pytest command (e.g., pytest cdl/tests).

PierreRaybaut commented 2 months ago

FYI, I've renamed the submodule 'signal' to 'signals' in DataLab V0.16.2, so that this specific error will no longer occur (even though I suspect that it's not the real problem here).

rolandmas commented 2 months ago

I haven't seen the "signal" problem yet (build is still running), but now I get the following error: ImportError: cannot import name 'AboutTool' from partially initialized module 'plotpy.tools' (most likely due to a circular import) (/usr/lib/python3/dist-packages/plotpy/tools/init.py) (With plotpy 2.4.0 and datalab 0.16.2)

PierreRaybaut commented 2 months ago

This may (again) be a working directory issue, because some modules are not meant to be executed from some directory (it's sometimes difficult to cover all the conflict import combinations).

So, without any execution context, I can't tell you more about this.

PierreRaybaut commented 2 months ago

In other words, a copy/paste of the original log would certainly help.

PierreRaybaut commented 2 months ago

I'm in the verge of releasing a new bugfix version (V0.16.2). So now would really be the time to fix all those remaining issues once and for all: I'm available for assisting you, please do not hesitate...

rolandmas commented 2 months ago

Are you on IRC or Matrix by any chance?

rolandmas commented 2 months ago

Current build (in progress):

   debian/rules override_dh_auto_test
make[1]: Entering directory '/build/datalab-0.16.2'
set -e; for v in $(py3versions -vs) ; do HOME=$(mktemp -d);trap "rm -rf $HOME" EXIT;cd /build/datalab-0.16.2/.pybuild/cpython3_${v}_datalab/build ; PYTHONPATH=. XDG_RUNTIME_DIR=$HOME xvfb-run python$v ./cdl/tests/all_tests.py && rm -r $HOME ; done
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Traceback (most recent call last):
  File "/build/datalab-0.16.2/.pybuild/cpython3_3.11_datalab/build/cdl/tests/features/images/imagetools_app_test.py", line 16, in <module>
    from plotpy.tools import CrossSectionTool
  File "/usr/lib/python3/dist-packages/plotpy/tools/__init__.py", line 24, in <module>
    from .cross_section import (
  File "/usr/lib/python3/dist-packages/plotpy/tools/cross_section.py", line 20, in <module>
    from plotpy.tools.image import update_image_tool_status
  File "/usr/lib/python3/dist-packages/plotpy/tools/image.py", line 37, in <module>
    from plotpy.plot import BasePlot
  File "/usr/lib/python3/dist-packages/plotpy/plot/__init__.py", line 7, in <module>
    from .manager import PlotManager
  File "/usr/lib/python3/dist-packages/plotpy/plot/manager.py", line 16, in <module>
    from plotpy.tools import (
ImportError: cannot import name 'AboutTool' from partially initialized module 'plotpy.tools' (most likely due to a circular import) (/usr/lib/python3/dist-packages/plotpy/tools/__init__.py)
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
Qt: Session management error: None of the authentication protocols specified are supported
[…]
rolandmas commented 2 months ago

(Note the lack of "QStandardPaths: error creating runtime directory", which went away with XDG_RUNTIME_DIR=$HOME)

PierreRaybaut commented 2 months ago

I've read that the Qt: Session management error: None of the authentication protocols specified are supported is related to the SESSION_MANAGER environment variable. This variable has to be unset apparently (see here for example).

PierreRaybaut commented 2 months ago

By the way, I see that you do not run tests using the official test suite which is based on pytest (since a few versions back). Could you please replace python ./cdl/tests/all_tests.py by pytest?

PierreRaybaut commented 2 months ago

It could be interesting that you take a look at the test script for Ubuntu's GitHub build action... for inspiration?

PierreRaybaut commented 2 months ago

All the issues above were fixed thanks to further discussions in a private chat room.

Here is the final version of the command line executing tests in the context of the Debian package build process:

set -e
for v in $(py3versions -vs); do 
    HOME=$(mktemp -d)
    trap "rm -rf $HOME" EXIT
    BUILD_DIR=/build/datalab-0.16.2/.pybuild/cpython3_${v}_datalab/build
    cd $BUILD_DIR
    unset SESSION_MANAGER
    PYTHONPATH=$BUILD_DIR
    XDG_RUNTIME_DIR=$HOME
    xvfb-run
    python$v -m pytest
    rm -r $HOME
done

With those modifications, only one test was still failing: cdl/tests/features/common/reorder_app_test.py:

=================================== FAILURES ===================================
_________________________________ test_reorder _________________________________

    def test_reorder():
        """Run signals/images reorder test scenario"""
        with cdl_app_context(exec_loop=True):
            win = app.create(h5files=get_test_fnames("reorder*"))
            panel = win.signalpanel
            view, model = panel.objview, panel.objmodel

            # Select multiple signals
            objs = [model.get_object_from_number(idx) for idx in (2, 4, 5, 9)]
            view.select_objects(objs)
            # Move up
            view.move_up()
            # Check that the order is correct (note: objects 4 and 5 are not affected
            # by the move up action because they are moved up from the top of their group
            # to the bottom of the previous group)
            assert [obj.number for obj in objs] == [1, 4, 5, 8]
            # Move down
            view.move_down()
            # Check that the order is correct (note: objects 4 and 5 are not affected
            # by the move down action because they are moved down from the bottom of their
            # group to the top of the next group)
            assert [obj.number for obj in objs] == [2, 4, 5, 9]

            # Select multiple groups
            groups = [model.get_group_from_number(idx) for idx in (2, 3)]
            view.select_groups(groups)
            # Move up
            view.move_up()
            assert [group.number for group in groups] == [1, 2]
            # Move down
            view.move_down()
            assert [group.number for group in groups] == [2, 3]

            # Testing other methods of the `ObjectModel` class in unattended mode only
            if execenv.unattended:
                # Get group
                group = model.get_group_from_number(2)
                assert group.number == 2
                # Get the same group from its uuid
                group = model.get_group(group.uuid)
                assert group.number == 2
                group = model.get_object_or_group(group.uuid)
                assert group.number == 2
                # Remove group
                model.remove_group(group)
>               assert len(model.get_groups()) == 2
E               AssertionError: assert 5 == 2
E                +  where 5 = len([<cdl.core.gui.objectmodel.ObjectGroup object at 0x7f5d3730ef10>, <cdl.core.gui.objectmodel.ObjectGroup object at 0x7f...gui.objectmodel.ObjectGroup object at 0x7f5d34b0ed10>, <cdl.core.gui.objectmodel.ObjectGroup object at 0x7f5d3730df50>])
E                +    where [<cdl.core.gui.objectmodel.ObjectGroup object at 0x7f5d3730ef10>, <cdl.core.gui.objectmodel.ObjectGroup object at 0x7f...gui.objectmodel.ObjectGroup object at 0x7f5d34b0ed10>, <cdl.core.gui.objectmodel.ObjectGroup object at 0x7f5d3730df50>] = <bound method ObjectModel.get_groups of {'861eeb82-b1e1-4fdd-b1b4-70d5388dc4ac': <cdl.core.model.signal.SignalObj obje...t 0x7f5d4eb7a9d0>, '9658c18d-5cf5-4298-8ff6-2beb4926656c': <cdl.core.model.signal.SignalObj object at 0x7f5d4eb19b50>}>()
E                +      where <bound method ObjectModel.get_groups of {'861eeb82-b1e1-4fdd-b1b4-70d5388dc4ac': <cdl.core.model.signal.SignalObj obje...t 0x7f5d4eb7a9d0>, '9658c18d-5cf5-4298-8ff6-2beb4926656c': <cdl.core.model.signal.SignalObj object at 0x7f5d4eb19b50>}> = {'861eeb82-b1e1-4fdd-b1b4-70d5388dc4ac': <cdl.core.model.signal.SignalObj object at 0x7f5d37649a90>, '292232e9-2bd8-4c...at 0x7f5d4eb7a9d0>, '9658c18d-5cf5-4298-8ff6-2beb4926656c': <cdl.core.model.signal.SignalObj object at 0x7f5d4eb19b50>}.get_groups

In V0.16.2, a workaround was implemented so that this test is now OK: instead of checking the absolute number of groups after removing one (assert len(model.get_groups()) == 2), we just check that the number of groups has decreased of one unit: assert len(model.get_groups()) == nb_groups - 1.

Here is the associated patch:

--- a/cdl/tests/features/common/reorder_app_test.py
+++ b/cdl/tests/features/common/reorder_app_test.py
@@ -65,8 +65,9 @@ def test_reorder():
             group = model.get_object_or_group(group.uuid)
             assert group.number == 2
             # Remove group
+            n_groups = len(model.get_groups())
             model.remove_group(group)
-            assert len(model.get_groups()) == 2
+            assert len(model.get_groups()) == n_groups - 1

 if __name__ == "__main__":
PierreRaybaut commented 2 months ago

At first, writing the previous comment seemed inessential... but apparently it was a good idea to do so because it made me realized what was the real issue regarding the reorder_app_test failure mentioned above (see #85): it is now fixed!