search

sardana-org / sardana

Moved to GitLab: https://gitlab.com/sardana-org/sardana
39 stars 52 forks source link

NXscanH5_FileRecorder VDS on Windows does not work #1562

Open reszelaz opened 3 years ago

reszelaz commented 3 years ago

This still requires further investigation. It looks like it has to do with the windows style paths and the URI regular expression we use when recognizing the value references.

For the moment I just paste the output of the test execution on Windows.

PS Y:\> pytest src/sardana/macroserver/recorders/test/test_h5storage.py::test_VDS
================================================= test session starts =================================================
platform win32 -- Python 3.8.9, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
rootdir: Y:\, configfile: pytest.ini
plugins: forked-1.3.0, xdist-2.2.1
collected 1 item

src\sardana\macroserver\recorders\test\test_h5storage.py F                                                       [100%]

====================================================== FAILURES =======================================================
______________________________________________________ test_VDS _______________________________________________________

recorder = <sardana.macroserver.recorders.h5storage.NXscanH5_FileRecorder object at 0x000001E2816C3FD0>

    @pytest.mark.skipif(not hasattr(h5py, "VirtualLayout"),
                        reason="VDS not available in this version of h5py")
    def test_VDS(recorder):
        """Test creation of VDS when channel reports URIs (str) of h5file
        scheme in a simulated sardana scan (3 points).
        """
        nb_records = 3
        # create partial files
        part_file_name_pattern = "test_vds_part{0}.h5"
        part_file_paths = []
        for i in range(nb_records):
            path = os.path.join(os.path.dirname(recorder.filename),
                                part_file_name_pattern.format(i))
            part_file_paths.append(path)
            part_file = h5py.File(path, "w")
            img = numpy.array([[i, i], [i, i]])
            dataset = "dataset"
            part_file.create_dataset(dataset, data=img)
            part_file.flush()
            part_file.close()
        try:
            # create description of channel data
            data_desc = [
                ColumnDesc(name=COL1_NAME, label=COL1_NAME, dtype="float64",
                           shape=(2, 2), value_ref_enabled=True)
            ]
            environ = ENV.copy()
            environ["datadesc"] = data_desc

            # simulate sardana scan
            environ["starttime"] = datetime.now()
            record_list = RecordList(environ)
            recorder._startRecordList(record_list)
            for i in range(nb_records):
                ref = "h5file://" + part_file_paths[i] + "::" + dataset
                record = Record({COL1_NAME: ref}, i)
                recorder._writeRecord(record)
            environ["endtime"] = datetime.now()
            recorder._endRecordList(record_list)

            # assert if reading datasets from the sardana file access to the
            # dataset of the partial files
            file_ = h5py.File(recorder.filename)
            for i in range(nb_records):
                expected_img = numpy.array([[i, i], [i, i]])
                img = file_["entry0"]["measurement"][COL1_NAME][i]
                msg = "VDS extracted image does not match"
                # TODO: check if this assert works well
>               numpy.testing.assert_array_equal(img, expected_img, msg)
E               AssertionError:
E               Arrays are not equal
E               VDS extracted image does not match
E               Mismatched elements: 1 / 1 (100%)
E                x: array(b'h5file://C:\\Users\\IEUser\\AppData\\Local\\Temp\\pytest-of-IEUser\\pytest-45\\test_VDS0\\test_vds_part0.h5::dataset',
E                     dtype='|S107')
E                y: array([[0, 0],
E                      [0, 0]])

src\sardana\macroserver\recorders\test\test_h5storage.py:212: AssertionError
------------------------------------------------ Captured stderr call -------------------------------------------------
--- Logging error ---
Traceback (most recent call last):
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\logging\__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
OSError: [WinError 6] The handle is invalid
Call stack:
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\IEUser\AppData\Local\Programs\Python\Python38\Scripts\pytest.exe\__main__.py", line 7, in <module>
    sys.exit(console_main())
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\_pytest\config\__init__.py", line 185, in console_main
    code = main()
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\_pytest\config\__init__.py", line 162, in main
    ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\pluggy\hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\pluggy\manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\pluggy\callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\_pytest\main.py", line 316, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\_pytest\main.py", line 269, in wrap_session
    session.exitstatus = doit(config, session) or 0
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\_pytest\main.py", line 323, in _main
    config.hook.pytest_runtestloop(session=session)
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\pluggy\hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\pluggy\manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\pluggy\callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\_pytest\main.py", line 348, in pytest_runtestloop
    item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\pluggy\hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\pluggy\manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\pluggy\callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\_pytest\runner.py", line 109, in pytest_runtest_protocol
    runtestprotocol(item, nextitem=nextitem)
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\_pytest\runner.py", line 126, in runtestprotocol
    reports.append(call_and_report(item, "call", log))
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\_pytest\runner.py", line 215, in call_and_report
    call = call_runtest_hook(item, when, **kwds)
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\_pytest\runner.py", line 254, in call_runtest_hook
    return CallInfo.from_call(
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\_pytest\runner.py", line 311, in from_call
    result: Optional[TResult] = func()
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\_pytest\runner.py", line 255, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\pluggy\hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\pluggy\manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\pluggy\callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\_pytest\runner.py", line 162, in pytest_runtest_call
    item.runtest()
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\_pytest\python.py", line 1641, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\pluggy\hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\pluggy\manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\pluggy\callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\_pytest\python.py", line 183, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "Y:\src\sardana\macroserver\recorders\test\test_h5storage.py", line 202, in test_VDS
    recorder._endRecordList(record_list)
  File "Y:\src\sardana\macroserver\recorders\h5storage.py", line 384, in _endRecordList
    self.warning(msg)
  File "\\vboxsvr\taurus\lib\taurus\core\util\log.py", line 825, in warning
    self.log_obj.warning(msg, *args, **kw)
Message: 'Unsupported reference h5file://C:\\Users\\IEUser\\AppData\\Local\\Temp\\pytest-of-IEUser\\pytest-45\\test_VDS0\\test_vds_part0.h5::dataset'
Arguments: ()
-------------------------------------------------- Captured log call --------------------------------------------------
WARNING  NXscanH5_FileRecorder:log.py:825 Unsupported reference h5file://C:\Users\IEUser\AppData\Local\Temp\pytest-of-IEUser\pytest-45\test_VDS0\test_vds_part0.h5::dataset
================================================== warnings summary ===================================================
c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\future\standard_library\__init__.py:65
  c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\future\standard_library\__init__.py:65: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\tango\utils.py:181
  c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\tango\utils.py:181: DeprecationWarning: `np.str` is a deprecated alias for the builtin `str`. To silence this warning, use `str` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.str_` here.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    CmdArgType.DevString: numpy.str,

src/sardana/macroserver/recorders/test/test_h5storage.py::test_VDS
  c:\users\ieuser\appdata\local\programs\python\python38\lib\site-packages\numpy\testing\_private\utils.py:790: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
    val = comparison(x, y)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=============================================== short test summary info ===============================================
FAILED src/sardana/macroserver/recorders/test/test_h5storage.py::test_VDS - AssertionError:
============================================ 1 failed, 3 warnings in 0.94s ============================================
MainThread     WARNING  2021-04-16 13:52:29,444 NXscanH5_FileRecorder: Unsupported reference h5file://C:\Users\IEUser\AppData\Local\Temp\pytest-of-IEUser\pytest-45\test_VDS0\test_vds_part0.h5::dataset
PS Y:\>
cpascual commented 3 years ago

I set the size:M label because it needs to be debugged on a windows environment (and the impact:low because it affects only Win, which is not majoritary among sardana users). But for someone with Windows as the primary Platform, it would be (size:S and impact:medium)