nomad-coe / nomad-schema-plugin-example

This is an example for NOMAD schema plugins. It should be forked to create actual plugins.
Apache License 2.0
14 stars 17 forks source link

Test not working - nomad-lab install issue #2

Closed ericpre closed 1 year ago

ericpre commented 1 year ago

I have installed this example and tried to run the tests but I have the following error:

============================================================================= test session starts ==============================================================================
platform win32 -- Python 3.9.16, pytest-7.4.0, pluggy-1.2.0 -- C:\Users\M0041User\mambaforge\envs\nomad\python.exe
cachedir: .pytest_cache
rootdir: nomad-schema-plugin-example
plugins: anyio-3.7.1
collected 0 items / 1 error

==================================================================================== ERRORS ====================================================================================
____________________________________________________________________ ERROR collecting tests/test_schema.py _____________________________________________________________________
..\..\..\..\mambaforge\envs\nomad\lib\site-packages\_pytest\runner.py:341: in from_call
    result: Optional[TResult] = func()
..\..\..\..\mambaforge\envs\nomad\lib\site-packages\_pytest\runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
..\..\..\..\mambaforge\envs\nomad\lib\site-packages\_pytest\python.py:531: in collect
    self._inject_setup_module_fixture()
..\..\..\..\mambaforge\envs\nomad\lib\site-packages\_pytest\python.py:545: in _inject_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
..\..\..\..\mambaforge\envs\nomad\lib\site-packages\_pytest\python.py:310: in obj
    self._obj = obj = self._getobj()
..\..\..\..\mambaforge\envs\nomad\lib\site-packages\_pytest\python.py:528: in _getobj
    return self._importtestmodule()
..\..\..\..\mambaforge\envs\nomad\lib\site-packages\_pytest\python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
..\..\..\..\mambaforge\envs\nomad\lib\site-packages\_pytest\pathlib.py:565: in import_path
    importlib.import_module(module_name)
..\..\..\..\mambaforge\envs\nomad\lib\importlib\__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1030: in _gcd_import
    ???
<frozen importlib._bootstrap>:1007: in _find_and_load
    ???
<frozen importlib._bootstrap>:986: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:680: in _load_unlocked
    ???
..\..\..\..\mambaforge\envs\nomad\lib\site-packages\_pytest\assertion\rewrite.py:178: in exec_module
    exec(co, module.__dict__)
tests\test_schema.py:2: in <module>
    from nomad.client import parse, normalize_all
..\..\..\..\mambaforge\envs\nomad\lib\site-packages\nomad\client\__init__.py:19: in <module>
    from .archive import ArchiveQuery
..\..\..\..\mambaforge\envs\nomad\lib\site-packages\nomad\client\archive.py:28: in <module>
    from nomad import config, metainfo as mi
..\..\..\..\mambaforge\envs\nomad\lib\site-packages\nomad\config\__init__.py:48: in <module>
    from .plugins import Plugins, Plugin, Parser, Schema
..\..\..\..\mambaforge\envs\nomad\lib\site-packages\nomad\config\plugins.py:88: in <module>
    class Schema(PythonPluginBase):
pydantic\main.py:198: in pydantic.main.ModelMetaclass.__new__
    ???
pydantic\fields.py:506: in pydantic.fields.ModelField.infer
    ???
pydantic\fields.py:436: in pydantic.fields.ModelField.__init__
    ???
pydantic\fields.py:552: in pydantic.fields.ModelField.prepare
    ???
pydantic\fields.py:668: in pydantic.fields.ModelField._type_analysis
    ???
..\..\..\..\mambaforge\envs\nomad\lib\typing.py:852: in __subclasscheck__
    return issubclass(cls, self.__origin__)
E   TypeError: issubclass() arg 1 must be a class
=============================================================================== warnings summary ===============================================================================
..\..\..\..\mambaforge\envs\nomad\lib\site-packages\nomad\config\models.py:27
  C:\Users\M0041User\mambaforge\envs\nomad\lib\site-packages\nomad\config\models.py:27: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import get_distribution, DistributionNotFound

..\..\..\..\mambaforge\envs\nomad\lib\site-packages\pkg_resources\__init__.py:2871
  C:\Users\M0041User\mambaforge\envs\nomad\lib\site-packages\pkg_resources\__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================================================================== short test summary info ============================================================================
ERROR tests/test_schema.py - TypeError: issubclass() arg 1 must be a class
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================================================= 2 warnings, 1 error in 0.90s =========================================================================

pip list:

Package             Version
------------------- --------
aniso8601           7.0.0
anyio               3.7.1
ase                 3.19.0
asttokens           2.2.1
backcall            0.2.0
cachetools          4.2.4
certifi             2023.5.7
cffi                1.15.1
charset-normalizer  3.1.0
click               8.1.4
cloudpickle         2.2.1
colorama            0.4.6
comm                0.1.3
contourpy           1.1.0
cryptography        41.0.1
cycler              0.11.0
debugpy             1.6.7
decorator           5.1.1
docstring-parser    0.12
ecdsa               0.18.0
elasticsearch       7.17.9
elasticsearch-dsl   7.4.0
exceptiongroup      1.1.2
executing           1.2.0
fonttools           4.40.0
h11                 0.14.0
h5grove             1.3.0
h5py                3.9.0
httpcore            0.16.3
httpx               0.23.3
idna                3.4
importlib-metadata  6.7.0
importlib-resources 5.12.0
iniconfig           2.0.0
ipykernel           6.24.0
ipython             8.14.0
jedi                0.18.2
jmespath            0.10.0
jupyter_client      8.3.0
jupyter_core        5.3.1
kiwisolver          1.4.4
lxml                4.9.3
matplotlib          3.7.2
matplotlib-inline   0.1.6
memoization         0.4.0
nest-asyncio        1.5.6
nomad-lab           1.2.0rc0
nptyping            1.4.4
numpy               1.25.0
orjson              3.6.0
packaging           23.1
pandas              1.3.5
parso               0.8.3
pickleshare         0.7.5
Pillow              10.0.0
Pint                0.17
pip                 23.1.2
platformdirs        3.8.1
pluggy              1.2.0
prompt-toolkit      3.0.39
psutil              5.9.5
pure-eval           0.2.2
pyasn1              0.5.0
pyasn1-modules      0.3.0
pycparser           2.21
pydantic            1.10.5
Pygments            2.15.1
pyparsing           3.0.9
pytest              7.4.0
python-dateutil     2.8.2
python-jose         3.3.0
python-keycloak     0.26.1
python-magic        0.4.24
pytz                2022.7.1
pywin32             306
PyYAML              6.0
pyzmq               25.1.0
requests            2.28.2
rfc3161ng           2.1.3
rfc3986             1.5.0
rsa                 4.9
scipy               1.11.1
setuptools          68.0.0
six                 1.16.0
sniffio             1.3.0
spyder-kernels      2.4.4
stack-data          0.6.2
tifffile            2023.7.4
tomli               2.0.1
tornado             6.3.2
traitlets           5.9.0
typing_extensions   4.7.1
typish              1.9.3
urllib3             1.26.16
wcwidth             0.2.6
wheel               0.40.0
wrapt               1.12.1
zipp                3.15.0

Any idea what causes the issue? Thanks!

markus1978 commented 1 year ago

I just tried it myself and could reproduce it. You can quickly fix it with

pip install pydantic==1.10.9

Ignore the dependency error.

I will try to look into this in more detail. We are planning a new release for the nomad-lab package soon and a proper fix will be included.

ericpre commented 1 year ago

Thanks, it helps to get one step further but I am hitting another issue with magic library being missing - the python wrapper is installed but not the C library. As this is no build for windows on conda-forge, it will most likely not be easy to install on windows.

==================================================================================== test session starts =====================================================================================
platform win32 -- Python 3.9.16, pytest-7.4.0, pluggy-1.2.0 -- C:\Users\M0041User\mambaforge\envs\nomad\python.exe
cachedir: .pytest_cache
rootdir: nomad-schema-plugin-example
plugins: anyio-3.7.1
collected 1 item

tests/test_schema.py::test_schema FAILED

========================================================================================== FAILURES ==========================================================================================
________________________________________________________________________________________ test_schema _________________________________________________________________________________________

    def test_schema():
        test_file = os.path.join(os.path.dirname(__file__), 'data', 'test.archive.yaml')
>       entry_archive = parse(test_file)[0]

tests\test_schema.py:6:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
..\..\..\..\mambaforge\envs\nomad\lib\site-packages\nomad\client\processing.py:50: in parse
    parser, mainfile_keys = parsers.match_parser(mainfile_path, strict=strict, parser_name=parser_name)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

mainfile_path = 'C:\\Users\\M0041User\\OneDrive - UK Atomic Energy Authority\\MBU Corrosion database\\NOMAD_plugin\\nomad-schema-plugin-example\\tests\\data\\test.archive.yaml', strict = True
parser_name = None

    def match_parser(mainfile_path: str, strict=True, parser_name: str = None) -> Tuple[Parser, List[str]]:
        '''
        Performs parser matching. This means it take the given mainfile and potentially
        opens it with the given callback and tries to identify a parser that can parse
        the file.

        This is determined by filename (e.g. *.out), mime type (e.g. text/*, application/xml),
        and beginning file contents.

        Arguments:
            mainfile_path: Path to the mainfile
            strict: Only match strict parsers, e.g. no artificial parsers for missing or empty entries.
            parser_name: Optional, to force the matching to test only a specific parser

        Returns:
            A tuple (`parser`, `mainfile_keys`). The `parser` is the matched parser, and
            `mainfile_keys` defines the keys to use for child entries, if any. If there are
            no child entries, `mainfile_keys` will be None. If no parser matches, we return
            (None, None).
        '''
        mainfile = os.path.basename(mainfile_path)
        if mainfile.startswith('.') or mainfile.startswith('~'):
            return None, None

        with open(mainfile_path, 'rb') as f:
>           compression, open_compressed = _compressions.get(f.read(3), (None, open))
E           NameError: name '_compressions' is not defined

..\..\..\..\mambaforge\envs\nomad\lib\site-packages\nomad\parsing\parsers.py:77: NameError
====================================================================================== warnings summary ======================================================================================
..\..\..\..\mambaforge\envs\nomad\lib\site-packages\nomad\config\models.py:26
  C:\Users\M0041User\mambaforge\envs\nomad\lib\site-packages\nomad\config\models.py:26: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import get_distribution, DistributionNotFound

..\..\..\..\mambaforge\envs\nomad\lib\site-packages\pkg_resources\__init__.py:2871
  C:\Users\M0041User\mambaforge\envs\nomad\lib\site-packages\pkg_resources\__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================== short test summary info ===================================================================================
FAILED tests/test_schema.py::test_schema - NameError: name '_compressions' is not defined
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=============================================================================== 1 failed, 2 warnings in 5.51s ================================================================================
markus1978 commented 1 year ago

There is this extra package with the needed .dll that is described in the python-magic docs. https://pypi.org/project/python-magic/

I can't test it though. The only experience I have is with people using WSL and here libmagic seemed to be present.

ericpre commented 1 year ago

Sorry, I missed the relevant part of the installation of python-magic for windows! :man_facepalming: It is possible to install the binaries using pip install python-magic-bin and it works fine! I tend to be bit suspicious when installing binaries from pypi but it seems works fine. I can run the tests now! :)