search

Curts0 / PyTabular

Connect to Tabular Models via Python
https://curts0.github.io/PyTabular/
MIT License
67 stars 11 forks source link

Linux Compatibility - Currently Windows Only #36

Open Bugzey opened 1 year ago

Bugzey commented 1 year ago

Describe the bug Running the provided test suite produces an error with importing clr with Python 3.10 and Manjaro Linux. If we try importing the package from an interactive Python shell, the error is more specific about the clr package not having an AddReference attribute. Logs attached

To Reproduce

  1. Create and activate Python a virtual environment
  2. Install the package to the virtual environment
  3. run pytest
  4. Pytest gets interrupted during collection

Expected behavior Tests load correctly and start running.

Screenshots (None)

Desktop (please complete the following information):

Additional context Seeing that the module comes bundled with .dll files led me to believe that this package was never meant to run under anything other than Microsoft Windows even though any automation of data processes would likely be carried out by data engineers, developed and tested under Linux or the Windows Subsystem for Linux (WSL), and deployed to some cloud service such as functions, app services, containers or virtual machines. All of those predominantly run Linux. The current package should either try to support and test on such systems or explicitly state that they are unsupported in the README file.

Pytest log:

=================================================================================== test session starts ====================================================================================
platform linux -- Python 3.10.6, pytest-7.1.1, pluggy-1.0.0 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /home/radi/Code/PyTabular, configfile: pyproject.toml
plugins: anyio-3.6.1
collected 0 items / 1 error                                                                                                                                                                

========================================================================================== ERRORS ==========================================================================================
__________________________________________________________________________ ERROR collecting test/test_tabular.py ___________________________________________________________________________
ImportError while importing test module '/home/radi/Code/PyTabular/test/test_tabular.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../.local/lib/python3.10/site-packages/_pytest/python.py:608: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
../../.local/lib/python3.10/site-packages/_pytest/pathlib.py:533: in import_path
    importlib.import_module(module_name)
/usr/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1050: in _gcd_import
    ???
<frozen importlib._bootstrap>:1027: in _find_and_load
    ???
<frozen importlib._bootstrap>:1006: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:688: in _load_unlocked
    ???
../../.local/lib/python3.10/site-packages/_pytest/assertion/rewrite.py:168: in exec_module
    exec(co, module.__dict__)
test/test_tabular.py:1: in <module>
    import pytabular
pytabular/__init__.py:32: in <module>
    import clr
E   ModuleNotFoundError: No module named 'clr'
================================================================================= short test summary info ==================================================================================
ERROR test/test_tabular.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
===================================================================================== 1 error in 0.08s =====================================================================================

Import log:

Python 3.10.6 (main, Aug  3 2022, 17:39:45) [GCC 12.1.1 20220730] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytabular
22/10/01 10:06:51 +0300::__init__::<module>::INFO::Logging configured...
22/10/01 10:06:51 +0300::__init__::<module>::INFO::To update PyTabular logger...
22/10/01 10:06:51 +0300::__init__::<module>::INFO::>>> import logging
22/10/01 10:06:51 +0300::__init__::<module>::INFO::>>> pytabular.logger.setLevel(level=logging.INFO)
22/10/01 10:06:51 +0300::__init__::<module>::INFO::See https://docs.python.org/3/library/logging.html#logging-levels
22/10/01 10:06:51 +0300::__init__::<module>::DEBUG::Python Version::3.10.6 (main, Aug  3 2022, 17:39:45) [GCC 12.1.1 20220730]
22/10/01 10:06:51 +0300::__init__::<module>::DEBUG::Python Location::/home/radi/Code/PyTabular/venv
22/10/01 10:06:51 +0300::__init__::<module>::DEBUG::Package Location::/home/radi/Code/PyTabular/pytabular/__init__.py
22/10/01 10:06:51 +0300::__init__::<module>::DEBUG::Working Directory::/home/radi/Code/PyTabular
22/10/01 10:06:51 +0300::__init__::<module>::DEBUG::Platform::linux-5.19.7-1-MANJARO
22/10/01 10:06:51 +0300::__init__::<module>::DEBUG::Beginning CLR references...
22/10/01 10:06:51 +0300::__init__::<module>::DEBUG::Adding Reference Microsoft.AnalysisServices.AdomdClient
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/radi/Code/PyTabular/pytabular/__init__.py", line 35, in <module>
    clr.AddReference("Microsoft.AnalysisServices.AdomdClient")
AttributeError: module 'clr' has no attribute 'AddReference'
Curts0 commented 1 year ago

@Bugzey it does specify that OS is only windows in the pyproject.toml. Good call out though and I will add that specification to the README. I welcome any insights to get it compatible with other OS. Current use cases work fine in windows vms. I've currently got Azure Pipelines running for my requirements.

Curts0 commented 1 year ago

Added disclaimer and put 'windows' tag in top right tags. Converting to Enhancement with hope that some smart people will take a look at it.

Bugzey commented 1 year ago

My initial run had for some reason installed and imported a different clr library - https://pypi.org/project/clr/. Why doesn't pythonnet just define pythonnet? Anyway, the package now imports correctly, and I can import objects from Microsoft.AnalysisServices.

Now I'd love to at least get the tests running, but I can't figure out what the import local line in tests.test_tabular actually imports. Can you please clarify what that is? Link

PS: My tests don't necessarily prove Linux compatibility yet since I'm not sure why pythonnet and misc C# code is successfully running on my system. I most probably have some remnant Mono or other Microsoft tools installed as part of other packages. Optimally, we should set up the package and run the tests on a blank virtual machine in order to test out exactly what needs to be set up to get this package up and running.

Curts0 commented 1 year ago

local is a workaround I have so I could continue testing and moving this forward. I have #10 to get rid of that workaround. Because local contains info to get connected to my local models (an AAS model and Gen2 model).

My thoughts are getting an AdventureWorks DB and Model spun up when running tests? Then run the tests off that? What do you think? Rather than having the user needing to supply their own model.

Looking at all the local references though... I can git rid of the need for local.SINGLEVALTESTPATH and local.DFVALTESTPATH because those are just file paths to the existing .dax files in the repo in the test folder.

Which means all that is left is solving a universal Tabular model for the test to run on.