search

CaringCaribou / caringcaribou

A friendly car security exploration tool for the CAN bus
GNU General Public License v3.0
751 stars 197 forks source link

Use importlib.resources to fix module loading #109

Closed obbardc closed 4 months ago

obbardc commented 5 months ago

When running caringcaribou from a local directory with python3 -m caringcaribou.caringcaribou -h, no modules are found.

Furthermore, pkg_resources is a deprecated API:

DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html

Refactor the module loading code to fix loading modules when running from a local directory as well as fix the code which uses a deprecated API.

obbardc commented 5 months ago

It's a draft as I never checked if this works when installed :-)

kasperkarlsson commented 4 months ago

Hello,

Thank you for reaching out. Removing the deprecated pkg_resources API would be very welcome.

I do however note that this PR changes the behavior of the application, as trying to run any module causes an import of doipclient, which is only supposed to be loaded when using the doip module. This can be illustrated by running e.g. the dump or test module on a system where doipclient is not installed:

$ caringcaribou -i vcan0 dump
Could not import module caringcaribou.modules.doip: No module named 'doipclient'

-------------------
CARING CARIBOU v0.6 - python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
-------------------

Loading module 'dump'

Could not import module caringcaribou.modules.doip: No module named 'doipclient'
$ caringcaribou -i vcan0 test
Could not import module caringcaribou.modules.doip: No module named 'doipclient'

-------------------
CARING CARIBOU v0.6 - python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
-------------------

Loading module 'test'

Could not import module caringcaribou.modules.doip: No module named 'doipclient'

Is it possible that this suggested change causes all imports from all modules to be loaded, regardless of which module is run?

EDIT: Looking closer on the code, this change does indeed load all modules inside the function available_modules_dict() (which should not perform any import). This should be a quick fix 🙂

kasperkarlsson commented 4 months ago

I made some minor changes to resolve the issues mentioned above, and it works great now!

I will wait a little while before merging the PR in case you wish to review my changes or provide any comments.

obbardc commented 4 months ago

Seems like I can't install in a venv using python setup.py install anymore. I don't know what happened - maybe something upgraded? - I really don't know python that well:

(venv) obbardc@obbardc-t14:~/projects/canbus/caringcaribou$ python3 -m venv venv
(venv) obbardc@obbardc-t14:~/projects/canbus/caringcaribou$ source venv/bin/activate
(venv) obbardc@obbardc-t14:~/projects/canbus/caringcaribou$ python setup.py install
Traceback (most recent call last):
  File "/home/obbardc/projects/canbus/caringcaribou/setup.py", line 11, in <module>
    from setuptools import find_packages, setup
ModuleNotFoundError: No module named 'setuptools'
(venv) obbardc@obbardc-t14:~/projects/canbus/caringcaribou$ pip install setuptools
Collecting setuptools
  Using cached setuptools-70.1.1-py3-none-any.whl.metadata (6.0 kB)
Using cached setuptools-70.1.1-py3-none-any.whl (883 kB)
Installing collected packages: setuptools
Successfully installed setuptools-70.1.1
(venv) obbardc@obbardc-t14:~/projects/canbus/caringcaribou$ python setup.py install
-----------------------------------
 Installing Caring Caribou version 0.6
-----------------------------------

/home/obbardc/projects/canbus/caringcaribou/venv/lib/python3.12/site-packages/setuptools/config/_apply_pyprojecttoml.py:72: _MissingDynamic: `keywords` defined outside of `pyproject.toml` is ignored.
!!

        ********************************************************************************
        The following seems to be defined outside of `pyproject.toml`:

        `keywords = ['automotive', 'security', 'CAN', 'automotive protocols', 'fuzzing']`

        According to the spec (see the link below), however, setuptools CANNOT
        consider this value unless `keywords` is listed as `dynamic`.

        https://packaging.python.org/en/latest/specifications/pyproject-toml/#declaring-project-metadata-the-project-table

        To prevent this problem, you can list `keywords` under `dynamic` or alternatively
        remove the `[project]` table from your file and rely entirely on other means of
        configuration.
        ********************************************************************************

!!
  _handle_missing_dynamic(dist, project_table)
/home/obbardc/projects/canbus/caringcaribou/venv/lib/python3.12/site-packages/setuptools/config/_apply_pyprojecttoml.py:72: _MissingDynamic: `entry-points` defined outside of `pyproject.toml` is ignored.
!!

        ********************************************************************************
        The following seems to be defined outside of `pyproject.toml`:

        `entry-points = {'caringcaribou.modules': ['dcm = caringcaribou.modules.dcm', 'doip = caringcaribou.modules.doip', 'dump = caringcaribou.modules.dump', 'fuzzer = caringcaribou.modules.fuzzer', 'listener = caringcaribou.modules.listener', 'send = caringcaribou.modules.send', 'test = caringcaribou.modules.test', 'uds_fuzz = caringcaribou.modules.uds_fuzz', 'uds = caringcaribou.modules.uds', 'xcp = caringcaribou.modules.xcp']}`

        According to the spec (see the link below), however, setuptools CANNOT
        consider this value unless `entry-points` is listed as `dynamic`.

        https://packaging.python.org/en/latest/specifications/pyproject-toml/#declaring-project-metadata-the-project-table

        To prevent this problem, you can list `entry-points` under `dynamic` or alternatively
        remove the `[project]` table from your file and rely entirely on other means of
        configuration.
        ********************************************************************************

!!
  _handle_missing_dynamic(dist, project_table)
/home/obbardc/projects/canbus/caringcaribou/venv/lib/python3.12/site-packages/setuptools/config/_apply_pyprojecttoml.py:72: _MissingDynamic: `scripts` defined outside of `pyproject.toml` is ignored.
!!

        ********************************************************************************
        The following seems to be defined outside of `pyproject.toml`:

        `scripts = ['cc.py=caringcaribou.caringcaribou:main', 'caringcaribou=caringcaribou.caringcaribou:main']`

        According to the spec (see the link below), however, setuptools CANNOT
        consider this value unless `scripts` is listed as `dynamic`.

        https://packaging.python.org/en/latest/specifications/pyproject-toml/#declaring-project-metadata-the-project-table

        To prevent this problem, you can list `scripts` under `dynamic` or alternatively
        remove the `[project]` table from your file and rely entirely on other means of
        configuration.
        ********************************************************************************

!!
  _handle_missing_dynamic(dist, project_table)
/home/obbardc/projects/canbus/caringcaribou/venv/lib/python3.12/site-packages/setuptools/config/_apply_pyprojecttoml.py:72: _MissingDynamic: `dependencies` defined outside of `pyproject.toml` is ignored.
!!

        ********************************************************************************
        The following seems to be defined outside of `pyproject.toml`:

        `dependencies = ['python-can']`

        According to the spec (see the link below), however, setuptools CANNOT
        consider this value unless `dependencies` is listed as `dynamic`.

        https://packaging.python.org/en/latest/specifications/pyproject-toml/#declaring-project-metadata-the-project-table

        To prevent this problem, you can list `dependencies` under `dynamic` or alternatively
        remove the `[project]` table from your file and rely entirely on other means of
        configuration.
        ********************************************************************************

!!
  _handle_missing_dynamic(dist, project_table)
/home/obbardc/projects/canbus/caringcaribou/venv/lib/python3.12/site-packages/setuptools/config/_apply_pyprojecttoml.py:79: SetuptoolsWarning: `install_requires` overwritten in `pyproject.toml` (dependencies)
  corresp(dist, value, root_dir)
running install
/home/obbardc/projects/canbus/caringcaribou/venv/lib/python3.12/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
/home/obbardc/projects/canbus/caringcaribou/venv/lib/python3.12/site-packages/setuptools/_distutils/cmd.py:66: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` and ``easy_install``.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://github.com/pypa/setuptools/issues/917 for details.
        ********************************************************************************

!!
  self.initialize_options()
running bdist_egg
running egg_info
creating caringcaribou.egg-info
writing caringcaribou.egg-info/PKG-INFO
writing dependency_links to caringcaribou.egg-info/dependency_links.txt
writing top-level names to caringcaribou.egg-info/top_level.txt
writing manifest file 'caringcaribou.egg-info/SOURCES.txt'
reading manifest file 'caringcaribou.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'caringcaribou.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
copying caringcaribou/__init__.py -> build/lib/caringcaribou
copying caringcaribou/caringcaribou.py -> build/lib/caringcaribou
copying caringcaribou/utils/constants.py -> build/lib/caringcaribou/utils
copying caringcaribou/utils/iso14229_1.py -> build/lib/caringcaribou/utils
copying caringcaribou/utils/can_actions.py -> build/lib/caringcaribou/utils
copying caringcaribou/utils/__init__.py -> build/lib/caringcaribou/utils
copying caringcaribou/utils/common.py -> build/lib/caringcaribou/utils
copying caringcaribou/utils/iso15765_2.py -> build/lib/caringcaribou/utils
copying caringcaribou/modules/dcm.py -> build/lib/caringcaribou/modules
copying caringcaribou/modules/listener.py -> build/lib/caringcaribou/modules
copying caringcaribou/modules/uds_fuzz.py -> build/lib/caringcaribou/modules
copying caringcaribou/modules/test.py -> build/lib/caringcaribou/modules
copying caringcaribou/modules/__init__.py -> build/lib/caringcaribou/modules
copying caringcaribou/modules/module_template.py -> build/lib/caringcaribou/modules
copying caringcaribou/modules/dump.py -> build/lib/caringcaribou/modules
copying caringcaribou/modules/doip.py -> build/lib/caringcaribou/modules
copying caringcaribou/modules/send.py -> build/lib/caringcaribou/modules
copying caringcaribou/modules/uds.py -> build/lib/caringcaribou/modules
copying caringcaribou/modules/fuzzer.py -> build/lib/caringcaribou/modules
copying caringcaribou/modules/xcp.py -> build/lib/caringcaribou/modules
copying caringcaribou/tests/test_iso_14229_1.py -> build/lib/caringcaribou/tests
copying caringcaribou/tests/test_iso_15765_2.py -> build/lib/caringcaribou/tests
copying caringcaribou/tests/__init__.py -> build/lib/caringcaribou/tests
copying caringcaribou/tests/test_send.py -> build/lib/caringcaribou/tests
copying caringcaribou/tests/test_module_uds.py -> build/lib/caringcaribou/tests
copying caringcaribou/tests/mock/mock_ecu_uds.py -> build/lib/caringcaribou/tests/mock
copying caringcaribou/tests/mock/__init__.py -> build/lib/caringcaribou/tests/mock
copying caringcaribou/tests/mock/mock_ecu.py -> build/lib/caringcaribou/tests/mock
copying caringcaribou/tests/mock/mock_ecu_iso_tp.py -> build/lib/caringcaribou/tests/mock
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/caringcaribou
creating build/bdist.linux-x86_64/egg/caringcaribou/utils
copying build/lib/caringcaribou/utils/constants.py -> build/bdist.linux-x86_64/egg/caringcaribou/utils
copying build/lib/caringcaribou/utils/iso14229_1.py -> build/bdist.linux-x86_64/egg/caringcaribou/utils
copying build/lib/caringcaribou/utils/can_actions.py -> build/bdist.linux-x86_64/egg/caringcaribou/utils
copying build/lib/caringcaribou/utils/__init__.py -> build/bdist.linux-x86_64/egg/caringcaribou/utils
copying build/lib/caringcaribou/utils/common.py -> build/bdist.linux-x86_64/egg/caringcaribou/utils
copying build/lib/caringcaribou/utils/iso15765_2.py -> build/bdist.linux-x86_64/egg/caringcaribou/utils
creating build/bdist.linux-x86_64/egg/caringcaribou/modules
copying build/lib/caringcaribou/modules/dcm.py -> build/bdist.linux-x86_64/egg/caringcaribou/modules
copying build/lib/caringcaribou/modules/listener.py -> build/bdist.linux-x86_64/egg/caringcaribou/modules
copying build/lib/caringcaribou/modules/uds_fuzz.py -> build/bdist.linux-x86_64/egg/caringcaribou/modules
copying build/lib/caringcaribou/modules/test.py -> build/bdist.linux-x86_64/egg/caringcaribou/modules
copying build/lib/caringcaribou/modules/__init__.py -> build/bdist.linux-x86_64/egg/caringcaribou/modules
copying build/lib/caringcaribou/modules/module_template.py -> build/bdist.linux-x86_64/egg/caringcaribou/modules
copying build/lib/caringcaribou/modules/dump.py -> build/bdist.linux-x86_64/egg/caringcaribou/modules
copying build/lib/caringcaribou/modules/doip.py -> build/bdist.linux-x86_64/egg/caringcaribou/modules
copying build/lib/caringcaribou/modules/send.py -> build/bdist.linux-x86_64/egg/caringcaribou/modules
copying build/lib/caringcaribou/modules/uds.py -> build/bdist.linux-x86_64/egg/caringcaribou/modules
copying build/lib/caringcaribou/modules/fuzzer.py -> build/bdist.linux-x86_64/egg/caringcaribou/modules
copying build/lib/caringcaribou/modules/xcp.py -> build/bdist.linux-x86_64/egg/caringcaribou/modules
copying build/lib/caringcaribou/__init__.py -> build/bdist.linux-x86_64/egg/caringcaribou
copying build/lib/caringcaribou/caringcaribou.py -> build/bdist.linux-x86_64/egg/caringcaribou
creating build/bdist.linux-x86_64/egg/caringcaribou/tests
copying build/lib/caringcaribou/tests/test_iso_14229_1.py -> build/bdist.linux-x86_64/egg/caringcaribou/tests
copying build/lib/caringcaribou/tests/test_iso_15765_2.py -> build/bdist.linux-x86_64/egg/caringcaribou/tests
copying build/lib/caringcaribou/tests/__init__.py -> build/bdist.linux-x86_64/egg/caringcaribou/tests
copying build/lib/caringcaribou/tests/test_send.py -> build/bdist.linux-x86_64/egg/caringcaribou/tests
creating build/bdist.linux-x86_64/egg/caringcaribou/tests/mock
copying build/lib/caringcaribou/tests/mock/mock_ecu_uds.py -> build/bdist.linux-x86_64/egg/caringcaribou/tests/mock
copying build/lib/caringcaribou/tests/mock/__init__.py -> build/bdist.linux-x86_64/egg/caringcaribou/tests/mock
copying build/lib/caringcaribou/tests/mock/mock_ecu.py -> build/bdist.linux-x86_64/egg/caringcaribou/tests/mock
copying build/lib/caringcaribou/tests/mock/mock_ecu_iso_tp.py -> build/bdist.linux-x86_64/egg/caringcaribou/tests/mock
copying build/lib/caringcaribou/tests/test_module_uds.py -> build/bdist.linux-x86_64/egg/caringcaribou/tests
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/utils/constants.py to constants.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/utils/iso14229_1.py to iso14229_1.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/utils/can_actions.py to can_actions.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/utils/__init__.py to __init__.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/utils/common.py to common.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/utils/iso15765_2.py to iso15765_2.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/modules/dcm.py to dcm.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/modules/listener.py to listener.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/modules/uds_fuzz.py to uds_fuzz.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/modules/test.py to test.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/modules/__init__.py to __init__.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/modules/module_template.py to module_template.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/modules/dump.py to dump.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/modules/doip.py to doip.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/modules/send.py to send.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/modules/uds.py to uds.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/modules/fuzzer.py to fuzzer.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/modules/xcp.py to xcp.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/__init__.py to __init__.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/caringcaribou.py to caringcaribou.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/tests/test_iso_14229_1.py to test_iso_14229_1.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/tests/test_iso_15765_2.py to test_iso_15765_2.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/tests/__init__.py to __init__.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/tests/test_send.py to test_send.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/tests/mock/mock_ecu_uds.py to mock_ecu_uds.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/tests/mock/__init__.py to __init__.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/tests/mock/mock_ecu.py to mock_ecu.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/tests/mock/mock_ecu_iso_tp.py to mock_ecu_iso_tp.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/caringcaribou/tests/test_module_uds.py to test_module_uds.cpython-312.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying caringcaribou.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying caringcaribou.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying caringcaribou.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying caringcaribou.egg-info/not-zip-safe -> build/bdist.linux-x86_64/egg/EGG-INFO
copying caringcaribou.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
creating 'dist/caringcaribou-0.6-py3.12.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing caringcaribou-0.6-py3.12.egg
creating /home/obbardc/projects/canbus/caringcaribou/venv/lib/python3.12/site-packages/caringcaribou-0.6-py3.12.egg
Extracting caringcaribou-0.6-py3.12.egg to /home/obbardc/projects/canbus/caringcaribou/venv/lib/python3.12/site-packages
Adding caringcaribou 0.6 to easy-install.pth file

Installed /home/obbardc/projects/canbus/caringcaribou/venv/lib/python3.12/site-packages/caringcaribou-0.6-py3.12.egg
Processing dependencies for caringcaribou==0.6
Finished processing dependencies for caringcaribou==0.6
-------------------------------------------------------------------
 Installation completed, run `caringcaribou --help` to get started
-------------------------------------------------------------------

(venv) obbardc@obbardc-t14:~/projects/canbus/caringcaribou$ caringcaribou
bash: caringcaribou: command not found

But I am fine with your changes and am happy if you wish to squash my commit etc !

kasperkarlsson commented 4 months ago

All right! I tried moving the keywords definition to pyproject.toml - could you please see if this works better for you in the venv installation?

obbardc commented 4 months ago

Same thing:


Installed /home/obbardc/projects/canbus/caringcaribou/venv/lib/python3.12/site-packages/caringcaribou-0.6-py3.12.egg
Processing dependencies for caringcaribou==0.6
Finished processing dependencies for caringcaribou==0.6
-------------------------------------------------------------------
 Installation completed, run `caringcaribou --help` to get started
-------------------------------------------------------------------

(venv) obbardc@obbardc-t14:~/projects/canbus/caringcaribou$ caringcaribou
bash: caringcaribou: command not found

But I doubt this is related to this PR.

obbardc commented 4 months ago

@kasperkarlsson Turns out we need some bits in pyproject such that it builds. Not sure all of it is needed.

kasperkarlsson commented 4 months ago

It seems to work well for my setup as well - let's merge this, and in case you (or anyone else) comes up with further improvements, that could be handled in another PR 🙂 Thank you for your help!