search

ckan / ckan

CKAN is an open-source DMS (data management system) for powering data hubs and data portals. CKAN makes it easy to publish, share and use data. It powers catalog.data.gov, open.canada.ca/data, data.humdata.org among many other sites.
https://ckan.org/
Other
4.43k stars 1.98k forks source link

Migrate ckanext namespace package to PEP420 (or away from namespace packages) #8465

Open amercader opened 1 week ago

amercader commented 1 week ago

Initially discussed in https://github.com/ckan/ckan/issues/8382#issuecomment-2353014999.

The ckanext package used by plugins is a namespace package. CKAN currently uses a deprecated setuptools pkg_resources-based way of defining namespaces packages. The recommendation seems to be to start migrating away from this method.

To gather a bit more information my suggested plan is:

  1. Migrate CKAN itself to native namespace packages (there are plenty of ckanext extensions in CKAN core)
  2. Migrate a couple of extensions to native namespace packages
  3. Run a suite of tests similar to these ones but tailored to our usual setups to identify the problematic combinations

I started work on 1 in the pep420-namespace-packages branch (it also includes https://github.com/ckan/ckan/pull/8456) and it is not as straightforward as it seems. Switching the namespace package method changes imports in subtle ways, that are reflected in seemingly unrelated test failures. For instance the last ones are caused by trying to capture an exception class defined in one of the tests modules (AuthTestException). Inspecting this in the work branch gives us:

<class 'datastore.tests.test_chained_auth_functions.AuthTestException'>

While on master:

<class 'ckanext.datastore.tests.test_chained_auth_functions.AuthTestException'>

Other worrying signs include warnings like this one, which don't occur in master:

python -c "from ckanext.datastore import helpers"
/home/adria/dev/pyenvs/pypackaging4/ckan/ckan/plugins/core.py:10: UserWarning: Module ckanext was already imported from None, but /home/adria/dev/pyenvs/pypackaging4/ckan is being added to sys.path
  from pkg_resources import iter_entry_points

Quick tests in extensions for point 2 also resulted in errors .

All this to say that more investigation is required in order to plan a transition. Any suggestions from users with experience around these topics is appreciated.

wardi commented 1 week ago

To migrate an existing package, all packages sharing the namespace must be migrated simultaneously.

Should we create a new namespace for extensions to allow for the transition? e.g. ckanext for legacy pkg_resources-namespace and ckanx/ ckanext3 or something for the new way?