Move "auto" command out of core package.

[schettino72]

External dependencies make it hard to package.

Being external to doit core also makes it easy to transition to a new underlying file watcher implementation.

[schettino72]

@bollwyvl

I prefer to have a separate package than using [extra]. doit has an ok support for command installed as plugins... I will do this change on next release 0.36.

Not clear to me if 0.35 is ok or broken. Should I do another release immediately to fix this? Not a problem to me...

[bollwyvl]

0.35 is ok or broken

No, nothing is broken right now for 0.35 on any of the existing supported platforms/pythons (or any more broken than they were in <0.35). The challenges are coming for new things.

For example, on conda-forge: I can't build doit, unpatched, for the new apple silicon processors because i can't trust the build of macfsevents because all the tests fail... but it wasn't working for py3.8+ anyway (#372), but there are no builds of python 3.7 for the m1. This is semi-academic, as I have no real way to test an m1, as we still can only use virtualized x86 machines with emulation to build.

separate package

Yes, by all means! extras, as implemented, are fairly fiddly bits, and pip for example won't honor pins from them on subsequent installs. But it would be a way to keep it as part of core... but having a focused, more robust plugin is preferable!

Further, if the core package's platform-specific dependencies went away, doit itself could be noarch, and we'd only have to do one build per version instead of py x os x arch builds.

if the new package (e.g. doit-auto) package were to use watchfiles, then it, too, could be noarch, which would be great, as it could centralize the requirement for a working rust toolchain on all the platforms onto that package (which i don't maintain!)

ok support for command installed as plugins...

Right: now that doit is 3.8+, it should be able to rely on getting entry_points from stdlib's importlib.metadata... but unfortunately, the original implementation was horribly inefficient until 3.10, and so in the near-term relying on importlib-metadata >=3.6 for python < 3.10 is required so that just the groups wanted are loaded:

import sys

if sys.version_info < (3,10):
    from importlib_metadata import entry_points 
else:
    from importlib.metadata import entry_points 

# ...
    entry_points(group='{self.entry_point_prefix}.{category}')

another release

So it sounds like to minimize feature loss, this would entail some, but probably all, of the following:

• remove doit auto
• removing all the platform-specific extras
• add importlib_metadata as a dependency
• make plugins extras a no-op
• make a new package providing doit auto
  • offer a version that preserves the pyinotify and macfsevents implementations

At some time in the future:

• offer a new, cross-platform version of the doit auto package that uses watchfiles

[schettino72]

@bollwyvl thanks for detailed roadmap :100:

Implemented below: https://github.com/pydoit/doit/commit/5e1c695902626ca68236115da4fb827d0097e4f4 https://github.com/pydoit/doit/commit/f4a4e6c09ea9d6721890ba8d82a8db4fcab0f984

make plugins extras a no-op

Why not just remove it?

[nrbgt]

make plugins extras a no-op

Why not just remove it?

I guess just to keep as much backwards compatibility as possible.... but I guess even the first-party stuff like doit-py didn't use doit[plugins] as its install_requires.