Please document how to develop builder plugins by turning off build isolation.

[gl-yziquel]

I tried create a builder plugin in hatch / hatchling and understood, rather late, that I had to turn off build isolation.

I believe this should be documented somewhere on the builder plugin page in the docs. Otherwise, newcommers to these kind of questions like tend to run in circles while developing builder plugins.

[gl-yziquel]

Because it is completely unclear how to perform build isolation disabling in hatch to build builder plugins. I'm running in circles and will likely continue to run in circles for a rather long time.

[ofek]

Do you mean this? https://hatch.pypa.io/latest/how-to/plugins/testing-builds/

[gl-yziquel]

Do you mean this? https://hatch.pypa.io/latest/how-to/plugins/testing-builds/

Seems like it. Though this is about testing a plugin. Not quite about how to import it in another hatch package.

I figured out the @ trick fo git urls yesterday night, but I can't get the environment right. It seems to pull in my new builder plugin, but complains about hatchling missing in the environment.

mini-me@virtucon ~/h/s/mud-pj (master)> just build                                                                  
hatch build --target bytecode                                                                                          
ERROR: Exception:                                                                                                      
Traceback (most recent call last):              
  File "/home/mini-me/.local/share/hatch/env/virtual/mud-pj/l53_DRsO/mud-pj-build/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)                                                                                           
             ^^^^^^^^^^^^^^^                            
  File "/home/mini-me/.local/share/hatch/env/virtual/mud-pj/l53_DRsO/mud-pj-build/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mini-me/.local/share/hatch/env/virtual/mud-pj/l53_DRsO/mud-pj-build/lib/python3.12/site-packages/pip/_internal/commands/install.py", line 377, in run
    requirement_set = resolver.resolve(
                      ^^^^^^^^^^^^^^^^^
  File "/home/mini-me/.local/share/hatch/env/virtual/mud-pj/l53_DRsO/mud-pj-build/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 76, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mini-me/.local/share/hatch/env/virtual/mud-pj/l53_DRsO/mud-pj-build/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 534, in collect_root_requirements
    reqs = list(
           ^^^^^
  File "/home/mini-me/.local/share/hatch/env/virtual/mud-pj/l53_DRsO/mud-pj-build/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 490, in _make_requirements_from_install_req
    cand = self._make_base_candidate_from_link(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mini-me/.local/share/hatch/env/virtual/mud-pj/l53_DRsO/mud-pj-build/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 228, in _make_base_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
                                       ^^^^^^^^^^^^^^
  File "/home/mini-me/.local/share/hatch/env/virtual/mud-pj/l53_DRsO/mud-pj-build/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 290, in __init__
    super().__init__(
  File "/home/mini-me/.local/share/hatch/env/virtual/mud-pj/l53_DRsO/mud-pj-build/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
                ^^^^^^^^^^^^^^^
  File "/home/mini-me/.local/share/hatch/env/virtual/mud-pj/l53_DRsO/mud-pj-build/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 222, in _prepare
    dist = self._prepare_distribution()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mini-me/.local/share/hatch/env/virtual/mud-pj/l53_DRsO/mud-pj-build/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 301, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mini-me/.local/share/hatch/env/virtual/mud-pj/l53_DRsO/mud-pj-build/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 525, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mini-me/.local/share/hatch/env/virtual/mud-pj/l53_DRsO/mud-pj-build/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 640, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mini-me/.local/share/hatch/env/virtual/mud-pj/l53_DRsO/mud-pj-build/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 71, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
  File "/home/mini-me/.local/share/hatch/env/virtual/mud-pj/l53_DRsO/mud-pj-build/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 67, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/home/mini-me/.local/share/hatch/env/virtual/mud-pj/l53_DRsO/mud-pj-build/lib/python3.12/site-packages/pip/_internal/req/req_install.py", line 579, in prepare_metadata
    self.metadata_directory = generate_metadata(
                              ^^^^^^^^^^^^^^^^^^
  File "/home/mini-me/.local/share/hatch/env/virtual/mud-pj/l53_DRsO/mud-pj-build/lib/python3.12/site-packages/pip/_internal/operations/build/metadata.py", line 35, in generate_metadata
    distinfo_dir = backend.prepare_metadata_for_build_wheel(metadata_dir)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mini-me/.local/share/hatch/env/virtual/mud-pj/l53_DRsO/mud-pj-build/lib/python3.12/site-packages/pip/_internal/utils/misc.py", line 766, in prepare_metadata_for_build_wheel
    return super().prepare_metadata_for_build_wheel(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mini-me/.local/share/hatch/env/virtual/mud-pj/l53_DRsO/mud-pj-build/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 186, in prepare_metadata_for_build_wheel
    return self._call_hook('prepare_metadata_for_build_wheel', {
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mini-me/.local/share/hatch/env/virtual/mud-pj/l53_DRsO/mud-pj-build/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 321, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
  File "/home/mini-me/.local/share/hatch/env/virtual/mud-pj/l53_DRsO/mud-pj-build/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
    obj = import_module(mod_path)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mini-me/home/cellar/pyenv/versions/3.12.2/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1324, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'hatchling'

[gl-yziquel]

Errata: the above stack trace is what you get when you mess around: you follow the various example snippets of working hatch plugins on the net AND you have PIP_NO_BUILD_ISOLATION set to False in your shell because you've been torturing your computer to dive into hatch plugins.

@randomuser: If you get the above stack trace, unset PIP_NO_BUILD_ISOLATION, and this will get back to normal.

@ofek yes. These development instructions are indeed available in the docs. I just did not stumble upon them. I guess I'm kind of stuck mentally in the 90s and yet haven't really learnt to use a mouse with wepbages compared to using nroff in a terminal. All apologies. An example of a git repo containing two folders, one for the plugin, and for something using the plugin, fully functional out of the box would nonetheless help save much tinkering time. Thanks for hatch.

[gl-yziquel]

Is there a reason why my updates are not picked up with requires = ["hatchling", "hatch-bytecode@file:///home/mini-me/home/stove/hatch-bytecode"] ? Even after "hatch env prune", Any modifications I make are ignored when I run hatch build to test the builder plugin.

[gl-yziquel]

"The issue with this is that after the first test session, the project will be forever cached by pip based on the file path. Therefore, subsequent tests runs will never use updated code. To invalidate the cache, copy your code to a new path for every test session:"

Argh !!!! This is completely nuts !

[gl-yziquel]

pip cache remove my_plugin should be added to the doc instead of this path copying suggestion.

[ofek]

Feel free to add that to the page as a suggestion when manually testing but I would like to keep the current approach as recommended because the command is modifying the global cache on your system rather than working around the issue at the test level. Also, one might not have access to pip if using UV as the installer.

edit: I like your idea of having an example repository for each plugin type, I'll work on that!

[gl-yziquel]

One thing that would need perhaps correction is the reachability of this "how-to" page from the rest of the documentation. I'm not sure that there is a link from the default builder plugin page to this howto page. Which would be a shame as it is the blocking point for developing builder plugins.

[gl-yziquel]

I added such a link to the how to from the builder plugins page in the PR.