Feature/better hatch build system

[mattprintz]

There's a little bit of todos for future work, but this seems to cover all the cases so far and works well enough. Right now it just does auto-detection, but want to allow people to explicitly list the contexts/subkernels in the pyproject instead, but figure that can be a future update.

[brandomr]

This looks good to me just looking through the code. What's the best way to test it outside this repo for a new project/context?

[mattprintz]

Note: Between the question above and now, I fixed a couple bugs that was preventing the package to build from github and Docker, and I went ahead and did some refactoring in beaker_kernel to make class names and using the "base" clasess easier and more intuitive.

To build a new context, you would first start from a new/existing python project that uses the hatch builder.

In the source files, you will want to define at least a new context class based off of (the newly renamed) BeakerContext class. You will probably want to define an agent as well, but it's possible to use the default (BeakerAgent) for basic interactions.

The new context class MUST have a slug defined as a class attribute. It will probably have init and setup methods defines as well, along with any tools, etc.

A basic context file would look something like this: my_project/src/my_project/context.py

from beaker_kernel import BeakerContext
from .agent import MyAgent

class MyContext(BeakerContext):

    enabled_subkernels = ["python3"]

    SLUG = "my_context"

    def __init__(self, beaker_kernel: "BeakerKernel", config: Dict[str, Any]):
        super().__init__(beaker_kernel, MyAgent, config)

    async def setup(self, context_info=None, parent_header=None):
        super().setup(context_info, parent_header)

With the context existing in a .py file in the src tree for the package, it will be discovered by the auto-discovery in the builder.

To build the package so that the context is properly registered, you can updated your pyproject.toml as follows:

dependencies = [
    "beaker_kernel@git+https://github.com/jataware/beaker-kernel.git@feature/better-hatch-build-system#egg=beaker_kernel",
]

[tool.hatch.build]
require-runtime-dependencies = true

[tool.hatch.build.hooks.beaker]

The require-runtime-dependencies is required because it ensures that beaker_kernel is installed during build time so we can check if the classes in the src tree are subclasses of the BeakerContext class defined therein. I would love to make that be inherent for the plugin, but so far haven't figured out how to do so as yet.

Right now, there are no options for tool.hatch.build.hooks.beaker, but there will probably be some in the future as described above.

Once the new package has been installed, beaker can be run in a normal way, (docker or beaker-cli) and the context should show up in the list and be usable.

A shortcut for the build magic would be to run unzip -l dist/*.whl and confirm that a json file exists in the zip file with the name of the file matching the slug defined on the class.

Let me know if you have any questions. @brandomr

[brandomr]

I'm getting this error:

.931   × Preparing editable metadata (pyproject.toml) did not run successfully.
1.931   │ exit code: 1
1.931   ╰─> [21 lines of output]
1.931       Traceback (most recent call last):
1.931         File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 167, in prepare_metadata_for_build_editable
1.931           hook = backend.prepare_metadata_for_build_editable
1.931       AttributeError: module 'hatchling.build' has no attribute 'prepare_metadata_for_build_editable'
1.931       
1.931       During handling of the above exception, another exception occurred:
1.931       
1.931       Traceback (most recent call last):
1.931         File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
1.931           main()
1.931         File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
1.931           json_out['return_val'] = hook(**hook_input['kwargs'])
1.931         File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 176, in prepare_metadata_for_build_editable
1.931           whl_basename = build_hook(metadata_directory, config_settings)
1.931         File "/tmp/pip-build-env-6xjxursf/overlay/lib/python3.10/site-packages/hatchling/build.py", line 83, in build_editable
1.931           return os.path.basename(next(builder.build(directory=wheel_directory, versions=['editable'])))
1.931         File "/tmp/pip-build-env-6xjxursf/overlay/lib/python3.10/site-packages/hatchling/builders/plugin/interface.py", line 116, in build
1.931           configured_build_hooks = self.get_build_hooks(directory)
1.931         File "/tmp/pip-build-env-6xjxursf/overlay/lib/python3.10/site-packages/hatchling/builders/plugin/interface.py", line 386, in get_build_hooks
1.931           raise UnknownPluginError(message)
1.931       hatchling.plugin.exceptions.UnknownPluginError: Unknown build hook: beaker
1.931       [end of output]

Could you share a full pyproject.toml that worked for you? I've got:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "test"
dynamic = ["version"]
description = ''
requires-python = ">=3.10"
license = "MIT"
keywords = []
authors = [
  { name = "Brandon Rose", email = "brandon@jataware.com" },
]

[tool.hatch.version]
path = "src/test/__about__.py"

dependencies = [
    "beaker_kernel@git+https://github.com/jataware/beaker-kernel.git@feature/better-hatch-build-system#egg=beaker_kernel",
]

[tool.hatch.metadata]
allow-direct-references = true

[tool.hatch.build]
require-runtime-dependencies = true

[tool.hatch.build.hooks.beaker]

[mattprintz]

Ok, so I created a new project and basically copied your pyproject there exactly and the package built just fine out of the box:

matt@mavis:~/development/beaker/beaker-buildtest$ hatch build
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── sdist ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
dist/beaker_buildtest-0.0.1.tar.gz
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── wheel ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Found the following contexts:
  'my_context': TestContext in package beaker_buildtest.context

dist/beaker_buildtest-0.0.1-py3-none-any.whl
matt@mavis:~/development/beaker/beaker-buildtest$ unzip -l dist/beaker_buildtest-0.0.1-py3-none-any.whl 
Archive:  dist/beaker_buildtest-0.0.1-py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
      129  2020-02-02 00:00   beaker_buildtest/__about__.py
      107  2020-02-02 00:00   beaker_buildtest/__init__.py
      136  2020-02-02 00:00   beaker_buildtest/agent.py
      467  2020-02-02 00:00   beaker_buildtest/context.py
       98  2020-02-02 00:00   beaker_buildtest-0.0.1.data/data/share/beaker/contexts/my_context.json
     1585  2020-02-02 00:00   beaker_buildtest-0.0.1.dist-info/METADATA
       87  2020-02-02 00:00   beaker_buildtest-0.0.1.dist-info/WHEEL
     1099  2020-02-02 00:00   beaker_buildtest-0.0.1.dist-info/licenses/LICENSE.txt
      801  2020-02-02 00:00   beaker_buildtest-0.0.1.dist-info/RECORD
---------                     -------
     4509                     9 files

Full set of files here: beaker-example.tar.gz

What I think is happening is that I think that your hatch/pip is installing a cached version of beaker_kernel instead of fetching from git.

Let's try clearing your caches and such and see if that fixes it with your existing setup: pip cache purge hatch env prune

Then try building again normally.

[brandomr]

Getting back to this since I'm trying to create a new context with it. I tried to run your example context in Docker and get a circular import error:

jupyter-1  | │ /jupyter/src/beaker_buildtest/context.py:2 in <module>                       │
jupyter-1  | │                                                                              │
jupyter-1  | │    1 from typing import Dict, Any                                            │
jupyter-1  | │ ❱  2 from beaker_kernel import BeakerContext, BeakerKernel                   │
jupyter-1  | │    3                                                                         │
jupyter-1  | │    4 from .agent import TestAgent                                            │
jupyter-1  | │    5                                                                         │
jupyter-1  | │                                                                              │
jupyter-1  | │ ╭────── locals ──────╮                                                       │
jupyter-1  | │ │  Any = typing.Any  │                                                       │
jupyter-1  | │ │ Dict = typing.Dict │                                                       │
jupyter-1  | │ ╰────────────────────╯                                                       │
jupyter-1  | ╰──────────────────────────────────────────────────────────────────────────────╯
jupyter-1  | ImportError: cannot import name 'BeakerContext' from partially initialized 
jupyter-1  | module 'beaker_kernel' (most likely due to a circular import) 
jupyter-1  | (/usr/local/lib/python3.10/site-packages/beaker_kernel/__init__.py)
jupyter-1 exited with code 1

any ideas?

EDIT: I think it's just the imports needing to be from beaker_kernel.lib.agent import BeakerAgent etc

2nd EDIT: that import fixed things but localhost:8888 throws a 404