Concerns around developer experience for plugin and package development

[jasonrhodes]

In my attempts to make dependency management simpler for Kibana engineers in Observability, I've run into some resistance to options such as smaller-scoped plugins or, even better, single-concern, stateless packages with injected dependencies. Some of that resistance comes from a healthy desire to avoid over-abstraction or early abstraction, but much of the resistance I've heard seems to reference problems with the developer experience around splitting code up in Kibana.

Below I've listed a few of the concerns I've heard, as best as I understand them, but I'd like to get others' input on this. I'd like to use this issue as a place to collect these kinds of concerns and talk with Kibana platform teams to see how we can improve this experience in small, simple ways (avoiding huge refactors that eat up months of roadmap).

General Concerns

Build times and procedures

When you break out code into a package, especially, how is that package rebuilt on changes? If you are using the package in your plugin code, and you're developing a change that crosses that boundary, will the package rebuild automatically on code change? When (if ever) do you need to run special commands such as node scripts/build_kibana_platform_plugins.js, etc? Where is this process documented, and are there clear examples of how and when to do this?

Multi-part test, lint, and type check runs

From @sqren:

As an aside, one problem with the tiered approach that will become more emphasised as plugins are broken into smaller pieces is the ability to run scripts across multiple plugins. Take the apm plugin as an example. As a first step it has been broken into apm_data_access and apm. I now have bash aliases for these commands:

Linting:

node scripts/eslint.js x-pack/plugins/apm && node scripts/eslint.js x-pack/plugins/apm_data_access

Type checking:

node scripts/type_check.js --project x-pack/plugins/apm/tsconfig.json && node scripts/type_check.js --project x-pack/plugins/apm_data_access/tsconfig.json

It’s more painful when it comes to jest. Previously I had this alias:

apm-jest() {
    echo "Running jest"
    cd ~/elastic/kibana/ && node scripts/jest.js --config x-pack/plugins/apm/jest.config.js "$@" && cd -
}

That meant that I could run all apm tests using apm-jest. I could also run individual tests using apm-jest my_test_file . This is no longer possible when there are multiple plugins and I have yet to find a solution for it.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[rudolf]

The plugin jest problem applies to packages too, there's no way to easily run jest on e.g. all of Core's packages or all packages in a directory.

[mistic]

@jasonrhodes thanks for putting together this issue and share what you found as concerns with us.

In the past we already heavily looked into the things you mentioned and its definitely part of the future things we want to accomplish.

For the Multi-part test, lint, and type check runs section the answer will be to apply into Kibana an extensive build system like Nx. That will allow us to run any kind of task in a more granular way with cache. However, in order to be ready for it, we need to develop ground work like being able to run typechecking per project, lint and unit testing per project. Currently we still have a blocker in the way for this which is remove what it remains on our system from Bazel and check how we could build a couple stateful packages we have reliably with Nx , but we will get there in the future.

In relation to the Build times and procedures and the package uses, right now if you change a package the change will be reflected without any further action wether you have already run yarn start or about to. The only place where you want to make sure to run node scripts/build_kibana_platform_plugins.js is before running the functional test runner. In any case the current optimizer is a highly complex piece of software developed on top of webpack. In the future we want to get time to R&D and experiment building the application differently from what we are doing today, exactly to allow for a better cache as well as a better management of shared dependencies between bundles. We want to try to implement this using rspack but there are difficult requirements (like support for building external third party packages) that we will have to be able to achieve in order to move on with such project.

Hope that small summary provides useful information and also let you all know that we are following those things closely for the future.

[hardfist]

@mistic if you want to try rspack, we(Rspack team) are glad to help, as you may know we help discord move from webpack to rspack https://twitter.com/aweary/status/1716890502682018290, we're glad to help kibana move from webpack to rspack.

[mistic]

@hardfist that would be fun and helpful. Whats the best way to get in touch with you ? Your official Discord channel?

[hardfist]

@hardfist that would be fun and helpful. Whats the best way to get in touch with you ? Your official Discord channel?

@mistic yeah, we can also setup a private support channel for you if you need.