New Kibana platform

[epixa]

The underlying platform in Kibana that we use to build new and existing features and plugins should be something we can independently document, test, and reason about. Building against the platform should be a straightforward and repeatable process from feature to feature. Our existing codebase doesn't really have a formal concept of a "platform", and the places that do have consistent patterns and conventions tend to leak internal details in their abstractions (e.g. the hapi server object).

This is one part of the overall "rearchitecture" effort we've been planning for a few months now.

Key objectives for the core platform

• The directory structure should reflect the platform architecture. With only a very high level understanding of the platform, a person should be able to reasonably find a relevant module in the system by navigating the file system.
• The platform must not be designed around the patterns or conventions of third party frameworks. To the contrary, the platform architecture should be as technology-agnostic as practical, and should help us absorb technological shifts in the JavaScript community over time by isolating dependencies between systems.
• The entire platform should be built around the notion of an asynchronous life cycle that moves between initialization, running, and stopping. This way, we can tie into any part of the Kibana life cycle at different integration points (including plugins) and avoid chasing down as many lifecycle related timing issues that we encounter now for lower level systems.
• Platform capabilities should be described in top level services which represent the core abstractions. These top level abstractions or services are considered necessary for building the types of features we want in Kibana rather than existing purely because they can.
• State is built and maintained through runtime execution rather than exposed as global singletons in JS modules. In other words, simply importing a module should not have side effects.
• The platform exists on both the browser and server, and though each runtime will have unique concerns for their particular environment, the platform architecture and conventions will be consistent in both places. A person familiar with interacting with or developing code for the platform in the browser should feel right at home on the server.
• The platform should expose a concrete, fully tested plugin system that does not expose internals or dependencies to plugins and can itself be treated as a first class feature of Kibana.
• Plugins should not be forced to use any specific technology other than native JavaScript.
• The platform code must be built (with something like webpack) independently of plugins so as not to share any dependencies that it doesn't explicitly expose via the plugin system.
• Plugin dependencies (on other plugins) should be explicit, and dependencies should be able to be optional.
• The platform architecture should be documented, including the plugin system itself.
• The platform should have full unit test coverage.

Systems

Systems are strictly isolated code paths. All of the functionality and capabilities of the Kibana process and web app is the result of composing multiple systems together into a single runtime. In practice, Kibana has a single core system and any number of plugin systems.

Systems rarely cross-link statically, though they will expose contracts which other system can use to communicate with one another during runtime. Examples of this communication can be anything from TCP access over a bound port to an explicit JavaScript object.

Every system follows an identical lifecycle as all other systems.

The capabilities of a system are spread across a set of services.

By their decoupled nature, all systems have certain qualities:

• Any system can be executed and functionally tested independently of all other systems.
• Systems must communicate with each other only at runtime and through well defined interfaces. State cannot reliably be shared implicitly through code imports.
• Systems can safely be served as distinct bundles.

System Contracts

Systems will expose any number of mechanisms that other systems can then use to communicate with them, the sum of which we call the system’s contract. Contracts can contain both implicit and explicit components.

The most common explicit contract is a structured JavaScript object that will expose observables for data and/or setters for modifying internal state of the system. Another example of an explicit contract would be a REST API.

An example of an implicit contract is an intentionally placed custom HTML attribute on a UI component with the expectation that other systems would directly access the global DOM element via that attribute.

Observables

Given the ever-changing nature of the state of systems, data should be exposed as observables rather than in their raw current form when dealing with JavaScript object contracts. This helps reinforce the notion that consuming systems must be designed to react to changes to the external data that they depend on.

Exposed observables should be minimally future-spec compliant, though static utilities for working with observables may be exposed to the platform globally.

Lifecycle

Every system follows the same lifecycle. This lifecycle is explicitly handled within each system, and it cannot be altered externally.

The interface(s) that a system has access to can and will change depending on its current lifecycle event. For example, a backend plugin system may have access to a function for registering REST endpoints during initialization, but it won’t have access to that function while it’s running or stopping.

At the same time, the interface(s) that a system has access to will not change within the context of a single lifecycle event.

There are three stages in a system’s lifecycle that execute in sequence:

Setup is when a system is going through its internal bootstrap and startup process. This is where you configure the bulk of the functionality in Kibana that will then be "kicked off" in the run phase.

Running is generally where Kibana runs most of the time.

Stopping is when a system is shutting down and will usually be used by a system to clean up after itself or gracefully stop ongoing operations.

Services

Services are technically implementation details of their system, so they are not directly exposed for use externally. If a system wishes to expose the capabilities of any given service, it will do so through one or more system contracts. This will be done as a testable client object that sits in front of the service so the public contract is a smaller subset of overall service capabilities, and the internal service can be evolved without necessarily breaking the public contract.

Like systems themselves, services are lifecycle aware. In fact, any system lifecycle function will primarily just be invoking the corresponding lifecycle functions on the services within.

Unlike systems, services can explicitly depend on other services within the system rather than an intermediary contract.

Diagrams

These diagrams serve as a visual guide to how the concepts of systems, services, and contracts interact. The actual contents of the images (service names, contract contents/origination/targets) should not be taken literally as they will not be kept up to date. Refer to the text of this issue instead.

The overall platform design. Blue boxes are discrete systems. Plugin zips can contain server and/or browser systems. The contract between core systems and plugins is a JavaScript object. The contract between the server system and browser system is a REST API.

The core server system. Teal boxes are independent services within the system. The contracts from the overall platform diagram each originate from one of those services.

The core browser system. It is structured exactly like the core server system, and it interacts with plugins in the same way as well. The actual services differ from the server due to the differing nature of servers and browsers, but some responsibilities are similar. Despite the same name for some services, this code is not directly shared between the two.

An example JavaScript object contract provided to an example plugin. Note that core and plugin contracts are passed as sibling arguments, so plugins can expose extension points as first class citizens. The available functions in the plugin mixin will be determined by the specified dependencies of the plugin itself, so in this example the plugin depends on xpackLicense, xpackSecurity, and timelion.

Rollout plan

The new platform will be built independently of the existing Kibana core. We'll verify that it continues to work as expected with system level integration tests, and then once it has reached sufficient capability, we will start moving our existing plugins over to the new platform. Once all of our plugins are moved over, we can "turn off" the old core.

We don't need to wait until the whole new platform is "done" before moving plugins over. Different plugins require different extension points, so we can start moving some plugins over while we complete the rest of the services.

A select few plugins (namely xpack security) will need to tie into both the new platform and the old core while they are both running.

[epixa]

/cc @spalger @kjbekkelund

[epixa]

Roll out plan for the implementation is probably the biggest unanswered question at this point. Ideally we can start using the new services as we create them in the context of the old system so this can be rolled out gradually over multiple releases. It'll take longer this way, but it'll significantly lower the risk here.

[spalger]

The entire platform should be built around the notion of an asynchronous life cycle that moves between initialization, running, and stopping.

I can't help but think we should consider "update" a lifecycle point. Maybe at this level "restart" is sufficient?

[epixa]

@spalger What's the use case?

[uboness]

I can't help but think we should consider "update" a lifecycle point. Maybe at this level "restart" is sufficient?

@spalger what do you mean by "update" and by "restart is sufficient"?

[cjcenizal]

Plugin notes

This is related to @rashidkpc's issue about plugin observations (https://github.com/elastic/kibana/issues/6313).

Also, when we redesign our plugin system, we should consider the correct interface for surfacing plugin content in the LocalNav, e.g. the Reporting link.

[jimgoodwin]

Do we need some goals around security ?

Do plugins ever interact with each other ? Do we need to state how the container would mediate ?

Also do we want lifecycle events like "pre-install","post-install","pre-remove","post-remove" so that plugins can encapsulate their own upgrade/uninstall functionality if they want ?

I presume that the design and implementation literal set of services and contexts provided to plugins is also another meta-issue that we need to draft ?

[jimgoodwin]

Also after looking at the UI cleanup issue #9708 I wonder if we need a statement here about component binding, watching, events, managing asynchronous behavior ?

[epixa]

Some notes (paraphrased) from prior discussions:

• REST service may not need to exist at all. We may be able to roll it into the HTTP service. -@uboness
• Web service may be overengineering -@uboness
• API service is too abstract of a name, so we should rename it -@uboness
• UI service purpose is not clear -@uboness
• We should explore using an Environment object (not a service) to abstract away access to browser services to remove explicit browser dependency entirely. -@uboness

[epixa]

I updated this ticket with all the latest information on the new platform including fleshed out definitions, example diagrams, and high level rollout plan. There still a few things to sort out, primarily around naming of services, but we can address details like this during implementation.

We should be ready to proceed with implementation.

[kimjoar]

We've decided to work on the new Platform in a feature branch for now, so my initial PR has been merged into the new-platform branch.

[fazil1987]

@epixa what's the current status of this new platform. Also, If I understand it rightly, with this new platform, even angular 2 front end browser plugin can be integrated. Right ?

[epixa]

@fazil1987 We're going to start merging foundational elements and new platform capabilities over the next couple Kibana versions, with new platform plugin support tentatively being introduced for 6.5. The big ticket items for this roll out are broken down as individual issues, and you can see most of them in this project: https://github.com/elastic/kibana/projects/2

You will be able to use whatever framework you want, so long as the dependencies don't have global side effects that would interfere with the functionality of things like the sidebar navigation. Our recommended setup is to use react, since that's what the rest of Kibana will be built in, and there's extensive tooling around things like EUI that is built for react, but technically speaking you can use whatever you want.

[fazil1987]

Thanks @epixa, it's great to know that it's technology agnostic.

Regarding the new platform integration that's planned to happen over the next couple Kibana version, what is the timeline we are looking at for the stable version of the platform? even a ball park figure should do.

[epixa]

@fazil1987 What features are you looking to use in your plugin? When the new platform is useful to you is likely not the same thing as when everything in Kibana is powered exclusively by the new platform.

[fazil1987]

@epixa my requirement is simple, that is to create simple web component using Polymer or Angular ( not decided yet ) to display some data outta elastic search

[epixa]

@fazil1987 You don't need to do that in the new platform, but if you did want to wait, you probably want to follow #18840. It's at least a couple months out.

[fazil1987]

@epixa by you don't need to do that in the new platform, you mean it's not possible to do it in that.

Also, If I understand it rightly, with #18840 we should be able to create our own front end plugins. Any ball park figure on the couple months could help

[epixa]

@fazil1987 Unfortunately we don't have a better estimate than "a couple of months". We're still working on some foundational elements that need to be in place first.

[fbaligand]

Is this new platform planned for K7 ?

[epixa]

@fbaligand The new platform isn't a single change that targets one particular version. The platform itself exists already, it first made its way into the product in 6.4 and in 6.5 it is the entry point on both the server and in the browser. It's here: https://github.com/elastic/kibana/tree/master/src/core

The new platform supports server-side plugins as of https://github.com/elastic/kibana/pull/25473.

Plugin support for UI code should be soon.

At this point, the limiting factor with the new platform is a lack of capabilities. We need to create new core services and migrate key existing functionality to new platform services (e.g. elasticsearch, saved objects, etc.) so that plugins in the new platform can actually do valuable things, since right now they can basically just validate config and log messages.

The :New Platform label is tracking the broader effort there, but https://github.com/elastic/kibana/projects/2 is a more targeted list of the most important things we need to complete in order to make the platform powerful enough to get rid of the legacy plugin system entirely.

[fbaligand]

Thanks for the explanation @epixa It is more clear to me.

[joshdover]

After nearly 4 years of planning, building, refactoring, and coordination, this project is now "complete". The Kibana Platform design has been implemented, all Elastic plugins have been migrated, and the legacy plugin system has been removed.