search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.65k stars 8.23k forks source link

[Dashboards][META] Dedicated HTTP API for managing Dashboards for GitOps #174497

Open thomasneirynck opened 10 months ago

thomasneirynck commented 10 months ago

tl:dr; Add a dedicated http REST CRUD API to manage Dashboards to support infra-as-code use-cases.

Current approach: using the saved object API

Dashboards can be programmatically managed with the saved object API. This is a general purpose API for all resource types in Kibana (e.g. Dashboards, individual visualizations, saved searches, etc...) supporting basic CRUD operations.

A typical dashboard would consist of some dashboard metadata (title, description, layout, ...), and a collection of panels, each with panel specific contents (either by value or reference).

For example:

Adoption

Several tools provide dashboards-as-code integrations by using the Saved Object API.

Limitations

There are a number of limitations with using Saved Objects API.

General limitations

Dashboard specific limitations

Proposal

Introduce a dedicated REST API to perform Dashboard CRUD, and higher level, domain specific operations.

Incremental approach

Given the scope of the work, we would propose an incremental approach of delivery. Below a rough ordering of priority.

1) Extract basic functionality

CRUD for dashboards

This is similar to the current SO, support, but

for example:

PUT http://kibanaproject/{spaceid?}/dashboard/{id}
Accept: application/json
Content-Type: application/json
elastic-version: 2024-01-01
{
     options: { ...}
     panels: [
         { layout: ..., content: ...},
         { layout: ..., content: ...},
     ]
}

2) Add content validation

Add content validation for Dashboard contents

3) Fully type panels

Fully type API to include all panels as a first level object.

Scope limits

To limit scope, consider only:

Comparisons

Similar functionality already exists.

Internal comparisons

Several other resource types in Kibana already have a dedicated http API. They support basic CRUD, but also domain specific operations.

External comparisons

Peer products also provide dedicated http APIs to manage dashboards.

Describe a specific use case for the feature:

Infa-as-code use-cases and machine integrations like Terraform providers, Ansible, ...

elasticmachine commented 10 months ago

Pinging @elastic/kibana-presentation (Team:Presentation)

Kushmaro commented 10 months ago

Thanks for opening this issue @thomasneirynck !

Will the new API support already existing kibana dashboards? I'm afraid that "neglecting current saved objects" and moving to a new API that only supports newly created dashboards, would leave a significant chunk of customers behind, and will see very slow adoption.

thomasneirynck commented 10 months ago

@Kushmaro wrt https://github.com/elastic/kibana/issues/174497#issuecomment-1884586387

Will the new API support already existing kibana dashboards?

It should.

In the description, I added some possible limitations to keep scope smaller (e.g. limit to by-value panels only). These can be re-evaluated, as they may conflict with current use (e.g. by-reference panels remain popular). But in general the API would definitely have to support Dashboards that were created previously.

Kushmaro commented 10 months ago

Thanks @thomasneirynck . Pardon my Kibana ignorance, are by-value panels such that (i assume) contain only a specific type of value rather than a result of a search query?

nreese commented 10 months ago

Pardon my Kibana ignorance, are by-value panels such that (i assume) contain only a specific type of value rather than a result of a search query?

by-value and by-reference references how panel state is saved within the dashboard state.

by-value stores panel state directly in the dashboard.

Benefits include:

  1. Faster loading. The dashboard does not have to load panel state with a network call.
  2. Panel state is local to the dashboard so the panel will not be negatively impacted by changes to a saved object.

Disadvantages include:

  1. Panel state can not be shared between dashboards. If the panel needs to be updated, then each panel in each dashboard has to be maintained separately.

by-reference links panel state to a saved object via a saved object id.

Benefits include:

  1. Being able to reuse saved objects between dashboards.

Disadvantage of:

  1. Slower dashboard loading (since dashboard has to load panel state with another network request).
  2. Any changes to the underlying saved object will show up in all dashboards. This may be a negative if a change is made for one dashboard but not optimal for all dashboards.
thomasneirynck commented 9 months ago

thanks @nreese for the thorough explanation.

@Kushmaro , just to write add to it here explicilty as well, but none of the panel-"state" (whether by value or by reference) includes the actual data of the charts. That is in both cases a query to Elasticsearch. The state is things like line-color of a chart, the query (but not the result of a query), ...

vadimkibana commented 9 months ago

+1 on this effort. Just a side note on terminology, one way to talk about this is to call this the public API for dashboards.

Kushmaro commented 9 months ago

thanks @nreese @thomasneirynck , yep now I understand. Can you talk through about what are some of the next steps here?

roshan-elastic commented 9 months ago

@carlyrichmond cc

thomasneirynck commented 9 months ago

@Kushmaro this is on the horizon, but no clear date wrt delivery (as of writing)

would be great to collaborate on the details of the requirements as well.

Kushmaro commented 9 months ago

ack @thomasneirynck i'd be happy to collaborate on reqs do any of the kibana pm / leads have a doc started already? or is this the main thing we have as of now?

thomasneirynck commented 9 months ago

@Kushmaro work needs to be started on this one still.

IanLee1521 commented 7 months ago

@teresaalvarezsoler and I were talking about this a bit today, and I wanted to throw a particular use case for this where it could help a lot, namely if this was a viable way to do some "advanced editing" of an existing dashboard.

Example: If I have a dashboard that I'm building, and I want to put 4 panels horizontally that are the same size, that is currently quite tedious to move them all around and resize them to get them the same, and so that they fit in the box. What I'd much rather do is be able to edit some underlying code (e.g. the panels: {...} suggestion above) to set each of the panels to have width: "25%" and then just be done.

This is something like what Splunk provides today with it's ability to edit the raw XML that controls the dashboard. That provides a lot of power for sizing, and embedding other things like variables in particular panels.

A bit selfishly, but I'll also just mention that we wrote a tool (https://github.com/LLNL/elastic-stacker) for exporting / importing saved objects and other REST API accessible objects to allow collaboration and sharing of dashboards, panels, etc, so something like what you're describing above sounds great to me.

thomasneirynck commented 7 months ago

thx @IanLee1521 for the feedback. Agreed that relative placement would be a nice feature.

Thank you for linking https://github.com/LLNL/elastic-stacker and confirming the use-case!

TinaHeiligers commented 6 months ago

Core fully supports Introducing a dedicated REST API to perform Dashboard CRUD. We have advocated this approach for the upcoming removal of the SO HTTP APIs.

We're also aware that in the Dashboards case, it's not as "simple" as other saved objects, given the by-reference/by-value complexity.

Would an initial approach of duplicating the existing HTTP APIs in core and scoping them down to only dashboard saved object types be feasible?

We're targeting v9 for complete removal. If we delay any longer, there's the risk that folks carry on consuming them, severely limiting what we can do in the next major.

thomasneirynck commented 6 months ago

thx @TinaHeiligers ;

we are looking at something similar to duplicated routes, similar to what you are describing.

Think something along the lines of:

POST https://mykibana.foobar.com/{space?}/dashboard
{
    title: "My dashbaord"
    panels: [
        {
            type: "maps",
            layers: [ ... ]
        }
        ]
}

or 

PUT https://mykibana.foobar.com/{space?}/dashboard/aGFzaA==
{
    title: "My dashbaord with more charts"
    panels: [
        {
            type: "maps",
            layers: [ ... ]
        }, 
                {
            type: "visualization",
                          ...       
                }
       ]
}

So this would be quite similar to the existing (deprecated) SavedObjectAPI.

Unlike the existing SO-API, what we do not look at supporting is the idea of a single package import/export. So suppose for the by-ref use-case, it would require two calls. One for the referenced object, another for the dashboard-object that references that object.

thomasneirynck commented 2 months ago

@nickpeihl and I discussed offline. The first PR should ideally include the following:

1) add public route registration to Dashboard-server (/content/dashboard/...). This should not be an abstraction in content-management 2) introduce new de-stringified content-type for Dashboards (schema mapping effort). This is a new v3 content-type, virtually identical to the existing one, but it has destringified properties.
3) modify DashboardStorage-class to do translation between the CM-type and SO-type. This is the transofrmation between the CM-type from (2) sent over the wire, and the SO-type that is being stored. 4) Ensure Dashboard-app is using v3 internally over the CM-rcp layer and have transformations from v2->v3 in content-management. Using the same type for internal use and for public use will greatly reduce the mental overhead in trying to juggle multiple types.

Later PRs then can add refinement: