In-section filter for sections where multiple families and instances exist

[hugovalente-pm]

Goal

We need to provide flattened composite charts on certain sections the Single Node view page. The charts in these section will be presented as composite charts and through the use of an in-section filter bar (similar like k8s) users will be able to decide how to slice and dice the data on these.

The in-section filter bar will, potentially, have the following options to filter:

• families
• instances
• nodes
• dimensions

The in-section filter bar needs to be sticky to the section and will only "slide-away" when the top of the last chart pushes this to go over the viewport.

Some examples of sections where this will apply are:

• Disks - families on in-section filter bar
• Mount Points - families on in-section filter bar
• Network interfaces - families on in-section filter bar
• mysql - instances on in-section filter bar
• Cockroach DB - instances on in-section filter bar
• VerneMQ - instances on in-section filter bar
• Applications / User Groups / Users / systemd - dimensions need to be on the filter bar

The rules to apply to this are:

• if for a given context (section) more than one family exists then these are candidates to be displayed with an in-section filter over families:

"disk.md126": {
    "id": "disk.md126",
    "name": "disk.rootfs",
    "family": "/",
    "context": "disk.io",
    "priority": 2000,
    "chart_labels": null,
    "chart_type": "area"
},
"disk.md127": {
    "id": "disk.md127",
    "name": "disk.md127",
    "family": "md127",
    "context": "disk.io",
    "priority": 2000,
    "chart_labels": null,
    "chart_type": "area"
} 

* families
  * /
  * md127

• if for a given context (section) more than one instance exists these are candidates to be displayed as an in-section filter bar for instances:

The approach to take is apply some heuristics with parsing metrics names (applied on `main.js`  and `dashboard_info.js`) to identify the `instance-name` and `instance-type`. 

This is to be applied for now on metrics coming from:
* python.d.plugin (apps/services)
* go.d.plugin (apps/services, including prometheus and HTTP checks)
* cgroups.plugin (containers)

Expected results for instance-types and instances:
* k8s cntr
   * app cloud-agent-data-ctrl-service-0-547bbfd9d9-ln5f9 cloud-agent-data-ctrl-service
   * app cloud-alarm-config-mqtt-input-service-2-78cfbd7766-hrs6q cloud-alarm-config-mqtt-input-service
   ...
* Cockroach DB
   * cockroachdb-infra crdb-agent-cockroachdb-2 db tcp 8080
   * cockroachdb-infra crdb-cockroachdb-4 db tcp 8080
   ...
* Http Check
   * Bangalore Demo Site (CloudFlare)
   * Frankfurt Demo Site (CloudFlare)
   ...
* prometheus
   * cpeh exporter local
   * php-fpm exporter
* fping
   * www.google.com
   * netdata.cloud

Note requirements had some from netdata/product#2349

• in some particular cases we need to expose the dimensions on the in-section filter bar these will apply for:
  • Applications
  • User Groups
  • Users
  • systemd

[thiagoftsm]

Hello @hugovalente-pm ,

Only to confirm that I understood the propose considering old meetings I had with PMs in the past. When you talk about mysql, Cockroach DB, and VerneMQ, does this mean that we will have per application submenu? Are we going to keep apps and services submenu with this new approach?

Best regards!

[hugovalente-pm]

hey @thiagoftsm , at moment yes we will keep the apps and services

we are trying to see a way to tackle this Single Node Dashboard should be replaced with Overview Dashboard filtering just one node #278 and for that we need to expose some information that isn't available on the to the FE on the Overview page and besides we want to improve the usability of this, so if you are on a section with multiple VerneMQs or other applications you should have a way to get data for a specific one - still trying to find the best approach to expose if UI/UX-wise

[hugovalente-pm]

@novykh I've capture the requirements for the in-section filter bar, please take a look and let me know what you think

[novykh]

Yeah they look good. My only asterisk is about the instances, we should check when we start working on this how to extract the instance consistently. (Maybe we are already doing this, but need to check the code)

[hugovalente-pm]

I'm not sure this help but for example the instances and the famlies can be seen on the list of the charts on a composite chart, e.g.:

Disks

Cockroach DB

[ktsaou]

Yeah they look good. My only asterisk is about the instances, we should check when we start working on this how to extract the instance consistently. (Maybe we are already doing this, but need to check the code)

probably we need a ticket just for that.

[hugovalente-pm]

We had this one netdata/product#2349 but this was more towards moving these instances as levels on the TOC, I can open a new one if it helps breakdown things @novykh

[hugovalente-pm]

@christophidesp just a heads-up, with the conversations about the centralized grouping it was mentioned the potential need to have that also available on each section. here the question will be:

• shall we have just the in-section grouping available then? my concern it could still be not noticeable by the users
• would it be too noisy/confusing to have the grouping available both on top, in-section and on the chart?

not sure if we can easily answer just by ourselves or we need to try to get feedback from users

[christophidesp]

@hugovalente-pm I think that we should evaluate the value of having an overall "Group by" or a per section one. Either of these for me makes sense, but having both of them (along with the on chart selection), might over-complicate things. As for the in-section "Group by", I think that if it is at the same level with the title and to the far right, it will be noticeable. I will create a quick mockup so we can check.

[hugovalente-pm]

The in-section filter, I think, will be our end-game since we'll have more options related to the context of the section you are looking at.

the overall one was identified before we started the in-section conversations and we were trying to see if it would an easy/fast delivery, which it seems it isn't so probably we should focus our efforts on the in-section and try to make it noticeable as you say

[hugovalente-pm]

As discussed on the Overview tab - centralized grouping netdata/product#2081 we will need to have the Group by selection available as well on the in-section filter

[netdata-community-bot]

This issue has been mentioned on the Netdata Forums. There might be relevant details there:

https://community.netdata.cloud/t/group-docker-containers-in-menu/2797/2

[novykh]

@hugovalente-pm Any news on that?

[hugovalente-pm]

@novykh the in-section filters are on-hold, the focus is to work first on the:

• Global filter enhancements - https://github.com/netdata/product/issues/2778
• Add the composite chart filters - https://github.com/netdata/netdata-cloud/issues/434

the goal is to minimize spreading filter functionality in multiple places, if/when it is confirmed that in-section filters are needed we will revisit this

[hugovalente-pm]

we just had some user feedback about the need to have something like this available on custom dashboard this needs to be revisited and brought back into discussions, since it would make the usage on custom dashboards more flexible

for reference https://discord.com/channels/847502280503590932/847502280503590935/1088492426668290140