Platinum-only sub-feature privileges are hidden until Kibana is restarted

[jportner]

Kibana's sub-feature privileges are available with a Gold or better license by default, but some of them may only be available with a Platinum license. An example of this is the Reporting sub-features of the Dashboard feature:

https://github.com/elastic/kibana/blob/64f1dddcb05a4330145174c06df81b96b7cba8ca/x-pack/plugins/features/server/oss_features.ts#L491-L516

When you have a Platinum license installed and you go to create a new role, this is what the Kibana privileges screen looks like:

However, if you start with a Basic license and then upgrade to a Platinum or Trial license, the Platinum-only sub-feature privilege is missing:

This seems to happen intermittently, I'm anecdotally experiencing it about 60% of the time when installing a new license.

While Kibana is in this state, I checked the registered privileges with the GET _security/privilege API, and the feature_dashboard.generate_report sub-feature privilege is not present.

When you restart Kibana, the sub-feature privilege shows up as intended.

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[jportner]

This seems to happen intermittently, I'm anecdotally experiencing it about 60% of the time when installing a new license.

Testing this out more it's even less reliable than that. I've tried a couple dozen times and I'm only able to get Kibana into this state about 10% of the time. It appears to be some sort of race condition for the consumers of the license observable. My theory is that the race condition is as follows:

1. The AuthorizationService detects a license change and calls the registerPrivilegesWithCluster function
2. That function fetches the privileges from the PrivilegesService, which filters out any privileges and sub-feature privileges that have a minimumLicense specified (because it thinks it still has a Basic license) -- ~but it still returns sub-feature privileges because it isn't aware that those should only be registered when Kibana has a Gold+ license~ Edit: it appears that it's also supposed to filter out sub-feature privileges for Basic licenses... so I'm not sure about this
3. The AuthorizationService gets those privileges and goes to serialize them, but it doesn't filter out sub-feature privileges (because it is aware that it has a Gold+ license)
4. Kibana registers the new privileges in Elasticsearch and doesn't attempt to do so again until it restarts

I added some logging as follows:

diff --git a/x-pack/plugins/security/server/authorization/register_privileges_with_cluster.ts b/x-pack/plugins/security/server/authorization/register_privileges_with_cluster.ts
index f32835e7727..c4f07e861b6 100644
--- a/x-pack/plugins/security/server/authorization/register_privileges_with_cluster.ts
+++ b/x-pack/plugins/security/server/authorization/register_privileges_with_cluster.ts
@@ -53,8 +53,12 @@ export async function registerPrivilegesWithCluster(
   };

   const expectedPrivileges = serializePrivileges(application, privileges.get());
-
-  logger.debug(`Registering Kibana Privileges with Elasticsearch for ${application}`);
+  const expectedPrivilegeKeys = Object.keys(expectedPrivileges[application]);
+  logger.debug(
+    `Registering ${
+      expectedPrivilegeKeys.length
+    } Kibana Privileges with Elasticsearch for ${application}: ${expectedPrivilegeKeys.join(',')}`
+  );

   try {
     // we only want to post the privileges when they're going to change as Elasticsearch has

When Kibana starts with a Basic license, here's what is logged:

Registering 105 Kibana Privileges with Elasticsearch for kibana-.kibana: all,read,space_all,space_read,feature_savedObjectsTagging.all,feature_savedObjectsTagging.read,feature_savedObjectsTagging.minimal_all,feature_savedObjectsTagging.minimal_read,feature_actions.all,feature_actions.read,feature_actions.minimal_all,feature_actions.minimal_read,feature_stackAlerts.all,feature_stackAlerts.read,feature_stackAlerts.minimal_all,feature_stackAlerts.minimal_read,feature_graph.all,feature_graph.read,feature_graph.minimal_all,feature_graph.minimal_read,feature_fleetv2.all,feature_fleetv2.read,feature_fleetv2.minimal_all,feature_fleetv2.minimal_read,feature_fleet.all,feature_fleet.read,feature_fleet.minimal_all,feature_fleet.minimal_read,feature_canvas.all,feature_canvas.read,feature_canvas.minimal_all,feature_canvas.minimal_read,feature_maps.all,feature_maps.read,feature_maps.minimal_all,feature_maps.minimal_read,feature_osquery.all,feature_osquery.read,feature_osquery.minimal_all,feature_osquery.minimal_read,feature_observabilityCases.all,feature_observabilityCases.read,feature_observabilityCases.minimal_all,feature_observabilityCases.minimal_read,feature_ml.all,feature_ml.read,feature_ml.minimal_all,feature_ml.minimal_read,feature_uptime.all,feature_uptime.read,feature_uptime.minimal_all,feature_uptime.minimal_read,feature_siem.all,feature_siem.read,feature_siem.minimal_all,feature_siem.minimal_read,feature_securitySolutionCases.all,feature_securitySolutionCases.read,feature_securitySolutionCases.minimal_all,feature_securitySolutionCases.minimal_read,feature_infrastructure.all,feature_infrastructure.read,feature_infrastructure.minimal_all,feature_infrastructure.minimal_read,feature_logs.all,feature_logs.read,feature_logs.minimal_all,feature_logs.minimal_read,feature_apm.all,feature_apm.read,feature_apm.minimal_all,feature_apm.minimal_read,feature_discover.all,feature_discover.read,feature_discover.minimal_all,feature_discover.minimal_read,feature_visualize.all,feature_visualize.read,feature_visualize.minimal_all,feature_visualize.minimal_read,feature_dashboard.all,feature_dashboard.read,feature_dashboard.minimal_all,feature_dashboard.minimal_read,feature_dev_tools.all,feature_dev_tools.read,feature_dev_tools.minimal_all,feature_dev_tools.minimal_read,feature_advancedSettings.all,feature_advancedSettings.read,feature_advancedSettings.minimal_all,feature_advancedSettings.minimal_read,feature_indexPatterns.all,feature_indexPatterns.read,feature_indexPatterns.minimal_all,feature_indexPatterns.minimal_read,feature_savedObjectsManagement.all,feature_savedObjectsManagement.read,feature_savedObjectsManagement.minimal_all,feature_savedObjectsManagement.minimal_read,reserved_fleet-setup,reserved_ml_user,reserved_ml_admin,reserved_ml_apm_user,reserved_monitoring

When Kibana goes through a license upgrade and gets into this buggy state, here's what is logged:

Registering 119 Kibana Privileges with Elasticsearch for kibana-.kibana: all,read,space_all,space_read,feature_savedObjectsTagging.all,feature_savedObjectsTagging.read,feature_savedObjectsTagging.minimal_all,feature_savedObjectsTagging.minimal_read,feature_actions.all,feature_actions.read,feature_actions.minimal_all,feature_actions.minimal_read,feature_stackAlerts.all,feature_stackAlerts.read,feature_stackAlerts.minimal_all,feature_stackAlerts.minimal_read,feature_graph.all,feature_graph.read,feature_graph.minimal_all,feature_graph.minimal_read,feature_fleetv2.all,feature_fleetv2.read,feature_fleetv2.minimal_all,feature_fleetv2.minimal_read,feature_fleet.all,feature_fleet.read,feature_fleet.minimal_all,feature_fleet.minimal_read,feature_canvas.all,feature_canvas.read,feature_canvas.minimal_all,feature_canvas.minimal_read,feature_maps.all,feature_maps.read,feature_maps.minimal_all,feature_maps.minimal_read,feature_osquery.all,feature_osquery.read,feature_osquery.minimal_all,feature_osquery.minimal_read,feature_osquery.live_queries_all,feature_osquery.live_queries_read,feature_osquery.run_saved_queries,feature_osquery.saved_queries_all,feature_osquery.saved_queries_read,feature_osquery.packs_all,feature_osquery.packs_read,feature_observabilityCases.all,feature_observabilityCases.read,feature_observabilityCases.minimal_all,feature_observabilityCases.minimal_read,feature_ml.all,feature_ml.read,feature_ml.minimal_all,feature_ml.minimal_read,feature_uptime.all,feature_uptime.read,feature_uptime.minimal_all,feature_uptime.minimal_read,feature_siem.all,feature_siem.read,feature_siem.minimal_all,feature_siem.minimal_read,feature_securitySolutionCases.all,feature_securitySolutionCases.read,feature_securitySolutionCases.minimal_all,feature_securitySolutionCases.minimal_read,feature_infrastructure.all,feature_infrastructure.read,feature_infrastructure.minimal_all,feature_infrastructure.minimal_read,feature_logs.all,feature_logs.read,feature_logs.minimal_all,feature_logs.minimal_read,feature_apm.all,feature_apm.read,feature_apm.minimal_all,feature_apm.minimal_read,feature_discover.all,feature_discover.read,feature_discover.minimal_all,feature_discover.minimal_read,feature_discover.url_create,feature_discover.store_search_session,feature_discover.generate_report,feature_visualize.all,feature_visualize.read,feature_visualize.minimal_all,feature_visualize.minimal_read,feature_visualize.url_create,feature_dashboard.all,feature_dashboard.read,feature_dashboard.minimal_all,feature_dashboard.minimal_read,feature_dashboard.url_create,feature_dashboard.store_search_session,feature_dashboard.download_csv_report,feature_dev_tools.all,feature_dev_tools.read,feature_dev_tools.minimal_all,feature_dev_tools.minimal_read,feature_advancedSettings.all,feature_advancedSettings.read,feature_advancedSettings.minimal_all,feature_advancedSettings.minimal_read,feature_indexPatterns.all,feature_indexPatterns.read,feature_indexPatterns.minimal_all,feature_indexPatterns.minimal_read,feature_savedObjectsManagement.all,feature_savedObjectsManagement.read,feature_savedObjectsManagement.minimal_all,feature_savedObjectsManagement.minimal_read,reserved_fleet-setup,reserved_ml_user,reserved_ml_admin,reserved_ml_apm_user,reserved_monitoring

When Kibana goes through a license upgrade correctly and/or starts with a Platinum+ license, here's what is logged:

Registering 122 Kibana Privileges with Elasticsearch for kibana-.kibana: all,read,space_all,space_read,feature_savedObjectsTagging.all,feature_savedObjectsTagging.read,feature_savedObjectsTagging.minimal_all,feature_savedObjectsTagging.minimal_read,feature_actions.all,feature_actions.read,feature_actions.minimal_all,feature_actions.minimal_read,feature_stackAlerts.all,feature_stackAlerts.read,feature_stackAlerts.minimal_all,feature_stackAlerts.minimal_read,feature_graph.all,feature_graph.read,feature_graph.minimal_all,feature_graph.minimal_read,feature_fleetv2.all,feature_fleetv2.read,feature_fleetv2.minimal_all,feature_fleetv2.minimal_read,feature_fleet.all,feature_fleet.read,feature_fleet.minimal_all,feature_fleet.minimal_read,feature_canvas.all,feature_canvas.read,feature_canvas.minimal_all,feature_canvas.minimal_read,feature_canvas.generate_report,feature_maps.all,feature_maps.read,feature_maps.minimal_all,feature_maps.minimal_read,feature_osquery.all,feature_osquery.read,feature_osquery.minimal_all,feature_osquery.minimal_read,feature_osquery.live_queries_all,feature_osquery.live_queries_read,feature_osquery.run_saved_queries,feature_osquery.saved_queries_all,feature_osquery.saved_queries_read,feature_osquery.packs_all,feature_osquery.packs_read,feature_observabilityCases.all,feature_observabilityCases.read,feature_observabilityCases.minimal_all,feature_observabilityCases.minimal_read,feature_ml.all,feature_ml.read,feature_ml.minimal_all,feature_ml.minimal_read,feature_uptime.all,feature_uptime.read,feature_uptime.minimal_all,feature_uptime.minimal_read,feature_siem.all,feature_siem.read,feature_siem.minimal_all,feature_siem.minimal_read,feature_securitySolutionCases.all,feature_securitySolutionCases.read,feature_securitySolutionCases.minimal_all,feature_securitySolutionCases.minimal_read,feature_infrastructure.all,feature_infrastructure.read,feature_infrastructure.minimal_all,feature_infrastructure.minimal_read,feature_logs.all,feature_logs.read,feature_logs.minimal_all,feature_logs.minimal_read,feature_apm.all,feature_apm.read,feature_apm.minimal_all,feature_apm.minimal_read,feature_discover.all,feature_discover.read,feature_discover.minimal_all,feature_discover.minimal_read,feature_discover.url_create,feature_discover.store_search_session,feature_discover.generate_report,feature_visualize.all,feature_visualize.read,feature_visualize.minimal_all,feature_visualize.minimal_read,feature_visualize.url_create,feature_visualize.generate_report,feature_dashboard.all,feature_dashboard.read,feature_dashboard.minimal_all,feature_dashboard.minimal_read,feature_dashboard.url_create,feature_dashboard.store_search_session,feature_dashboard.generate_report,feature_dashboard.download_csv_report,feature_dev_tools.all,feature_dev_tools.read,feature_dev_tools.minimal_all,feature_dev_tools.minimal_read,feature_advancedSettings.all,feature_advancedSettings.read,feature_advancedSettings.minimal_all,feature_advancedSettings.minimal_read,feature_indexPatterns.all,feature_indexPatterns.read,feature_indexPatterns.minimal_all,feature_indexPatterns.minimal_read,feature_savedObjectsManagement.all,feature_savedObjectsManagement.read,feature_savedObjectsManagement.minimal_all,feature_savedObjectsManagement.minimal_read,reserved_fleet-setup,reserved_ml_user,reserved_ml_admin,reserved_ml_apm_user,reserved_monitoring

Seeing as I can't reproduce this bug reliably, I'm having trouble getting more information than this. But it seems like there's a license-observable-related race condition. I'll have to table this investigation for now but hopefully we can revisit it soon. It does make me question what other observable-related race conditions we've introduced and are simply unaware of...

Also: I think I found a bug (?), the graph feature requires a platinum license, but the graph_feature privilege is registered even when Kibana only has a Basic license. It seems like the featurePrivilegeIterator should filter this out before it is returned from the PrivilegeService...