[Cloud Security] Hide internal Fleet server host URL and ES Output

[eyalkraft]

Motivation As part of Agentless CSPM we introduced Internal fleet server host url and internal ES output on serverless projects. This results with a confusing UX for serverless users using fleet.

Definition of done

• [ ] Hide the internal Fleet URL and ES output
• [ ] To be reviewed by @joshdover or someone else from @elastic/fleet

Implementation Proposal:

• Add an is_internal or is_hidden attribute to Fleet Server Host and Output, That can be preconfigured for Hosts/Outputs defined in kibana.yml.
• Change /api/fleet/outputs and /api/fleet/fleet_server_hosts to filter out internal results. (Maybe filter here for outputs and here for Fleet Hosts)

Out of scope

• Change the project controller to configure the Internal fleet host URL and internal ES output with the new is_internal attribute. This will be taken care of by @olegsu.

Related tasks/epics

• https://github.com/elastic/project-controller/pull/428

[elasticmachine]

Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security)

[maxcold]

prioritizing https://github.com/elastic/security-team/issues/7557 vs this one as the incorrect agent version is blocking the Cloudformation (and probably other methods from working) while this one is mostly confusing UX. Plus @CohenIdo seems to have started working on https://github.com/elastic/security-team/issues/7482#top which covers more or less the same problem. If we introduce is_internal or smth in one of the tasks, it should be straightforward to filter them in the fleet ui. @kfirpeled let me know if my reasoning is off

[kfirpeled]

@maxcold recently @CohenIdo suggested a fix to https://github.com/elastic/security-team/issues/7482 It appears we had many tickets on similar issue

Lets sync on that when you are available to understand what exactly is duplicated here

[eyalkraft]

These are not duplicate.

This issue has to do with Fleet UX, unrelated to cloudformation or even cloudsecurity

[maxcold]

@kfirpeled @eyalkraft The issues are related in the sense that here we want to hide the internal host and in https://github.com/elastic/security-team/issues/7482 we want this internal host not to be picked up for Cloudformation/ Cloud Shell params. True that these are separate issues in two different parts of Kibana, but my thinking was that depending on the implementation the logic of "ignoring" the internal host could have been shared. I will check the PR @CohenIdo created to see if my thinking makes sense. Anyway hiding the internal fleet server seems less critical than the incorrect agent version on our Cloudformation/Cloud Shell (https://github.com/elastic/security-team/issues/7557) that's why I want to fix the agent version first

[kfirpeled]

moved to blocked - waiting for agentless epic planning

[maxcold]

before we merge

• https://github.com/elastic/project-controller/pull/680

we need to wait till both

• https://github.com/elastic/kibana/pull/175983
• https://github.com/elastic/kibana/pull/175546

are deployed to all envs of kibana in serverless