[Change Proposal] Adding threatintel category option

[P1llus]

With an increase in planned threat intel related packages, both new and converted from filebeat modules, I think at some point in the near future we would need a dedicated category for this.

With category I am talking about this specific list here: https://github.com/elastic/package-spec/blob/master/versions/1/manifest.spec.yml#L14

From my understanding this is also impacting the integration overview page, and maybe how the fleet UI handles it?

Implementation

• [x] Support in the spec https://github.com/elastic/package-spec/pull/366
• [x] Support in Fleet https://github.com/elastic/kibana/issues/135525

[mtojek]

@mostlyjason @ruflin we need confirmation/approval on your side that is the way we'd like to go.

[P1llus]

Relates: https://github.com/elastic/integrations/issues/1328

[ruflin]

I remember when we defined the categories we agree to be very restrictive to add more categories. It is also the reason we put it into the spec to have some guard rails in place. Great to see it works :-) @tbragin I remember you were involved in this initial discussion from the product side. Could you chime in here if we can / should add a category?

[maxcold]

Hey all, I want to revive the discussion on this issue and connect it to the email discussion I started with the Fleet Team and Ecosystem Team.

In the new team Protections Experience, we are working on new capabilities for Threat Intelligence Analyst Persona in Security Solution. One of the first features we plan to deliver in 8.4 is a Threat Intelligence page with a list of IoCs. If a user doesn't have any IoCs in the system we want to help them by providing the link to the Integrations page where all the Threat Intelligence integrations are grouped together so that the user is not overwhelmed with more than two hundred integrations available in the Security category. The new category would also help the existing TI capabilities, namely the link to Integrations from the Threat Intelligence block on the Security Overview page.

There are currently 9 integrations that would go into this new "Threat Intelligence" category and there are features in the Security Solution which don't work without the data ingested from these integrations.

Here is the issue in our backlog related to that https://github.com/elastic/security-team/issues/4137 with more information.

We are aware that currently this grouping is solved by linking to the Integrations page with the search query "threat intelligence" prefilled but this solution is very fragile as it relies on these words being present in the short description of the integration. For example, it seems not to cover Mimecast integrations where the "Mimecast Threat Intel Malware Grid" feed can be enabled. Having a separate category for integrations providing access to Threat Intelligence feeds seems to be a more robust solution longer-term.

I see that there is some pushback to having new categories, but as currently we have categories with only one integration in them, I'd push for not waiting for the ideal longer-term solution for new "Threat Intelligence" category to be created.

How can our team help this happen?

[jsoriano]

@akshay-saraswat wdyt?

[SourinPaul]

Thanks for re-surfacing this @maxcold

Team, just adding my upvote 👍 here 😄

A dedicated TI category will help user workflow and will also simultaneously highlight our use-case-focused development for Threat Intelligence. Are there major disadvantages, besides containing the sprawl of pre-defined categories? I did a bit of digging, but could not surface one.

Looking into the future, I'm assuming as we bring additional functionalities into our solution(e.g., Threat Intelligence now, Entity Analytics packages in the future), our integration categories will have to expand to accommodate the growth in these use-cases.

Btw, I also really liked the possibility of scaling categorization via the ECS field but will leave it up to you, the experts, to define an optimal solution.

[mtojek]

where all the Threat Intelligence integrations are grouped together so that the user is not overwhelmed with more than two hundred integrations available in the Security category. The new category would also help the existing TI capabilities, namely the link to Integrations from the Threat Intelligence block on the Security Overview page.

In this case, it would be a temporary solution until more TI integrations appear. I wouldn't consider this as a strong argument.

For example, it seems not to cover Mimecast integrations where the "Mimecast Threat Intel Malware Grid" feed can be enabled.

Does it help if you adjust package manifests, description, title, etc.?

[maxcold]

In this case, it would be a temporary solution until more TI integrations appear. I wouldn't consider this as a strong argument.

The main issue is not the amount of the Security integrations, but the fact that Threat Intelligence ones are very special integrations that provide data into very specific Threat Intelligence features in the Security Solution. If there are 200 Threat Intelligence integrations that's ok that they are grouped. Plus we don't expect such a growth in this specific category in the near future anyway

Does it help if you adjust package manifests, description, title, etc.?

Mimecast integration currently has the following description "Collect logs from the Mimecast API with Elastic Agent". In the configuration of this integration, it is possible to enable 8 different types of logs, only 2 of which are the Threat Intelligence feeds. Probably, it's possible to make the description "Collect logs, including Threat Intelligence ones, from the Mimecast API with Elastic Agent" or something along these lines, but tbh I don't see it as a good solution to the user problem:

1. every time a new integration is added someone should remember to add these magic words "Threat Intelligence" in the description, otherwise users won't see the integration when navigating from our Threat Intelligence capabilities
2. every time the Threat Intelligence features are presented, the presenter would need to say smth like "you can enable Threat Intelligence feeds by going in Integrations and searching by words "threat intelligence" in the search bar there (and hope that all TI integrations have the correct description)"

To illustrate how the current situation looks for users, here is what they see at the moment in the TI block, and will see more in our new Threat Intelligence platform when they haven't enabled any TI integrations yet The "Enable Sources" button links to the Integrations page with the search "threat intelligence", that's why Mimecast is missing there currently. The solution is quite hacky anyway

[akshay-saraswat]

Do we have any data on searches or enhancement requests for "threat intelligence"? Would be interesting to see if users actually need the threat intelligence category. Categories are used for exploration and filtering purposes. Do users want to explore or filter by "Threat Intelligence"?

I tried looking at it from a competitive angle. Splunk seems to be the only competitor that has created a category for "Threat Intel".

Are there major disadvantages, besides containing the sprawl of pre-defined categories? I did a bit of digging, but could not surface one.

The sprawl of categories is the biggest disadvantage. If there are 200 categories, it will fail the purpose of having categories for organizing and filtering integrations in the first place. We can probably look into hierarchical categories.

[maxcold]

@akshay-saraswat thanks for asking these questions!

Do we have any data on searches or enhancement requests for "threat intelligence"? Would be interesting to see if users actually need the threat intelligence category.

It is a good question. If the decisions on adding existing categories were based on this data, do you know where one can find it or who can help with getting it?

Categories are used for exploration and filtering purposes. Do users want to explore or filter by "Threat Intelligence"?

Is it the only purpose categories serve? For me it looks like there is also a need in categories for better linking between features and integrations enabling these features. At least that's the main reason for our request of having this new category created. Is there a better way of solving this use case without categories?

I tried looking at it from a competitive angle. Splunk seems to be the only competitor that has created a category for "Threat Intel".

Great point, it definitely makes sense to look at what our competitors do. I checked Datadog and Logz.io, it seems that they simply don't provide the capability of enabling 3rd party Threat Intelligence feeds, that's why they don't have this category in their integrations. What were the competitors you looked at?

The sprawl of categories is the biggest disadvantage. If there are 200 categories, it will fail the purpose of having categories for organizing and filtering integrations in the first place. We can probably look into hierarchical categories.

As much as I like the idea of solving the categorization problem in a scalable and long-term way it would be great not to be blocked by this in making Kibana more user-friendly now. I don't think adding one category will make the situation with categories any worse from the useability perspective. But I do understand the concern that doing that might give the way for more similar requests. I'm very new to the company and for sure missing a lot of the context here, but are there that many requests to add new categories in general? I see that this issue was created almost a year ago and I don't see similar open issues asking for new categories. My concern is that if we are currently the only ones looking for a new category being created there won't be enough incentive to make a bigger change.

[akshay-saraswat]

TL;DR:. I am not arguing against the proposal for "Threat Intelligence" Category. I do see the benefits in it and I am sure I can easily justify the need. Just want to make sure that we capture the data here for historical context and provide a good example to follow for any future requests.

It is a good question. If the decisions on adding existing categories were based on this data, do you know where one can find it or who can help with getting it?

Not sure but Fullstory or Kibana Telemetry should have that data. I will look for it too as soon as I get a chance.

Is it the only purpose categories serve? For me it looks like there is also a need in categories for better linking between features and integrations enabling these features. At least that's the main reason for our request of having this new category created. Is there a better way of solving this use case without categories?

You are right, Categories can serve multiple purposes and we can certainly use them to link integrations and functionalities but as far as I know, this linking of integrations and solutions was not the primary use case for it. Please feel free to correct me if you know something about that decision that I don't. I would be reluctant to use the feature-integration link as the basis for introducing new categories because we have a lot of features in our stack.

Great point, it definitely makes sense to look at what our competitors do. I checked Datadog and Logz.io, it seems that they simply don't provide the capability of enabling 3rd party Threat Intelligence feeds, that's why they don't have this category in their integrations. What were the competitors you looked at?

I looked at Datadog, Splunk, Appdynamics, Grafana, logz.io, New Relic, Wavefront, and Dynatrace. I am not discounting the fact that at least one of our competitors has this category but I can't use that as the sole data point. It would have been a different story altogether if a majority had this category.

As much as I like the idea of solving the categorization problem in a scalable and long-term way it would be great not to be blocked by this in making Kibana more user-friendly now.

I completely agree with you. That's why I am clarifying my stand. I am not against introducing any new category but want to do it consciously with a clear justification. Happy to help in collecting the data to prove our hypothesis.

[maxcold]

thanks a lot for the answers and for clarifying your stand on this request! It totally makes sense to look at the data, will be waiting for more information from you on that topic then. In regards to competitors - it seems like Splunk is the only platform in the list which provides the capabilities of enabling 3rd party Threat Intel Feeds as we do. Some platforms don't have SEIM/XDR capabilities at all and some like Datadog rely on the TI feeds they manage on their own with the help of their partners. I will try to find out if there are more platforms like Elastic and Splunk, but with our Threat Intelligence capabilities, we plan to compete not only with the platforms but also with specialized SEIM/XDR and TI solutions. In the meantime, let me know if our team can help somehow to move this forward.

[SourinPaul]

@akshay-saraswat can you please confirm we are planning to add the "Threat Intel' category in 8.5? I noticed the related ticket.

We are investing to expand our Threat Intel capabilities within the security solution over coming releases. So want to ensure the underlying enablers are in place. Thanks in advance @akshay-saraswat

[akshay-saraswat]

@SourinPaul According to fullstory, 67 unique users searched for the "Threat Intelligence" keyword on the integrations page 5,740 times in the last 30 days. If I look at the data for the past 90 days, I see the same pattern. Considering what I wrote in my previous comment and the data I found in fullstory, I have prioritized it for 8.5 for the ecosystem team today. But I cannot promise that it will be delivered in 8.5 until we have gone through the capacity planning. I don't think this would block any of the capabilities you are working on. Please let me know if it does and is urgent.

[mtojek]

Keep in mind that it isn't just Ecosystem and package spec. AFAIK the Fleet team should also update the Kibana code (labels).

[jsoriano]

Open https://github.com/elastic/package-spec/pull/366 to add the category to the spec, and https://github.com/elastic/kibana/issues/135525 for the change in Fleet

[SourinPaul]

Thanks, much @akshay-saraswat @mtojek, and team. Having it delivered in the 8.5/8.6 timeframe will be perfect. 🚀

[jen-huang]

There are no changes needed on Fleet's side to support new categories. Fleet simply forwards the list of categories from the registry (https://github.com/elastic/package-registry/blob/main/categories.go#L25). Closing the Fleet issue.

[mtojek]

@jen-huang What about those files? They seem to be related to categories:

https://github.com/elastic/kibana/blob/f6eafdd74620ab57b63d8b01bb02960ab53d8d07/x-pack/plugins/fleet/common/types/models/package_spec.ts https://github.com/elastic/kibana/blob/ba96eef4330c96c19fd2bc8247f59220c3c74121/src/plugins/custom_integrations/common/index.ts

[jen-huang]

@mtojek replied in https://github.com/elastic/kibana/issues/135525#issuecomment-1175546327

Shall we call this category "Threat Intel" or "Threat Intelligence"? (I will assume the latter) Registry code needs updating too: https://github.com/elastic/package-registry/blob/dbffb08d5571de814d6941f8a747bae87ab6d375/packages/package.go#L29

[mtojek]

Thank you for spotting this, @jen-huang!

[jsoriano]

Shall we call this category "Threat Intel" or "Threat Intelligence"?

Good point, +1 to use Thread Intelligence, as seems to be the actual term (https://en.wikipedia.org/wiki/Threat_intelligence).

Should we also use threat_intelligence instead of threat_intel as the category? I guess we are now in time to change this in the spec before it starts to be used.

Registry code needs updating too: https://github.com/elastic/package-registry/blob/dbffb08d5571de814d6941f8a747bae87ab6d375/packages/package.go#L29

Oh, I wouldn't have expected that. Are these titles used directly in Kibana? I think the registry shouldn't provide texts.

[mtojek]

I'm afraid that you can't change EPR API now: link

[jen-huang]

Thanks all - closing this again as the registry and Kibana changes are now merged.