search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.62k stars 8.22k forks source link

Import / export of rules and connectors as Kibana saved object Export/Import #50266

Open mikecote opened 5 years ago

mikecote commented 5 years ago

This issue summarizes the steps required to implement the import and export of rules and connectors in order to support the impending removal of legacy multi-tenancy. Import and export will occur from the saved object management page, so for this issue, we will not be enforcing RBAC restrictions on the rules and connectors. This will not preclude us from moving this functionality to the rule management page in the future and adding RBAC restrictions.

The proposed workflow will be to export and import rules and connectors in a disabled state, with any api key or secrets information nullified. After import, it will be up to the user to re-enable rules and connectors, with UX to prompt them to enter any required secrets for connectors.

Implementation steps

1. Add concept of disabled connectors

2021/05/13 Update

There is currently a bug with the SO export process that prevents connectors from being exported correctly when they are exported as part of a rule reference:

2021/07/01 Update

Import/export of rules and connectors has been merged into 7.14, with the caveat that security and stack monitoring rules are excluded from the export. We have the following issues to track the progress of aligning security and stack monitoring rule export with the framework export:

elasticmachine commented 5 years ago

Pinging @elastic/kibana-stack-services (Team:Stack Services)

mikecote commented 4 years ago

What happens when secrets are missing?

There is an issue open to allow end-users to access encrypted properties #56013. This would allow a few options for actions:

  1. Export encrypted attributes and actions will work right away after import (auto generate new API key)
  2. Export without encrypted attributes and have some interactive import where users have to input credentials for each action
  3. Export without encrypted attributes and import actions as disabled until they are enabled with credentials
mikecote commented 4 years ago

Some additional challenges mentioned here: https://github.com/elastic/kibana/issues/60053#issuecomment-697344142.

  • When moving alert SOs to another cluster, the API keys will not exist within Elasticsearch
  • When importing alerts, task manager tasks will have to be created so these alerts run after importing
  • Is is possible that the user wants to provide different credentials in their connectors when moving to another cluster
  • The RBAC work we've done doesn't give the user direct access to the SOs
  • As mentioned, usage of encrypted attributes (we may revisit this down the line for allowing those to be exposed now that we have RBAC)
mikecote commented 4 years ago

I have some ideas how some of the concerns above could be handled.

Connectors

Here we could rely on something like exposing connector secrets which could be included in the export. This will be a breaking change as users wouldn't expect such a change in a minor.

An alternative option could be to add a feature to connectors that allows users to disable them. This could behave the same as when connectors are disabled by the license where they don't run. We could then omit secrets and disable the connector in the export data. We could then prompt the user after import that further steps are necessary to enable the connector.

Alerts

I think a lot of the workflows could be solved by exporting / importing the alerts as disabled. It won't require an API key or a task to be scheduled, it won't rely on the importing user's privileges to make the alert work. Here as well we could prompt the user after import that further steps are necessary to enable the alerts and their related connectors.

pmuellr commented 4 years ago

I'm not a fan of exporting the secrets. It's basically a back-door to exfiltrate them. Convince an admin to export the actions, the admin not realizing they contain secrets. Kinda thing.

I wonder how hard it would be to allow connectors with required secrets to be imported WITHOUT them. I think we'd need to change all the secret fields to schema.nullable() in order for that to work, but then we lose our schema checks for those being required when being used as expected. We're already partially there with the hasAuth stuff that got added to email and webhooks, you could imagine a similar field justImported (obviously a better name), where the validation code would then allow the values to be null when that was true, and not allow them to be null when it's false, and we'd reset justImported to false on every edit. We could then also have more precise error messages if you used an imported connector that didn't have the secrets added: "you forgot to add the secrets back". And also highlight these in the connectors list.

mikecote commented 4 years ago

The justImported method is not a bad idea.

How do you envision showing users that they need to fix a connector before it can be used? I was thinking the enable / disable connector feature could work for this where the fields would have to be entered before enabling the connector somehow and the connectors list could filter on these. Or maybe that logic is handled when enabling the alert?

pmuellr commented 4 years ago

One option would to use enabled on a connector in lieu of justImported. enabled: false could be used in the test for a connector's validation that the secrets which would normally be required, can be null. We'd want to have the enabled checkbox in the connector edit form, so that when they click enabled on, we won't let them save till it's validated, which at that point would mean the normally required secrets must not be null.

Using enabled may have an additional benefit of allowing a customer to temporarily "disable" an action from running, without having to actually delete it from the alert.

Also remember, we can disable action types via config (and I think some other ways?), so that "is an action enabled" gets a little more complicated / confusing to the customer. But at least it's still a "single concept", vs having a separate, kinda weird thing about "this action was imported so not usable yet".

Next question, can we hook the export to set this field to false on export? Or hook the import? Or probably both, for safety?

I also wonder how discoverable this will be. It's really the connector, so I think you wouldn't want to be able to add a new action to an alert with a disabled connector - we can add some badge to the connector indicating it's disabled. But what about editing alerts with existing actions whose connectors are disabled? Is the action editable? Maybe just a banner in the action edit form indicating the connector is disabled, with a link to go right to it's editor?

mikecote commented 4 years ago

One option would to use enabled on a connector in lieu of justImported. enabled: false could be used in the test for a connector's validation that the secrets which would normally be required, can be null. We'd want to have the enabled checkbox in the connector edit form, so that when they click enabled on, we won't let them save till it's validated, which at that point would mean the normally required secrets must not be null.

Using enabled may have an additional benefit of allowing a customer to temporarily "disable" an action from running, without having to actually delete it from the alert.

Also remember, we can disable action types via config (and I think some other ways?), so that "is an action enabled" gets a little more complicated / confusing to the customer. But at least it's still a "single concept", vs having a separate, kinda weird thing about "this action was imported so not usable yet".

100% in agreement regarding this. We have disabled via config and disabled via license, there's some UX in the alerts view for this already and it could be "free" for the new disabled feature (I don't recall exactly how the UX goes, some may prevent saving the alert with disabled by license connectors).

Next question, can we hook the export to set this field to false on export? Or hook the import? Or probably both, for safety?

If we are looking at building our own import / export API due to RBAC limitations, we'll have full control over this flow :)

I also wonder how discoverable this will be. It's really the connector, so I think you wouldn't want to be able to add a new action to an alert with a disabled connector - we can add some badge to the connector indicating it's disabled. But what about editing alerts with existing actions whose connectors are disabled? Is the action editable? Maybe just a banner in the action edit form indicating the connector is disabled, with a link to go right to it's editor?

Good questions, I think these will be good discussions. These seem answerable if this becomes the overall import / export approach. And I think it would work for a minor release (pre 8.0).

mikecote commented 4 years ago

Below is a summary of what I believe is needed to achieve import / export in a 7.x release based on the conversations above. The following is based on integrating and leveraging saved objects management UIs and APIs for import / export (https://github.com/elastic/kibana/issues/82027).

NOTE: Saved objects management displays all the saved objects in the system regardless of user feature privileges.

1. Ability to specify custom id for alerts and actions #50210

Since the encrypted saved objects (ESO) plugin doesn't allow specifying custom IDs (see here) and the saved objects management allows overwriting on import, we'll need to support such scenario.

We should verify that this step is needed and coordinate with the Kibana Security team to revive https://github.com/elastic/kibana/pull/42762. The work specific to the alerting APIs to support this could be split into a follow up issue or done at the same time.

2. Ability to enable / disable connectors

A new feature that could be useful to users who want to stop all executions of a connector. This is similar to using xpack.actions.enabledActionTypes in kibana.yml but dynamic and at the connector level.

The usage of this for import / export would be to export all the connectors as disabled and without secrets and have some UX when users try to enable connectors with missing secrets. Alerts can still reference disabled connectors but they will not execute. Exact UX to be determined.

3. Integrate alerts and connectors to saved objects management #82027

This would be an effort shared with the platform team to have alerts and connectors available in the saved objects management UIs. In the future, we can leverage copy to space and share across spaces feature (and/or integrate within our UIs).

For import / export to work, we will need custom logic to export alerts and connectors. Both would "disable" the alert / connector on export. The removal of secrets should already be done by the find API ESO wrapper.

mikecote commented 3 years ago

Moving from 7.x - Candidates to 7.13 after the latest 7.x planning session.

LeeDr commented 3 years ago

I didn't notice, but could have been in a linked issue, if there's been discussion about what the use cases for import/export of alerts and actions are? The first use cases I think of are;

  1. a backup (export) that could be restored (imported) on the same Kibana instance with the same cluster. In this case, it might be nice if the secrets could be encrypted in such a way that they can only be decrypted on the same instance without having them disabled or losing all the secrets. They're already stored encrypted, right?
  2. to transfer actions to another cluster. Maybe if I could provide the savedObjectEncryption key when I exported them (they'd be in plain text at this point and my responsibility for safe keeping) I could import them into another cluster where they would be encrypted with whatever key is on that instance.

I hate to think we can't come up with a safe way for a superuser of multiple clusters to backup/restore or replicate actions across clusters.

mikecote commented 3 years ago

@LeeDr The current use case driving this issue is regarding # 2 to move alerts/connectors to another cluster as Kibana drops multi-tenancy support. The approach we're going will not include sensitive (encrypted) data but have workflows to re-enter sensitive information before the connectors can be used. This decision was made because we have to deliver something in a minor release while ensuring we prevent exposing sensitive data.

We've discussed the option of exposing sensitive information in https://github.com/elastic/kibana/issues/78704 but haven't found it to be a priority at this time. If we do something in 8.x, it could be behind a feature flag.

kobelb commented 3 years ago

The ability to export hidden saved-objects and provide warnings on import have been addressed, so this should no longer be blocked. If you do happen to find some other infrastructure that is required to make alerts and actions importable/exportable, please let us know in https://github.com/elastic/kibana/issues/82020.

mikecote commented 3 years ago

Thanks @kobelb! We plan to start work on this soon 🙏

gnumoksha commented 3 years ago

tl;dr if this feature is "non-trivial because of the security implications of exporting the captured credentials", then exporting/importing only the alerts will be very helpful. Right now I'm manually migrating dozens of alerts from a buggy elastic cloud cluster.

spong commented 3 years ago

1. Ability to specify custom id for alerts and actions #50210

Since the encrypted saved objects (ESO) plugin doesn't allow specifying custom IDs (see here) and the saved objects management allows overwriting on import, we'll need to support such scenario.

We should verify that this step is needed and coordinate with the Kibana Security team to revive #42762. The work specific to the alerting APIs to support this could be split into a follow up issue or done at the same time.

With regards to importing rules (https://github.com/elastic/kibana/issues/94152), are we still planning for the above to be possible? Asking as this is in reference to exporting/importing security rules, and allowing a static_id such that there can be unique rules and so can be overridden on import. E.g. the Elastic Detection Pre-packaged rules should be able to be imported and override any existing pre-packaged rules that may be present (if the user decides to check Automatically overwrite conflicts).

ymao1 commented 3 years ago

With regards to importing rules (#94152), are we still planning for the above to be possible? Asking as this is in reference to exporting/importing security rules, and allowing a static_id such that there can be unique rules and so can be overridden on import. E.g. the Elastic Detection Pre-packaged rules should be able to be imported and override any existing pre-packaged rules that may be present (if the user decides to check Automatically overwrite conflicts).

@spong Just to clarify, are you asking if someone has loaded the prepackaged rules into a Kibana, then exports them through the Saved Objects Management UI, and then re-imports that export file, if the existing rules will be overwritten (if that option is checked)? If so, that is correct. This is all handled by the saved objects API.

spong commented 3 years ago

Yup, that's what I was curious about. My concern here is that while we now have support for specifying custom Rule ID's (https://github.com/elastic/kibana/issues/50210 / https://github.com/elastic/kibana/pull/89814), this hasn't been plumbed through to the the creation of the pre-packaged rules, and so those Rules will only have a unique _id to the cluster (resulting in duplicate rules if importing rules from another cluster via this UI). Even so, there's still the case where pre-packaged rules would have been created before this change, and so would need to be updated to use their static_id for _id, which I don't think is possible without a performing a delete/add (which has its own set of caveats).

Do you know the specifics around how the overwrite logic determines uniqueness? Is it just comparing _id on the SO itself? The pre-packaged rules store their static_id in the alert tags as __internal_rule_id:[ID_HERE], so I guess we could potentially introspect that field during import if it's possible to provide our own compare function. Another potential option would be taking care of this during the SO migration as we move from the single security rules to dedicated types. If this can't be addressed we may just have to disable the import/export of security rules within the SO Management UI and fallback to the solution specific import/export through 7.last.

ymao1 commented 3 years ago

@spong I see what you're saying. I believe the saved objects importer determines uniqueness per space per _id currently (@elastic/kibana-core can you confirm that this is correct).

I think it would make the most sense to filter out these internal rules on import as we have a little more control over that process. We can't currently optionally show/hide the prebuilt detection rules from the Saved Objects Management UI so it would seem to me a strange experience to have them listed in the UI and just skip them onExport. If we skipped them onImport, we can include that information in a warning message.

Can we skip importing all prebuilt detection rules or are you asking that we only avoid importing them if they are already loaded? The former would be the most straightforward so I would advocate for that option if it seems sufficient.

spong commented 3 years ago

Can we skip importing all prebuilt detection rules or are you asking that we only avoid importing them if they are already loaded?

If it's not feasible for us to update the _id to be the static rule_id during our SO migrations I'm actually proposing we disable the import/export of all Security Detection Rules from the SO Mgmt UI in favor of the the dedicated UI within Security Solution. Discussed this with @kobelb with regards to the 8.x migration here: https://github.com/elastic/kibana/issues/82537#issuecomment-721779265.

ymao1 commented 3 years ago

If it's not feasible for us to update the _id to be the static rule_id during our SO migrations I'm actually proposing we disable the import/export of all Security Detection Rules from the SO Mgmt UI in favor of the the dedicated UI within Security Solution. Discussed this with @kobelb with regards to the 8.x migration here: #82537 (comment).

Ok! Is there an issue for updating the _id to be the rule_id during migration that I can track? For now, I will update the import issue to check for and skip security rules on import and show a warning message in this case.

ymao1 commented 3 years ago

@kobelb It seems from the discussion on this issue that you don't see an issue with skipping the import/export of detection rules from the Saved Objects Management UI because security offers their own import/export mechanism. Am I interpreting that correctly? Just want to make sure our bases are covered from a legacy multi-tenancy perspective

ymao1 commented 3 years ago

@jasonrhodes @kobelb

There is a related issue with exporting/importing stack monitoring rules (which are auto-created when a user visits the stack monitoring page in Kibana). Because the stack monitoring rules don't have a fixed _id, there is the potential to have duplicate stack monitoring rules.

Here are the possible scenarios:

If we allow import/export of stack monitoring rules

If we don't allow import/export of stack monitoring rules

Is there one scenario that is preferable? Am I missing any scenarios?

kobelb commented 3 years ago

@kobelb It seems from the discussion on this issue that you don't see an issue with skipping the import/export of detection rules from the Saved Objects Management UI because security offers their own import/export mechanism. Am I interpreting that correctly?

That is correct. As long as the user has a way to import/export all of their data from one cluster to another, we're all good. Ideally, this would be as simple as using saved-object management's import/export; however, if we have to instruct the user to use a dedicated UI/API for this, that's tolerable.

There is a related issue with exporting/importing stack monitoring rules (which are auto-created when a user visits the stack monitoring page in Kibana). Because the stack monitoring rules don't have a fixed _id, there is the potential to have duplicate stack monitoring rules.

@jasonrhodes do you know how "heavy weight" the stack-monitoring rules are? What are the repercussions of us duplicating these?

spong commented 3 years ago

Ok! Is there an issue for updating the _id to be the rule_id during migration that I can track?

@ymao1 I've created https://github.com/elastic/kibana/issues/99741 for tracking this effort. Thank you for investigating further and opening the core dependency here. 🙂

jasonrhodes commented 3 years ago

Hey all, I'm getting up to speed on this. There are a lot of "id" values related to RAC, Saved Objects, etc., so forgive me for asking for some clarification here.

Because the stack monitoring rules don't have a fixed _id, there is the potential to have duplicate stack monitoring rules.

Which _id is this? How do other rules avoid this problem? Is this more related to the fact that we auto-create these rules when you visit the Stack Monitoring page, rather than the fixed _id? i.e., if we weren't auto-creating them, there wouldn't be a risk of duplicating them unless a user specifically opted into creating the SM rules on the original cluster, and then also re-creating the same rules on the new cluster where they are importing their old rules?

Second question: is there a way to specify a fixed, deterministic _id value? Or is that out of the question?

@jasonrhodes do you know how "heavy weight" the stack-monitoring rules are? What are the repercussions of us duplicating these?

@kobelb we recently landed a change so that these alerts are only auto-created for the default space, but my concern is that because these are auto-created, duplicating them is going to be annoying for users. Not only that, but the UI is currently not able to even handle multiples of the same alert without some issues (which we are working through).

Is this scheduled to be available in 7.14? Stack Monitoring rules and alerts have been giving us some issues so we may need to prioritize changing how they work in parallel with this effort.

ymao1 commented 3 years ago

Which _id is this? How do other rules avoid this problem? Is this more related to the fact that we auto-create these rules when you visit the Stack Monitoring page, rather than the fixed _id?

Second question: is there a way to specify a fixed, deterministic _id value? Or is that out of the question?

@jasonrhodes The _id field referenced here is the ID of the rule saved object. This is typically auto-generated when a rule is created, but with this recent PR, we now allow specifying the _id on rule creation, as long as it is a uuid v1 or v4.

i.e., if we weren't auto-creating them, there wouldn't be a risk of duplicating them unless a user specifically opted into creating the SM rules on the original cluster, and then also re-creating the same rules on the new cluster where they are importing their old rules?

Yes, that's correct. Because stack monitoring auto-creates the rules (currently without specifying a fixed _id), if auto-created stack monitoring rules are exported from Kibana A and then imported into Kibana B, they would have the _id from Kibana A. If Kibana B already had auto-created stack monitoring rules with the _id from Kibana B, there would be duplicates.

Is this scheduled to be available in 7.14? Stack Monitoring rules and alerts have been giving us some issues so we may need to prioritize changing how they work in parallel with this effort.

We were planning this for 7.14 but it is only a hard requirement that it is available in 7.last to support removal of legacy multi-tenancy in 8.0, so we can feature flag it until then if necessary.

smith commented 3 years ago

The usage of this for import / export would be to export all the connectors as disabled and without secrets and have some UX when users try to enable connectors with missing secrets. Alerts can still reference disabled connectors but they will not execute. Exact UX to be determined.

Is it still possible to import with secrets via the API? I'd expect to be able to store the secrets for my connectors in something like Vault and pull them out for use in importing with some API calls.

jasonrhodes commented 3 years ago

Thanks for the clarification, @ymao1 — when the rules are being exported or imported, is it an all or nothing UX or does the user get presented with a list of rules that they can choose to export/import?

ymao1 commented 3 years ago

@jasonrhodes The import/export is done through the Saved Object Management UI, which shows a list of saved objects and does allow a user to select one/multiple specific SOs to export. Screenshot here: https://www.elastic.co/guide/en/kibana/master/managing-saved-objects.html

jasonrhodes commented 3 years ago

OK thanks, although I guess it doesn't matter at the moment. I think if we allow users to import those rules and they end up with duplicates, it's going to cause issues. For now, we'll need to restrict Stack Monitoring rules from import/export until we solve https://github.com/elastic/kibana/issues/91145 at least, and possibly https://github.com/elastic/kibana/issues/100133 as well. Then we'd be fine to just let them be imported/exported as all other rules.

ymao1 commented 3 years ago

Added issue for feature flagging rule import/export: https://github.com/elastic/kibana/issues/100145

mikecote commented 3 years ago

If Kibana B already had auto-created stack monitoring rules with the _id from Kibana B, there would be duplicates.

@ymao1 keep me honest 🙂 @jasonrhodes in this scenario, I think the user will be made aware the rules already exist and ask if they wish to overwrite them or create duplicates. There may be the option for the user to ask for new IDs on import, at that point we wouldn't be able to detect if there are duplicates afterwards.

Is it still possible to import with secrets via the API? I'd expect to be able to store the secrets for my connectors in something like Vault and pull them out for use in importing with some API calls.

@smith at this time no, we've been cautious about breaking the saying that "once secrets are stored in Kibana, we don't expose them to users afterwards". We've been thinking about breaking that saying here but have not found a valid reason to break the behaviour, especially not in a minor release.

For now, we'll need to restrict Stack Monitoring rules from import/export until we solve https://github.com/elastic/kibana/issues/91145 at least, and possibly https://github.com/elastic/kibana/issues/100133 as well. Then we'd be fine to just let them be imported/exported as all other rules.

@jasonrhodes is it right to assume the work in those issues will be completed in 7.14 / 7.15? I'm asking to ensure we have a way for users to move their customized stack monitoring rules from one environment to another by 7.16.

ymao1 commented 3 years ago

If Kibana B already had auto-created stack monitoring rules with the _id from Kibana B, there would be duplicates.

@ymao1 keep me honest 🙂 @jasonrhodes in this scenario, I think the user will be made aware the rules already exist and ask if they wish to overwrite them or create duplicates. There may be the option for the user to ask for new IDs on import, at that point we wouldn't be able to detect if there are duplicates afterwards.

@mikecote Unfortunately, this is not the case. Since stack monitoring rules are autocreated without specifying a static _id, it is very likely (pretty much guaranteed) that the auto-created stack monitoring rules in Kibana A have different _ids than the auto-created stack monitoring rules in Kibana B, so importing from A -> B will not provide that "overwrite or create duplicates" popup.

What could happen is if stack monitoring rules are imported from A -> B before the stack monitoring rules are autocreated in Kibana B. Then the first time user visits stack monitoring in Kibana B, the auto-creation should be skipped because there are already (imported) rules of the stack monitoring types in B.

mikecote commented 3 years ago

If Kibana B already had auto-created stack monitoring rules with the _id from Kibana B, there would be duplicates.

@ymao1 keep me honest 🙂 @jasonrhodes in this scenario, I think the user will be made aware the rules already exist and ask if they wish to overwrite them or create duplicates. There may be the option for the user to ask for new IDs on import, at that point we wouldn't be able to detect if there are duplicates afterwards.

@mikecote Unfortunately, this is not the case. Since stack monitoring rules are autocreated without specifying a static _id, it is very likely (pretty much guaranteed) that the auto-created stack monitoring rules in Kibana A have different _ids than the auto-created stack monitoring rules in Kibana B, so importing from A -> B will not provide that "overwrite or create duplicates" popup.

What could happen is if stack monitoring rules are imported from A -> B before the stack monitoring rules are autocreated in Kibana B. Then the first time user visits stack monitoring in Kibana B, the auto-creation should be skipped because there are already (imported) rules of the stack monitoring types in B.

Ohh that makes total sense, I didn't think about that. I withdraw that comment 🙈

jasonrhodes commented 3 years ago

@jasonrhodes is it right to assume the work in those issues will be completed in 7.14 / 7.15? I'm asking to ensure we have a way for users to move their customized stack monitoring rules from one environment to another by 7.16.

@mikecote 7.14, no. 7.15, very hard to say. @ravikesarwani this is another thing driving the need to align SM alerts. This one is tricky because we probably have to solve most of the tickets in this meta ticket/epic before this will work without causing problems.

cc: @sgrodzicki / @paulb-elastic

ravikesarwani commented 3 years ago

FYI @mukeshelastic for awareness. I agree @jasonrhodes. It's very important to align the SM alerts to work like other alerts so that its a common experience for solution users, doesn't break as framework evolves and able to utilize new things that gets added to the framework. As we have discussed I am on board with the changes that you have proposed in the Meta issue.

mikecote commented 3 years ago

@jasonrhodes if it's not 7.15, is it 7.16 or 8.x? We are trying to decide what to do with the import/export for edge cases and have two options that require time:

  1. We let solutions that have edge cases build their own import/export experience to ensure user's customizations like rule actions can move to another Kibana. We need a heads up so we can work with @elastic/kibana-core team to exclude your rule saved objects (if possible, that's also TBD).
  2. We have solutions fix their issues before 7.last (7.15 hopefully to give time for remaining steps) and have a unified import/export place for our users to go to when moving away from multi-tenancy.

@spong we have planned to work on supporting customization of references. Would that be the only change requested on your end to have an initial unified import/export experience in 7.last? or still TBD?

spong commented 3 years ago

@spong we have planned to work on supporting customization of references. Would that be the only change requested on your end to have an initial unified import/export experience in 7.last? or still TBD?

Yup! Supporting references (https://github.com/elastic/kibana/issues/87992) is the only dependency we're tracking at the moment. With that change we'll be able to update rules to reference any related exception lists and better manage the import/export flow (along with perf enhancements around the Exceptions Table and bulk export (https://github.com/elastic/kibana/issues/85173)).

That said, and as I see was discussed a bit above, we still need to determine how we're going to support the export/import of actions. If a user has 500 rules in their cluster, each with their own exception list/value list, and custom action, we have the means of restoring the rules, will work a solution for bulk import of exceptions, but don't currently have a method for backing up/restoring actions (or the ability to bulk assign actions, which could be a suitable substitution). Are actions still planned as part of this effort, or just focusing on rules/connectors for the time being?

jasonrhodes commented 3 years ago

@jasonrhodes if it's not 7.15, is it 7.16 or 8.x? We are trying to decide what to do with the import/export for edge cases and have two options that require time:

@mikecote totally understand, I'll leave my opinion here and let @mukeshelastic and @ravikesarwani weigh in as well. I think your option (2) is the right option, and that I don't see us having bandwidth to build a custom import/export system for SM rules anyway. That said, I don't think we need to push our timeline problems onto you, either. There may need to be a way to exclude certain rules from the import/export flow temporarily, and the result would be "Stack Monitoring rules are not importable/exportable at this time, please visit the UI in your new cluster to recreate your rules". None of them have that much customizability options, so I don't think this would be a massive problem.

I do think we have a chance of getting our alerts aligned by the time you make this live in 7.x, but I definitely can't guarantee it, and weighed against the other things we are trying to deliver, making sure we avoid the slightly unfortunate problem I describe above sounds like a very low priority, relatively, from the Stack Monitoring perspective.

ymao1 commented 3 years ago

That said, and as I see was discussed a bit above, we still need to determine how we're going to support the export/import of actions. If a user has 500 rules in their cluster, each with their own exception list/value list, and custom action, we have the means of restoring the rules, will work a solution for bulk import of exceptions, but don't currently have a method for backing up/restoring actions (or the ability to bulk assign actions, which could be a suitable substitution). Are actions still planned as part of this effort, or just focusing on rules/connectors for the time being?

@spong rule actions are currently stored as SO references (referencing the connector that is backing the action) within the rule saved object, so when a rule is exported through the SO management UI with export referenced objects checked, both the rule, with its actions & the backing connector are included in the export. Is that what you are asking about, or is there another type of action?

spong commented 3 years ago

@spong rule actions are currently stored as SO references (referencing the connector that is backing the action) within the rule saved object, so when a rule is exported through the SO management UI with export referenced objects checked, both the rule, with its actions & the backing connector are included in the export. Is that what you are asking about, or is there another type of action?

Perfect, that's exactly what I was curious about. So users will be able to backup all their actions via connectors, so on the security side we'll just need to ensure the security rules maintain this reference as well so they can be re-associated on import. Thank you @ymao1! 🙂

ymao1 commented 3 years ago

So users will be able to backup all their actions via connectors

@spong Just to make sure we're on the same page: connectors are stored as saved objects and actions are stored within a rule saved object, with a reference to the connector saved object for each action (along with params specific to that action). So when you say backup all their actions via connectors, are you saying users should use the SO management UI to export their connectors and security solution will be updating their rule export to include the configured actions?

Also, please note that connectors will be exported without their secrets fields because secrets are encrypted and we can't guarantee that the encryption key will be the same if export occurs in Kibana A and then imported to Kibana B. So there is some manual intervention required to re-enter connector secrets after import.

spong commented 3 years ago

So when you say backup all their actions via connectors, are you saying users should use the SO management UI to export their connectors and security solution will be updating their rule export to include the configured actions?

My apologies, meant rules not connectors. So if we end up disabling the export of Security Rules within SO Management we'll need explicit support for including actions as part of the security Solution Rule import/export. I'll capture this in the issue I'm writing (https://github.com/elastic/kibana/issues/100956) to ensure we support actions importing/exporting on the security side. If we have success with #99741 / https://github.com/elastic/kibana/issues/87992, we might just be able to leverage the SO Management UI for Security Rules/Exceptions/Actions (value lists will still need to be managed via the dedicated Security Solution UI though).

ymao1 commented 3 years ago

@spong Gotcha. Thanks for the clarification!

mikecote commented 3 years ago

@spong

Yup! Supporting references (https://github.com/elastic/kibana/issues/87992) is the only dependency we're tracking at the moment. With that change we'll be able to update rules to reference any related exception lists and better manage the import/export flow (along with perf enhancements around the Exceptions Table and bulk export (https://github.com/elastic/kibana/issues/85173)).

Awesome, this is good to know!

That said, and as I see was discussed a bit above, we still need to determine how we're going to support the export/import of actions. If a user has 500 rules in their cluster, each with their own exception list/value list, and custom action, we have the means of restoring the rules, will work a solution for bulk import of exceptions, but don't currently have a method for backing up/restoring actions (or the ability to bulk assign actions, which could be a suitable substitution). Are actions still planned as part of this effort, or just focusing on rules/connectors for the time being?

With this part, I believe @ymao1 covered this but wanted to make sure. I believe you mentioned that your actions are stored as separate saved-objects? If so, you'd just need to add the reference to them in the rule (just like exceptions and value list).

@ymao1 is there a change needed on downstream/referenced saved-object types to include themselves in the export? or are they included automatically without that saved-object type marking itself exportable?

@jasonrhodes

None of them have that much customizability options, so I don't think this would be a massive problem.

I've seen a lot of user requests to be able to modify these SM rules and we've shown where to go for this. Users can modify things like rule actions? If so, they can customize anything they want there and wouldn't have an easy way to transfer that (they would have to go through each rule and re-write their customizations and templates for each rule action).

I do think we have a chance of getting our alerts aligned by the time you make this live in 7.x, but I definitely can't guarantee it, and weighed against the other things we are trying to deliver, making sure we avoid the slightly unfortunate problem I describe above sounds like a very low priority, relatively, from the Stack Monitoring perspective.

Based on this, I'll assume you are ok with the user experience risk that I stated above and we'll go ahead and talk to @elastic/kibana-core team to explore options of excluding SM rules from export (cc @ymao1).

pgayvallet commented 3 years ago

Based on this, I'll assume you are ok with the user experience risk that I stated above and we'll go ahead and talk to @elastic/kibana-core team to explore options of excluding SM rules from export (cc @ymao1).

Let's continue the discussion about this in https://github.com/elastic/kibana/issues/99680 then. There are a lot of implications that need to be discussed.

ymao1 commented 3 years ago

@ymao1 is there a change needed on downstream/referenced saved-object types to include themselves in the export? or are they included automatically without that saved-object type marking itself exportable?

@mikecote @spong Any downstream saved object types will need to be marked as importable and exportable to be exported through the SO management UI. This has been done for connectors, so rules that contain a connector (action SO type) in its references array will automatically have the connector exported as well. When https://github.com/elastic/kibana/issues/87992 is resolved and rules have exceptions and value lists in their references array, those SO types will need to be marked as importableAndExportable in order to be exported with its referencing rule.

jasonrhodes commented 3 years ago

Based on this, I'll assume you are ok with the user experience risk that I stated above and we'll go ahead and talk to @elastic/kibana-core team to explore options of excluding SM rules from export (cc @ymao1).

Yeah, I don't think we have any other option, unfortunately. As I said, we hope we get this sorted before then, but if not, this will have to be the fallback. cc: @ravikesarwani / @mukeshelastic / @sgrodzicki / @paulb-elastic for visibility.

gmmorris commented 3 years ago

As we are carrying this Epic over from 7.14 to 7.15 I have updated the candidate labels.

This is in order to enable the work needed for the two remaining issues that enable Import/Export in Security Solution and Stack Management.