[Security Solution] Some fields are not persisted when creating Endpoint integration

[juliaElastic]

Describe the bug: User reported here https://discuss.elastic.co/t/fleet-api-vs-fleet-ui/283918 When creating a package_policy Endpoint using API, some fields are not persisted. This is not happening when the package_policy is updated.

Kibana/Elasticsearch Stack version: 8.0.0, probably 7.x too

Server OS version: macOS Big Sur 11.5.2

Browser and Browser OS versions: N/A

Elastic Endpoint version:

Original install method (e.g. download page, yum, from source, etc.): download page

Functional Area (e.g. Endpoint management, timelines, resolver, etc.): Endpoint management

Steps to reproduce:

1. POST to /api/fleet/package_policies referring to a valid policy having antivirus_registration and popup.message fields in the body
2. query created policy, observe that above fields are not set
3. PUT to same policy with same fields, observe that the fields are persisted now

See example body here:

{
        "name": "Endpoint",
        "description": "",
        "namespace": "XXXX",
        "policy_id": "XXXX",
        "enabled": true,
        "output_id": "",
        "inputs": [
            {
            "streams": [],
            "type": "endpoint",
            "config": {
                "artifact_manifest": {
                "value": {
                    "schema_version": "v1",
                    "manifest_version": "1.0.43",
                    "artifacts": {
                    "endpoint-trustlist-windows-v1": {
                        "relative_url": "/api/fleet/artifacts/endpoint-trustlist-windows-v1/XXX",
                        "compression_algorithm": "zlib",
                        "decoded_size": 311,
                        "decoded_sha256": "XXX",
                        "encryption_algorithm": "none",
                        "encoded_sha256": "XXX",
                        "encoded_size": 153
                    },
                    "endpoint-eventfilterlist-windows-v1": {
                        "relative_url": "/api/fleet/artifacts/endpoint-eventfilterlist-windows-v1/XXX",
                        "compression_algorithm": "zlib",
                        "decoded_size": 4205,
                        "decoded_sha256": "XXX",
                        "encryption_algorithm": "none",
                        "encoded_sha256": "XXX",
                        "encoded_size": 495
                    },
                    "endpoint-exceptionlist-linux-v1": {
                        "relative_url": "/api/fleet/artifacts/endpoint-exceptionlist-linux-v1/XXX",
                        "compression_algorithm": "zlib",
                        "decoded_size": 14,
                        "decoded_sha256": "XXX",
                        "encryption_algorithm": "none",
                        "encoded_sha256": "XXX",
                        "encoded_size": 22
                    },
                    "endpoint-trustlist-macos-v1": {
                        "relative_url": "/api/fleet/artifacts/endpoint-trustlist-macos-v1/XXX",
                        "compression_algorithm": "zlib",
                        "decoded_size": 14,
                        "decoded_sha256": "XXX",
                        "encryption_algorithm": "none",
                        "encoded_sha256": "XXX",
                        "encoded_size": 22
                    },
                    "endpoint-exceptionlist-macos-v1": {
                        "relative_url": "/api/fleet/artifacts/endpoint-exceptionlist-macos-v1/XXX",
                        "compression_algorithm": "zlib",
                        "decoded_size": 14,
                        "decoded_sha256": "XXX",
                        "encryption_algorithm": "none",
                        "encoded_sha256": "XXX",
                        "encoded_size": 22
                    },
                    "endpoint-trustlist-linux-v1": {
                        "relative_url": "/api/fleet/artifacts/endpoint-trustlist-linux-v1/XXX",
                        "compression_algorithm": "zlib",
                        "decoded_size": 14,
                        "decoded_sha256": "XXXX",
                        "encryption_algorithm": "none",
                        "encoded_sha256": "XXX",
                        "encoded_size": 22
                    },
                    "endpoint-eventfilterlist-linux-v1": {
                        "relative_url": "/api/fleet/artifacts/endpoint-eventfilterlist-linux-v1/XXX",
                        "compression_algorithm": "zlib",
                        "decoded_size": 14,
                        "decoded_sha256": "XXX",
                        "encryption_algorithm": "none",
                        "encoded_sha256": "XXX",
                        "encoded_size": 22
                    },
                    "endpoint-exceptionlist-windows-v1": {
                        "relative_url": "/api/fleet/artifacts/endpoint-exceptionlist-windows-v1/XXX",
                        "compression_algorithm": "zlib",
                        "decoded_size": 14,
                        "decoded_sha256": "XXX",
                        "encryption_algorithm": "none",
                        "encoded_sha256": "XXX",
                        "encoded_size": 22
                    },
                    "endpoint-eventfilterlist-macos-v1": {
                        "relative_url": "/api/fleet/artifacts/endpoint-eventfilterlist-macos-v1/XXX",
                        "compression_algorithm": "zlib",
                        "decoded_size": 14,
                        "decoded_sha256": "XXX",
                        "encryption_algorithm": "none",
                        "encoded_sha256": "XXX",
                        "encoded_size": 22
                    }
                    }
                }
                },
                "policy": {
                "value": {
                    "linux": {
                    "popup": {
                        "malware": {
                            "message": "Todyl {action} {filename}",
                            "enabled": true
                        }
                    },
                    "malware": {
                        "mode": "detect"
                    },
                    "logging": {
                        "file": "info"
                    },
                    "events": {
                        "process": true,
                        "file": true,
                        "network": true
                    }
                    },
                    "windows": {
                    "popup": {
                        "malware": {
                            "message": "Todyl {action} {filename}",
                            "enabled": true
                        },
                        "ransomware": {
                            "message": "Todyl {action} {filename}",
                            "enabled": true
                        }
                    },
                    "malware": {
                        "mode": "prevent"
                    },
                    "logging": {
                        "file": "info"
                    },
                    "antivirus_registration": {
                        "enabled": true
                    },
                    "events": {
                        "registry": true,
                        "process": true,
                        "security": true,
                        "file": true,
                        "dns": true,
                        "dll_and_driver_load": true,
                        "network": true
                    },
                    "ransomware": {
                        "mode": "prevent",
                        "supported": true
                    }
                    },
                    "mac": {
                    "popup": {
                        "malware": {
                        "message": "Todyl {action} {filename}",
                        "enabled": true
                        }
                    },
                    "malware": {
                        "mode": "prevent"
                    },
                    "logging": {
                        "file": "info"
                    },
                    "events": {
                        "process": true,
                        "file": true,
                        "network": true
                    }
                    }
                }
                }
            },
            "enabled": true
            }
        ],
        "package": {
            "name": "endpoint",
            "title": "Endpoint Security",
            "version": "1.0.0"
        }
    }

Current behavior: Some fields are not persisted on creation

Expected behavior: All fields should be persisted on creation

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

server    log   [10:26:14.517] [warning][plugins][securitySolution] Possible problem creating detection signals index (undefined): Cannot read property 'getResourceName' of undefined
server    log   [10:26:14.518] [error][plugins][securitySolution] Unable to create detection rules automatically (undefined): Cannot read property 'getExecutionLogClient' of undefined

Any additional context (logs, chat logs, magical formulas, etc.): Errors are coming from this logic: : x-pack/plugins/security_solution/server/fleet_integration/handlers/install_prepackaged_rules.ts:63

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-onboarding-and-lifecycle-mgt (Team:Onboarding and Lifecycle Mgt)

[kevinlog]

@juliaElastic the Endpoint integration is initially enabled with a default policy, so this might be the issue.

For clarity, are you saying that if you add antivirus_registration and popup.message fields in your initial POST call that they are not present at all? Or are the fields present, but they don't retain the fields that you set?

A before and after of each of the Policy bodies will help. Thanks!

fyi @paul-tavares @parkiino let me know if there is anything in here that stands out.