search

OpenCTI-Platform / opencti

Open Cyber Threat Intelligence Platform
https://opencti.io
Other
6.13k stars 907 forks source link

Large volume of internal errors -- Engine unhandled rejection | Bulk Indexing failed | Failed: index is missing #5825

Open MaxwellDPS opened 7 months ago

MaxwellDPS commented 7 months ago

Description

Seeing lots of platform errors that seem fairly fatial. Can open multiple issues if desired, but its a lot.

I have provided logs for most of them, Please LMK what further info y'all want

_time Bulk indexing fail Cannot read properties of null (reading 'filter') Cannot read properties of null (reading 'split') Element(s) not found Engine unhandled rejection Execution timeout, too many concurrent call on the same entities Request timed out Unexpected error value: "[REDIS] Failed to extend resource" We cant update the same attribute multiple times in the same operation action_request_validation_exception Root causes: action_request_validation_exception: Validation Failed: 1: index is missing;2: index is missing; OTHER
2024-01-28 1373 14978 19436 62978 9805 22720 473 2956 76 198 254
2024-01-29 2011 8726 16248 30059 8546 18145 1230 1242 407 236 234
2024-01-30 3871 2710 23724 36652 11960 29708 758 1344 94 484 617
2024-01-31 977 3676 16760 16110 9767 11677 756 680 1378 208 357
2024-02-01 167 1280 8938 19879 5144 5815 340 222 674 48 105
2024-02-02 668 2176 9852 16073 5040 17882 4906 462 116 126 228

Environment

  1. OS (where OpenCTI server runs): CentOS Stream 9
  2. OpenCTI version: 5.12.
  3. OpenCTI client: Frontend
  4. Other environment details: clustered Scaled

Reproducible Steps

Steps to create the smallest reproducible scenario:

  1. Yes 😅 👉

Expected Output

N/A

Errors

Bulk indexing fail

{
  "category": "APP",
  "errors":
    [
      {
        "attributes":
          {
            "errors":
              [
                {
                  "index": "opencti_stix_cyber_observables-000001",
                  "index_uuid": "DX5ulhzKQfiROFqS2mITLA",
                  "reason": "[0aedc817-437e-4be3-aa80-838b3cc796c1]: version conflict, required seqNo [2739252], primary term [4]. current document has seqNo [2739253] and primary term [4]",
                  "shard": "0",
                  "type": "version_conflict_engine_exception",
                },
              ],
            "genre": "TECHNICAL",
            "http_status": 500,
          },
        "message": "Bulk indexing fail",
        "name": "DATABASE_ERROR",
        "stack": "DATABASE_ERROR: Bulk indexing fail\n at error (/opt/opencti/build/src/config/errors.js:8:10)\n at DatabaseError (/opt/opencti/build/src/config/errors.js:58:48)\n at /opt/opencti/build/src/database/engine.js:2520:13\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at elRemoveRelationConnection (/opt/opencti/build/src/database/engine.js:2663:5)\n at elDeleteElements (/opt/opencti/build/src/database/engine.js:2719:3)\n at mergeEntitiesRaw (/opt/opencti/build/src/database/middleware.js:1270:31)\n at mergeEntities (/opt/opencti/build/src/database/middleware.js:1409:26)\n at updateAttributeMetaResolved (/opt/opencti/build/src/database/middleware.js:1935:24)\n at upsertElement (/opt/opencti/build/src/database/middleware.js:2682:20)\n at createEntity (/opt/opencti/build/src/database/middleware.js:3320:16)\n at addStixCyberObservable (/opt/opencti/build/src/domain/stixCyberObservable.js:218:19)",
      },
    ],
  "inner_relation_creation": 0,
  "level": "warn",
  "message": "Bulk indexing fail",
  "operation": "StixCyberObservableAdd",
  "operation_query": "mutation StixCyberObservableAdd($type:String!$stix_id:StixId$x_opencti_score:Int$x_opencti_description:String$createIndicator:Boolean$createdBy:String$objectMarking:[String]$objectLabel:[String]$objectOrganization:[String]$externalReferences:[String]$update:Boolean$AutonomousSystem:AutonomousSystemAddInput$Directory:DirectoryAddInput$DomainName:DomainNameAddInput$EmailAddr:EmailAddrAddInput$EmailMessage:EmailMessageAddInput$EmailMimePartType:EmailMimePartTypeAddInput$Artifact:ArtifactAddInput$StixFile:StixFileAddInput$X509Certificate:X509CertificateAddInput$IPv4Addr:IPv4AddrAddInput$IPv6Addr:IPv6AddrAddInput$MacAddr:MacAddrAddInput$Mutex:MutexAddInput$NetworkTraffic:NetworkTrafficAddInput$Process:ProcessAddInput$Software:SoftwareAddInput$Url:UrlAddInput$UserAccount:UserAccountAddInput$WindowsRegistryKey:WindowsRegistryKeyAddInput$WindowsRegistryValueType:WindowsRegistryValueTypeAddInput$CryptographicKey:CryptographicKeyAddInput$CryptocurrencyWallet:CryptocurrencyWalletAddInput$Hostname:HostnameAddInput$Text:TextAddInput$UserAgent:UserAgentAddInput$BankAccount:BankAccountAddInput$PhoneNumber:PhoneNumberAddInput$PaymentCard:PaymentCardAddInput$MediaContent:MediaContentAddInput){stixCyberObservableAdd(type:$type stix_id:$stix_id x_opencti_score:$x_opencti_score x_opencti_description:$x_opencti_description createIndicator:$createIndicator createdBy:$createdBy objectMarking:$objectMarking objectLabel:$objectLabel update:$update externalReferences:$externalReferences objectOrganization:$objectOrganization AutonomousSystem:$AutonomousSystem Directory:$Directory DomainName:$DomainName EmailAddr:$EmailAddr EmailMessage:$EmailMessage EmailMimePartType:$EmailMimePartType Artifact:$Artifact StixFile:$StixFile X509Certificate:$X509Certificate IPv4Addr:$IPv4Addr IPv6Addr:$IPv6Addr MacAddr:$MacAddr Mutex:$Mutex NetworkTraffic:$NetworkTraffic Process:$Process Software:$Software Url:$Url UserAccount:$UserAccount WindowsRegistryKey:$WindowsRegistryKey WindowsRegistryValueType:$WindowsRegistryValueType CryptographicKey:$CryptographicKey CryptocurrencyWallet:$CryptocurrencyWallet Hostname:$Hostname Text:$Text UserAgent:$UserAgent BankAccount:$BankAccount PhoneNumber:$PhoneNumber PaymentCard:$PaymentCard MediaContent:$MediaContent){id standard_id entity_type parent_types indicators{edges{node{id pattern pattern_type}}}}}",
  "size": 1070,
  "time": 7549,
  "timestamp": "2024-02-03T07:16:08.829Z",
  "type": "WRITE_ERROR",
  "user":
    {
      "applicant_id": "6a4b11e1-90ca-4e42-ba42-db7bc7f7d505",
      "call_retry_number": "1",
      "group_ids": [],
      "ip": "192.168.1.156",
      "organization_ids": [],
      "socket": "query",
      "user_id": "6a4b11e1-90ca-4e42-ba42-db7bc7f7d505",
    },
  "version": "5.12.15"
}

Engine unhandled rejection

{
    "category": "APP",
    "errors": [
        {
            "attributes": {
                "genre": "TECHNICAL",
                "http_status": 500,
                "promise": {},
                "reason": {}
            },
            "message": "Engine unhandled rejection",
            "name": "UNKNOWN_ERROR",
            "stack": "UNKNOWN_ERROR: Engine unhandled rejection\n    at error (/opt/opencti/build/src/config/errors.js:8:10)\n    at UnknownError (/opt/opencti/build/src/config/errors.js:70:47)\n    at process.<anonymous> (/opt/opencti/build/src/boot.js:255:16)\n    at process.emit (node:events:514:28)\n    at process.O5a.process.emit (/opt/opencti/build/node_modules/source-map-support/source-map-support.js:516:21)\n    at emit (node:internal/process/promises:150:20)\n    at processPromiseRejections (node:internal/process/promises:284:27)\n    at processTicksAndRejections (node:internal/process/task_queues:96:32)\n    at runNextTicks (node:internal/process/task_queues:64:3)\n    at processImmediate (node:internal/timers:449:9)"
        }
    ],
    "level": "error",
    "message": "Engine unhandled rejection",
    "timestamp": "2024-02-03T07:08:11.089Z",
    "version": "5.12.15"
}

Validation Failed: 1: index is missing;2: index is missing;

{
  "category": "APP",
  "errors": [
    {
      "attributes": {
        "genre": "TECHNICAL",
        "http_status": 500
      },
      "message": "action_request_validation_exception\n\tRoot causes:\n\t\taction_request_validation_exception: Validation Failed: 1: index is missing;2: index is missing;",
      "name": "UNKNOWN_ERROR",
      "stack": "UNKNOWN_ERROR: action_request_validation_exception\n\tRoot causes:\n\t\taction_request_validation_exception: Validation Failed: 1: index is missing;2: index is missing;\n at error (/opt/opencti/build/src/config/errors.js:8:10)\n at UnknownError (/opt/opencti/build/src/config/errors.js:70:47)\n at Object._logWithError (/opt/opencti/build/src/config/conf.js:311:23)\n at Object.warn (/opt/opencti/build/src/config/conf.js:320:47)\n at Object.willSendResponse (/opt/opencti/build/src/graphql/loggerPlugin.js:100:20)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Promise.all (index 1)\n at b (/opt/opencti/build/node_modules/apollo-server-core/src/requestPipeline.ts:530:5)\n at processHTTPRequest (/opt/opencti/build/node_modules/apollo-server-core/src/runHttpQuery.ts:437:24)"
    },
    {
      "message": "action_request_validation_exception\n\tRoot causes:\n\t\taction_request_validation_exception: Validation Failed: 1: index is missing;2: index is missing;",
      "name": "ResponseError",
      "stack": "ResponseError: action_request_validation_exception\n\tRoot causes:\n\t\taction_request_validation_exception: Validation Failed: 1: index is missing;2: index is missing;\n at DLt.request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:553:17)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at a8e.BulkApi [as bulk] (/opt/opencti/build/node_modules/@elastic/elasticsearch/src/api/api/bulk.ts:75:10)\n at elRemoveRelationConnection (/opt/opencti/build/src/database/engine.js:2663:5)\n at elDeleteElements (/opt/opencti/build/src/database/engine.js:2719:3)\n at updateAttributeMetaResolved (/opt/opencti/build/src/database/middleware.js:2047:7)\n at upsertElement (/opt/opencti/build/src/database/middleware.js:2682:20)\n at createEntity (/opt/opencti/build/src/database/middleware.js:3320:16)\n at addObservedData (/opt/opencti/build/src/domain/observedData.js:111:30)"
    }
  ],
  "inner_relation_creation": 1,
  "level": "warn",
  "message": "Platform unmanaged direct error",
  "operation": "ObservedDataAdd",
  "operation_query": "mutation ObservedDataAdd($input:ObservedDataAddInput!){observedDataAdd(input:$input){id standard_id entity_type parent_types}}",
  "size": 1047,
  "time": 911,
  "timestamp": "2024-02-03T06:08:10.082Z",
  "type": "WRITE_ERROR",
  "user": {
    "applicant_id": "6a4b11e1-90ca-4e42-ba42-db7bc7f7d505",
    "call_retry_number": "1",
    "group_ids": [],
    "ip": "192.168.6.202",
    "organization_ids": [],
    "socket": "query",
    "user_id": "6a4b11e1-90ca-4e42-ba42-db7bc7f7d505"
  },
  "version": "5.12.15"
}

Request Timed out

  {
    "category": "APP",
    "errors": [
      {
        "attributes": {
          "genre": "TECHNICAL",
          "http_status": 500
        },
        "message": "Request timed out",
        "name": "UNKNOWN_ERROR",
        "stack": "UNKNOWN_ERROR: Request timed out\n at error (/opt/opencti/build/src/config/errors.js:8:10)\n at UnknownError (/opt/opencti/build/src/config/errors.js:70:47)\n at Object._logWithError (/opt/opencti/build/src/config/conf.js:311:23)\n at Object.warn (/opt/opencti/build/src/config/conf.js:320:47)\n at Object.willSendResponse (/opt/opencti/build/src/graphql/loggerPlugin.js:100:20)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Promise.all (index 1)\n at b (/opt/opencti/build/node_modules/apollo-server-core/src/requestPipeline.ts:530:5)\n at processHTTPRequest (/opt/opencti/build/node_modules/apollo-server-core/src/runHttpQuery.ts:437:24)"
      },
      {
        "message": "Request timed out",
        "name": "TimeoutError",
        "stack": "TimeoutError: Request timed out\n at DLt.request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:602:17)\n at a8e.BulkApi [as bulk] (/opt/opencti/build/node_modules/@elastic/elasticsearch/src/api/api/bulk.ts:75:10)\n at elUpdateEntityConnections (/opt/opencti/build/src/database/engine.js:2952:5)\n at concurrentEntitiesUpdate (/opt/opencti/build/src/database/middleware.js:1235:5)"
      }
    ],
    "inner_relation_creation": 0,
    "level": "warn",
    "message": "Platform unmanaged direct error",
    "operation": "StixCyberObservableAdd",
    "operation_query": "mutation StixCyberObservableAdd($type:String!$stix_id:StixId$x_opencti_score:Int$x_opencti_description:String$createIndicator:Boolean$createdBy:String$objectMarking:[String]$objectLabel:[String]$objectOrganization:[String]$externalReferences:[String]$update:Boolean$AutonomousSystem:AutonomousSystemAddInput$Directory:DirectoryAddInput$DomainName:DomainNameAddInput$EmailAddr:EmailAddrAddInput$EmailMessage:EmailMessageAddInput$EmailMimePartType:EmailMimePartTypeAddInput$Artifact:ArtifactAddInput$StixFile:StixFileAddInput$X509Certificate:X509CertificateAddInput$IPv4Addr:IPv4AddrAddInput$IPv6Addr:IPv6AddrAddInput$MacAddr:MacAddrAddInput$Mutex:MutexAddInput$NetworkTraffic:NetworkTrafficAddInput$Process:ProcessAddInput$Software:SoftwareAddInput$Url:UrlAddInput$UserAccount:UserAccountAddInput$WindowsRegistryKey:WindowsRegistryKeyAddInput$WindowsRegistryValueType:WindowsRegistryValueTypeAddInput$CryptographicKey:CryptographicKeyAddInput$CryptocurrencyWallet:CryptocurrencyWalletAddInput$Hostname:HostnameAddInput$Text:TextAddInput$UserAgent:UserAgentAddInput$BankAccount:BankAccountAddInput$PhoneNumber:PhoneNumberAddInput$PaymentCard:PaymentCardAddInput$MediaContent:MediaContentAddInput){stixCyberObservableAdd(type:$type stix_id:$stix_id x_opencti_score:$x_opencti_score x_opencti_description:$x_opencti_description createIndicator:$createIndicator createdBy:$createdBy objectMarking:$objectMarking objectLabel:$objectLabel update:$update externalReferences:$externalReferences objectOrganization:$objectOrganization AutonomousSystem:$AutonomousSystem Directory:$Directory DomainName:$DomainName EmailAddr:$EmailAddr EmailMessage:$EmailMessage EmailMimePartType:$EmailMimePartType Artifact:$Artifact StixFile:$StixFile X509Certificate:$X509Certificate IPv4Addr:$IPv4Addr IPv6Addr:$IPv6Addr MacAddr:$MacAddr Mutex:$Mutex NetworkTraffic:$NetworkTraffic Process:$Process Software:$Software Url:$Url UserAccount:$UserAccount WindowsRegistryKey:$WindowsRegistryKey WindowsRegistryValueType:$WindowsRegistryValueType CryptographicKey:$CryptographicKey CryptocurrencyWallet:$CryptocurrencyWallet Hostname:$Hostname Text:$Text UserAgent:$UserAgent BankAccount:$BankAccount PhoneNumber:$PhoneNumber PaymentCard:$PaymentCard MediaContent:$MediaContent){id standard_id entity_type parent_types indicators{edges{node{id pattern pattern_type}}}}}",
    "size": 1213,
    "time": 124273,
    "timestamp": "2024-02-03T07:10:08.247Z",
    "type": "WRITE_ERROR",
    "user": {
      "applicant_id": "6a4b11e1-90ca-4e42-ba42-db7bc7f7d505",
      "call_retry_number": "1",
      "group_ids": [],
      "ip": "192.168.6.27",
      "organization_ids": [],
      "socket": "query",
      "user_id": "6a4b11e1-90ca-4e42-ba42-db7bc7f7d505"
    },
    "version": "5.12.15"
  }

Cannot read properties of null (reading 'split')

  {
  "category": "APP",
  "errors": [
    {
      "attributes": {
        "genre": "TECHNICAL",
        "http_status": 500
      },
      "message": "Cannot read properties of null (reading 'split')",
      "name": "UNKNOWN_ERROR",
      "stack": "UNKNOWN_ERROR: Cannot read properties of null (reading 'split')\n    at error (/opt/opencti/build/src/config/errors.js:8:10)\n    at UnknownError (/opt/opencti/build/src/config/errors.js:70:47)\n    at Object._logWithError (/opt/opencti/build/src/config/conf.js:311:23)\n    at Object.warn (/opt/opencti/build/src/config/conf.js:320:47)\n    at Object.willSendResponse (/opt/opencti/build/src/graphql/loggerPlugin.js:100:20)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Promise.all (index 1)\n    at b (/opt/opencti/build/node_modules/apollo-server-core/src/requestPipeline.ts:530:5)\n    at processHTTPRequest (/opt/opencti/build/node_modules/apollo-server-core/src/runHttpQuery.ts:437:24)"
    },
    {
      "message": "Cannot read properties of null (reading 'split')",
      "name": "TypeError",
      "stack": "TypeError: Cannot read properties of null (reading 'split')\n    at cleanStixIds (/opt/opencti/build/src/database/stix.ts:104:29)\n    at rebuildAndMergeInputFromExistingData (/opt/opencti/build/src/database/middleware.js:974:16)\n    at mergeInput (/opt/opencti/build/src/database/middleware.js:998:33)\n    at fn (/opt/opencti/build/src/database/middleware.js:999:39)\n    at _map (/opt/opencti/build/node_modules/ramda/es/internal/_map.js:7:19)\n    at map (/opt/opencti/build/node_modules/ramda/es/map.js:62:14)\n    at fn (/opt/opencti/build/node_modules/ramda/es/internal/_dispatchable.js:44:15)\n    at map_default (/opt/opencti/build/node_modules/ramda/es/internal/_curry2.js:28:14)\n    at mergeInstanceWithUpdateInputs (/opt/opencti/build/src/database/middleware.js:999:28)\n    at updateAttributeMetaResolved (/opt/opencti/build/src/database/middleware.js:1860:19)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at upsertElement (/opt/opencti/build/src/database/middleware.js:2682:20)"
    }
  ],
  "inner_relation_creation": 1,
  "level": "warn",
  "message": "Platform unmanaged direct error",
  "operation": "StixSightingRelationshipAdd",
  "operation_query": "mutation StixSightingRelationshipAdd($input:StixSightingRelationshipAddInput!){stixSightingRelationshipAdd(input:$input){id standard_id entity_type parent_types}}",
  "size": 620,
  "time": 137,
  "timestamp": "2024-02-03T07:08:23.700Z",
  "type": "WRITE_ERROR",
  "user": {
    "applicant_id": "6a4b11e1-90ca-4e42-ba42-db7bc7f7d505",
    "call_retry_number": "1",
    "group_ids": [],
    "ip": "192.168.7.128",
    "organization_ids": [],
    "socket": "query",
    "user_id": "6a4b11e1-90ca-4e42-ba42-db7bc7f7d505"
  },
  "version": "5.12.15"
}

[REDIS] Failed to extend resource

{
  "category": "APP",
  "errors": [
    {
      "attributes": {
        "genre": "TECHNICAL",
        "http_status": 500
      },
      "message": "Unexpected error value: \"[REDIS] Failed to extend resource\"",
      "name": "UNKNOWN_ERROR",
      "stack": "UNKNOWN_ERROR: Unexpected error value: \"[REDIS] Failed to extend resource\"\n    at error (/opt/opencti/build/src/config/errors.js:8:10)\n    at UnknownError (/opt/opencti/build/src/config/errors.js:70:47)\n    at Object._logWithError (/opt/opencti/build/src/config/conf.js:311:23)\n    at Object.warn (/opt/opencti/build/src/config/conf.js:320:47)\n    at Object.willSendResponse (/opt/opencti/build/src/graphql/loggerPlugin.js:100:20)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Promise.all (index 1)\n    at b (/opt/opencti/build/node_modules/apollo-server-core/src/requestPipeline.ts:530:5)\n    at processHTTPRequest (/opt/opencti/build/node_modules/apollo-server-core/src/runHttpQuery.ts:437:24)"
    },
    {
      "message": "Unexpected error value: \"[REDIS] Failed to extend resource\"",
      "name": "NonErrorThrown",
      "stack": "NonErrorThrown: Unexpected error value: \"[REDIS] Failed to extend resource\"\n    at toError (/opt/opencti/build/node_modules/graphql/jsutils/toError.js:16:7)\n    at locatedError (/opt/opencti/build/node_modules/graphql/error/locatedError.js:20:38)\n    at /opt/opencti/build/node_modules/graphql/execution/execute.js:514:41\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at k (/opt/opencti/build/node_modules/apollo-server-core/src/requestPipeline.ts:504:14)\n    at processGraphQLRequest (/opt/opencti/build/node_modules/apollo-server-core/src/requestPipeline.ts:410:22)\n    at processHTTPRequest (/opt/opencti/build/node_modules/apollo-server-core/src/runHttpQuery.ts:437:24)"
    }
  ],
  "inner_relation_creation": 0,
  "level": "warn",
  "message": "Platform unmanaged direct error",
  "operation": "StixCoreRelationshipAdd",
  "operation_query": "mutation StixCoreRelationshipAdd($input:StixCoreRelationshipAddInput!){stixCoreRelationshipAdd(input:$input){id standard_id entity_type parent_types}}",
  "size": 598,
  "time": 3266,
  "timestamp": "2024-02-03T07:06:17.543Z",
  "type": "WRITE_ERROR",
  "user": {
    "applicant_id": "6a4b11e1-90ca-4e42-ba42-db7bc7f7d505",
    "call_retry_number": "1",
    "group_ids": [],
    "ip": "192.168.1.111",
    "organization_ids": [],
    "socket": "query",
    "user_id": "6a4b11e1-90ca-4e42-ba42-db7bc7f7d505"
  },
  "version": "5.12.15"
}
Jipegien commented 7 months ago

Hello! I don't understand what you are asking, sorry. You need help to resolve errors' causes or is the problem about the content of the error log ?

MaxwellDPS commented 7 months ago

Yeah, these seem fairly fatial, bulk index fails and DB errors occurring 3K times in a day seems not great. Wanting to figure out if this is an issue with the data of or platform.

These should be addressed, they are all bugs otherwise what are these error logs indicating?

I can get logs preceeding these, but Please LMK what further info y'all want.

Happy to open all these as separate issues :)