search

OpenCTI-Platform / opencti

Open Cyber Threat Intelligence Platform
https://opencti.io
Other
6.37k stars 942 forks source link

After migration from release 5.3.17 to 5.4.1, OpenCTI fails and does not start (crash). #2608

Open Aspidoquelona opened 1 year ago

Aspidoquelona commented 1 year ago

Description

When upgrading from release 5.3.17 (manually deployed and currently working) to this new release 5.4.1. I've got the following error (console):

TypeError: Uis.PyInterpreter is not a constructor
    at new Interpreter (/usr/local/opencti/build/node_modules/node-calls-python/index.js:45:19)
    at /usr/local/opencti/build/node_modules/node-calls-python/index.js:169:10
    at /usr/local/opencti/build/back.js:1:407
    at Object.<anonymous> (/usr/local/opencti/build/src/python/pythonBridge.js:3:34)
    at Module._compile (node:internal/modules/cjs/loader:1097:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1151:10)
    at Module.load (node:internal/modules/cjs/loader:975:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12)
    at node:internal/main/run_main_module:17:47

Environment

  1. OS: Ubuntu 20.04.5 LTS
  2. OpenCTI version: OpenCTI 5.4.1
  3. OpenCTI client: frontend
  4. Other environment details: Previous version 5.3.17 installed and running ok. The downloaded archive is: https://github.com/OpenCTI-Platform/opencti/releases/download/5.4.1/opencti-release-5.4.1.tar.gz

Reproducible Steps

Stop all services. Replace "opencti" directory (release 5.3.17) with the new one (5.4.1). Install python requirements.txt. Configure production.json. Start services. Start opencti app by command line: yarn serv.

Output (console):

TypeError: Uis.PyInterpreter is not a constructor
    at new Interpreter (/usr/local/opencti/build/node_modules/node-calls-python/index.js:45:19)
    at /usr/local/opencti/build/node_modules/node-calls-python/index.js:169:10
    at /usr/local/opencti/build/back.js:1:407
    at Object.<anonymous> (/usr/local/opencti/build/src/python/pythonBridge.js:3:34)
    at Module._compile (node:internal/modules/cjs/loader:1097:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1151:10)
    at Module.load (node:internal/modules/cjs/loader:975:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12)
    at node:internal/main/run_main_module:17:47

Steps to create the smallest reproducible scenario:

Same as above.

Expected Output

OpenCTI app running.

Actual Output

OpenCTI crashes with the error indicated above.

Additional information

I've noticed that some files have dos format (carriage return and line feed pairs as their newline characters).

Example: opencti/node_modules/node-calls-python/index.js

command: cat -A opencti/node_modules/node-calls-python/index.js | head

output (note the ^M chars): 

const { execSync } = require("child_process");^M$
const fs = require("fs");^M$
const path = require("path");^M$
const nodecallspython = require("./build/Release/nodecallspython.node");$
^M$
class Interpreter^M$
{^M$
    loadPython(dir)^M$
    {^M$
        const debug = process.env.NODECALLSPYTHON_DEBUG !== undefined;^M$
.
.
.

NOTE: I suppose that error message Uis.PyInterpreter is not a constructor correponds to this.PyInterpreter is not a constructor, but because of the "dos format" issue in index.js* file, it displays "U" instead o "th".

Screenshots (optional)

smclinden commented 1 year ago

I have the very same issue. I got part way there by deleting node_modules and doing a yarn rebuild but now I'm getting a different set of errors.

smclinden commented 1 year ago

There are also breaking changes to the JSON configuration file which, I submit, should be documented.

Currrently I am stuck at:

{"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running connector manager","timestamp":"2022-12-02T14:23:42.586Z","version":"5.4.1"}
{"category":"APP","error":{"context":{},"message":"Second argument is not a number","name":"TypeError","stack":"TypeError: Second argument is not a number\n    at setIntervalAsync (src/opencti/build/node_modules/set-interval-async/dist/fixed/set-interval-async.cjs:19:15)\n    at Object.start (src/opencti/build/src/manager/connectorManager.js:124:19)\n    at startModules (src/opencti/build/src/modules.js:46:28)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at boot (src/opencti/build/src/boot.js:16:5)"},"level":"error","message":"[OPENCTI] Platform start fail","timestamp":"2022-12-02T14:23:42.586Z","version":"5.4.1"}
richard-julien commented 1 year ago

Hi some information about the initial problem. To speedup the integration of indicator we introduce a new lib that open a python handle with a native express plugin. Unfortunalty as a native plugin this lib needs to be compiled on the target environment. So like you observed, a yarn install is now required to rebuild this lib.

For the breaking change on the configuration, it should be handle by the default.json file that contains default configuration that prevent any breaking configuration problem. Default.json should not be change on your environment. If its the case, please rollback the default.json to the original one and create a production.json overriding only the parameters you need.

smclinden commented 1 year ago

Ok, thanks. Did all that. Getting this error during the Dashboard rendering:

{"auth":{"email":"itsecops@opencti.org","ip":"10.223.130.132","user_id":"88ec0c6a-13ce-5e39-b486-354fe4a7084f"},"category":"AUDIT","lev
el":"info","message":"LOGIN","resource":{"provider":"Bearer"},"timestamp":"2022-12-02T15:27:35.022Z","version":"5.4.1"}
{"category":"APP","error":{"stacktrace":["TypeError: d is not iterable","at elDataConverter (src/opencti/build/src/d
atabase/engine.js:908:111)","at elFindByIds (src/opencti/build/src/database/engine.js:1075:25)","at runMicrotasks (<
anonymous>)","at processTicksAndRejections (node:internal/process/task_queues:96:5)","at convertAggregateDistributions (src/opencti/build/src/database/middleware.js:532:25)"]},"inner_relation_creation":0,"level":"error","message":"API Call","operat
ion":"StixCoreRelationshipsHorizontalBarsDistributionQuery","operation_query":"query StixCoreRelationshipsHorizontalBarsDistribution
Query($relationship_type:String!$toTypes:[String]$field:String!$operation:StatsOperation!$startDate:DateTime$endDate:DateTime$dateAt
tribute:String$limit:Int){stixCoreRelationshipsDistribution(relationship_type:$relationship_type toTypes:$toTypes field:$field opera
tion:$operation startDate:$startDate endDate:$endDate dateAttribute:$dateAttribute limit:$limit){label value entity{__typename ...on
 BasicObject{__isBasicObject:__typename entity_type}...on BasicRelationship{__isBasicRelationship:__typename entity_type}...on Attac
kPattern{name description id}...on Campaign{name description id}...on CourseOfAction{name description id}...on Individual{name descr
iption id}...on Organization{name description id}...on Sector{name description id}...on System{name description id}...on Indicator{n
ame description id}...on Infrastructure{name description id}...on IntrusionSet{name description id}...on Position{name description i
d}...on City{name description id}...on Country{name description id}...on Region{name description id}...on Malware{name description i
d}...on ThreatActor{name description id}...on Tool{name description id}...on Vulnerability{name description id}...on Incident{name d
escription id}...on Artifact{id}...on AutonomousSystem{id}...on BankAccount{id}...on Channel{id}...on CryptocurrencyWallet{id}...on
CryptographicKey{id}...on Directory{id}...on DomainName{id}...on EmailAddr{id}...on EmailMessage{id}...on EmailMimePartType{id}...on
 Event{id}...on ExternalReference{id}...on Grouping{id}...on Hostname{id}...on IPv4Addr{id}...on IPv6Addr{id}...on KillChainPhase{id
}...on Label{id}...on Language{id}...on MacAddr{id}...on MarkingDefinition{id}...on MediaContent{id}...on Mutex{id}...on Narrative{i
d}...on NetworkTraffic{id}...on Note{id}...on ObservedData{id}...on Opinion{id}...on PaymentCard{id}...on PhoneNumber{id}...on Proce
ss{id}...on Report{id}...on Software{id}...on StixCoreRelationship{id}...on StixCyberObservableRelationship{id}...on StixFile{id}...
on StixMetaRelationship{id}...on StixSightingRelationship{id}...on Text{id}...on Url{id}...on UserAccount{id}...on UserAgent{id}...o
n WindowsRegistryKey{id}...on WindowsRegistryValueType{id}...on X509Certificate{id}}}}","size":292,"time":1142,"timestamp":"2022-12-
02T15:27:35.076Z","type":"READ_ERROR","user":{"ip":"172.20.XX.YY", "referer":"https://opencti.org:4000/dashboard?","user_id":"XXXXXXXXXXXXXX"},"variables":{"dateAttribute":"created_at","endDate":null,"field":"internal_id","limit":10,"operation":"count","relationship_type":"stix-core-relationship","startDate":"2022-09-02T00:00:00-04:00","toTypes":["Threat-Actor","Intrusion-Set","Campaign","Malware","Tool","Vulnerability","Channel","Narrative"]},"version":"5.4.1"}
Aspidoquelona commented 1 year ago

Also did it. I've called "yarn install" and "yarn build" and now it runs without errors. Thanks a lot!