This project consists of an open source library allowing software to connect to data repositories using STIX Patterning, and return results as STIX Observations.
$ stix-shifter translate elastic_ecs query {} "[process:command_line = 'C:\Windows\system32\svchost.exe -k netsvcs -p -s Schedule']" {}
line 1:24 mismatched input ''' expecting {IntNegLiteral, IntPosLiteral, FloatNegLiteral, FloatPosLiteral, HexLiteral, BinaryLiteral, StringLiteral, BoolLiteral, TimestampLiteral}
2021-07-02 12:49:12,542 stix_shifter.stix_translation.stix_translation ERROR Caught exception: pop from empty list <class 'stix_shifter_utils.stix_translation.src.patterns.parser.ParserError'>
2021-07-02 12:49:12,543 stix_shifter_utils.stix_translation.stix_translation_error_mapper ERROR received exception => ParserError: pop from empty list
{
"success": false,
"code": "invalid_parameter",
"error": "Error when converting STIX pattern to data source query: pop from empty list"
}
$ stix-shifter translate elastic_ecs query {} "[process:name = 'svchost.exe']" {}
2021-07-02 12:49:46,413 stix_shifter_modules.elastic_ecs.stix_translation.query_translator INFO Converting STIX2 Pattern to data source query
{
"queries": [
"(process.name : \"svchost.exe\" OR process.parent.name : \"svchost.exe\") AND (@timestamp:[\"2021-07-02T16:44:46.413Z\" TO \"2021-07-02T16:49:46.413Z\"])"
]
}
- `qradar` connector example:
$ stix-shifter translate qradar query {} "[process:command_line = 'C:\Windowssystem32svchost.exe'] START t'2021-04-01T00:00:00.000Z' STOP t'2021-04-06T00:00:00.000Z'"
line 1:24 mismatched input ''' expecting {IntNegLiteral, IntPosLiteral, FloatNegLiteral, FloatPosLiteral, HexLiteral, BinaryLiteral, StringLiteral, BoolLiteral, TimestampLiteral}
2021-10-05 22:50:56,409 stix_shifter.stix_translation.stix_translation ERROR Caught exception: pop from empty list <class 'stix_shifter_utils.stix_translation.src.patterns.parser.ParserError'>
2021-10-05 22:50:56,410 stix_shifter_utils.stix_translation.stix_translation_error_mapper ERROR received exception => ParserError: pop from empty list
{
"success": false,
"code": "invalid_parameter",
"error": "Error when converting STIX pattern to data source query: pop from empty list"
}
$ stix-shifter translate qradar query {} "[process:command_line = 'C:Windowssystem32svchost.exe'] START t'2021-04-01T00:00:00.000Z' STOP t'2021-04-06T00:00:00.000Z'"
2021-10-05 22:53:18,033 stix_shifter_modules.qradar.stix_translation.query_translator INFO Converting STIX2 Pattern to ariel
{
"queries": [
"SELECT QIDNAME(qid) as qidname, qid, CATEGORYNAME(category) as categoryname, category as categoryid, CATEGORYNAME(highlevelcategory) as high_level_category_name, highlevelcategory as high_level_category_id, logsourceid, devicetype, LOGSOURCETYPENAME(devicetype) as logsourcetypename, LOGSOURCENAME(logsourceid) as logsourcename, starttime, endtime, devicetime, sourceaddress as sourceip, sourceport, sourcemac, destinationaddress as destinationip, destinationport, destinationmac, username, eventdirection as direction, identityip, identityhostname, eventcount, PROTOCOLNAME(protocolid) as protocol, UTF8(payload) as eventpayload, URL as url, magnitude, Filename as filename, \"File Hash\" as filehash, \"SHA1 Hash\" as sha1hash, \"SHA256 Hash\" as sha256hash, \"MD5 Hash\" as md5hash, \"File Path\" as filepath, severity as eventseverity, credibility, relevance, sourcegeographiclocation as sourcegeographic, destinationgeographiclocation as destinationgeographic, \"CRE Name\" as crename, \"CRE Description\" as credescription, creeventlist, rulename(creeventlist) as rulenames, domainid, DOMAINNAME(domainid) as domainname, EventID, Image, ParentImage, \"Process CommandLine\" as ProcessCommandLine, ParentCommandLine, TargetImage, \"Granted Access\" as GrantedAccess, \"Call Trace\" as CallTrace, SourceImage, PipeName, StartModule, StartFunction, Signed, Message, \"Registry Value Name\" as RegistryValueName, \"IMP Hash\" as IMPHash, ServiceFileName, \"Registry Key\" as RegistryKey, ObjectName, UrlHost, \"Process Name\" as ProcessName, \"Process ID\" as ProcessId, \"Parent Process ID\" as ParentProcessId, hasoffense FROM events WHERE (ProcessCommandLine = 'C:Windowssystem32svchost.exe' OR ParentCommandLine = 'C:Windowssystem32svchost.exe') limit 10000 START 1617235200000 STOP 1617667200000"
]
}
**To Solve**
The main parser needs update: https://github.com/opencybersecurityalliance/stix-shifter/tree/develop/stix_shifter_utils/stix_translation/src/patterns
Describe the bug The escape rules for STIX pattern are confusing. In order to match
One need to write the pattern as
This does not follow the STIX pattern doc, e.g.,
directory:path LIKE 'C:\\Windows\\%\\foo'
in http://docs.oasis-open.org/cti/stix/v2.0/stix-v2.0-part5-stix-patterning.htmlThe following does not work:
And one-level escape does not work, either:
To Reproduce
elastic_ecs
connector example:$ stix-shifter translate elastic_ecs query {} "[process:name = 'svchost.exe']" {} 2021-07-02 12:49:46,413 stix_shifter_modules.elastic_ecs.stix_translation.query_translator INFO Converting STIX2 Pattern to data source query { "queries": [ "(process.name : \"svchost.exe\" OR process.parent.name : \"svchost.exe\") AND (@timestamp:[\"2021-07-02T16:44:46.413Z\" TO \"2021-07-02T16:49:46.413Z\"])" ] }
$ stix-shifter translate qradar query {} "[process:command_line = 'C:\Windowssystem32svchost.exe'] START t'2021-04-01T00:00:00.000Z' STOP t'2021-04-06T00:00:00.000Z'" line 1:24 mismatched input ''' expecting {IntNegLiteral, IntPosLiteral, FloatNegLiteral, FloatPosLiteral, HexLiteral, BinaryLiteral, StringLiteral, BoolLiteral, TimestampLiteral} 2021-10-05 22:50:56,409 stix_shifter.stix_translation.stix_translation ERROR Caught exception: pop from empty list <class 'stix_shifter_utils.stix_translation.src.patterns.parser.ParserError'> 2021-10-05 22:50:56,410 stix_shifter_utils.stix_translation.stix_translation_error_mapper ERROR received exception => ParserError: pop from empty list { "success": false, "code": "invalid_parameter", "error": "Error when converting STIX pattern to data source query: pop from empty list" }
$ stix-shifter translate qradar query {} "[process:command_line = 'C:Windowssystem32svchost.exe'] START t'2021-04-01T00:00:00.000Z' STOP t'2021-04-06T00:00:00.000Z'" 2021-10-05 22:53:18,033 stix_shifter_modules.qradar.stix_translation.query_translator INFO Converting STIX2 Pattern to ariel { "queries": [ "SELECT QIDNAME(qid) as qidname, qid, CATEGORYNAME(category) as categoryname, category as categoryid, CATEGORYNAME(highlevelcategory) as high_level_category_name, highlevelcategory as high_level_category_id, logsourceid, devicetype, LOGSOURCETYPENAME(devicetype) as logsourcetypename, LOGSOURCENAME(logsourceid) as logsourcename, starttime, endtime, devicetime, sourceaddress as sourceip, sourceport, sourcemac, destinationaddress as destinationip, destinationport, destinationmac, username, eventdirection as direction, identityip, identityhostname, eventcount, PROTOCOLNAME(protocolid) as protocol, UTF8(payload) as eventpayload, URL as url, magnitude, Filename as filename, \"File Hash\" as filehash, \"SHA1 Hash\" as sha1hash, \"SHA256 Hash\" as sha256hash, \"MD5 Hash\" as md5hash, \"File Path\" as filepath, severity as eventseverity, credibility, relevance, sourcegeographiclocation as sourcegeographic, destinationgeographiclocation as destinationgeographic, \"CRE Name\" as crename, \"CRE Description\" as credescription, creeventlist, rulename(creeventlist) as rulenames, domainid, DOMAINNAME(domainid) as domainname, EventID, Image, ParentImage, \"Process CommandLine\" as ProcessCommandLine, ParentCommandLine, TargetImage, \"Granted Access\" as GrantedAccess, \"Call Trace\" as CallTrace, SourceImage, PipeName, StartModule, StartFunction, Signed, Message, \"Registry Value Name\" as RegistryValueName, \"IMP Hash\" as IMPHash, ServiceFileName, \"Registry Key\" as RegistryKey, ObjectName, UrlHost, \"Process Name\" as ProcessName, \"Process ID\" as ProcessId, \"Parent Process ID\" as ParentProcessId, hasoffense FROM events WHERE (ProcessCommandLine = 'C:Windowssystem32svchost.exe' OR ParentCommandLine = 'C:Windowssystem32svchost.exe') limit 10000 START 1617235200000 STOP 1617667200000" ] }