search

opencybersecurityalliance / firepit

Firepit - STIX Columnar Storage
Apache License 2.0
15 stars 12 forks source link

fast translation error #85

Closed subbyte closed 1 year ago

subbyte commented 1 year ago

Looks like I get an error with fast translation enabled (executed successfully with fast translation disabled).

proc = GET process FROM stixshifter://host101
       WHERE name = 'cmd.exe'
       START 2021-09-29T00:00:00Z STOP 2021-09-30T00:00:00Z

Where host101 points to our rainbow instance with indexes winlogbeat-* and beats dialects enabled (dialect is not an issue since it works with fast translation disabled).

Error

Screenshot 2023-04-12 at 2 41 05 PM

Log

  File "/Users/subx/venv/kestrel-release/lib/python3.9/site-packages/kestrel/codegen/commands.py", line 622, in _prefetch
    resp = ds_manager.query(data_source, stix_pattern, session_id, store)
  File "/Users/subx/venv/kestrel-release/lib/python3.9/site-packages/kestrel/datasource/manager.py", line 33, in query
    rs = i.query(uri, pattern, session_id, c, store)
  File "/Users/subx/venv/kestrel-release/lib/python3.9/site-packages/kestrel_datasource_stixshifter/interface.py", line 288, in query
    fast_translate(
  File "/Users/subx/venv/kestrel-release/lib/python3.9/site-packages/kestrel_datasource_stixshifter/interface.py", line 358, in fast_translate
    loop.run_until_complete(
  File "/Users/subx/venv/kestrel-release/lib/python3.9/site-packages/nest_asyncio.py", line 90, in run_until_complete
    return f.result()
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/futures.py", line 201, in result
    raise self._exception
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/tasks.py", line 256, in __step
    result = coro.send(None)
  File "/Users/subx/venv/kestrel-release/lib/python3.9/site-packages/firepit/aio/ingest.py", line 620, in ingest
    odf.columns = [c.rpartition(':')[2] for c in cols]
  File "/Users/subx/venv/kestrel-release/lib/python3.9/site-packages/pandas/core/generic.py", line 6002, in __setattr__
    return object.__setattr__(self, name, value)
  File "pandas/_libs/properties.pyx", line 69, in pandas._libs.properties.AxisProperty.__set__
  File "/Users/subx/venv/kestrel-release/lib/python3.9/site-packages/pandas/core/generic.py", line 730, in _set_axis
    self._mgr.set_axis(axis, labels)
  File "/Users/subx/venv/kestrel-release/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 225, in set_axis
    self._validate_set_axis(axis, new_labels)
  File "/Users/subx/venv/kestrel-release/lib/python3.9/site-packages/pandas/core/internals/base.py", line 70, in _validate_set_axis
    raise ValueError(
ValueError: Length mismatch: Expected axis has 7 elements, new values have 5 elements
subbyte commented 1 year ago

Python 3.9.6 on macOS with pip list results:

Package                          Version
-------------------------------- -----------
adal                             1.2.7
antlr4-python3-runtime           4.8
anyio                            3.6.2
anytree                          2.8.0
appdirs                          1.4.4
appnope                          0.1.3
argon2-cffi                      21.3.0
argon2-cffi-bindings             21.2.0
asttokens                        2.2.1
attrs                            22.2.0
Babel                            2.12.1
backcall                         0.2.0
beautifulsoup4                   4.12.2
bleach                           6.0.0
boto3                            1.26.84
botocore                         1.29.111
cattrs                           22.2.0
certifi                          2022.12.7
cffi                             1.15.1
charset-normalizer               3.1.0
click                            8.1.3
colorama                         0.4.6
colorlog                         6.7.0
comm                             0.1.3
cpe                              1.2.1
cryptography                     39.0.2
debugpy                          1.6.7
decorator                        5.1.1
deepmerge                        1.1.0
defusedxml                       0.7.1
docker                           6.0.1
exceptiongroup                   1.1.1
executing                        1.2.0
fastjsonschema                   2.16.3
firepit                          2.3.15
Flask                            2.2.3
flatten-json                     0.1.13
idna                             3.4
ijson                            3.2.0.post0
importlib-metadata               6.3.0
ipykernel                        6.22.0
ipython                          8.12.0
ipython-genutils                 0.2.0
itsdangerous                     2.1.2
jedi                             0.18.2
Jinja2                           3.1.2
jmespath                         1.0.1
json-fix                         0.5.1
json5                            0.9.11
jsonmerge                        1.9.0
jsonschema                       4.17.3
jupyter_client                   8.1.0
jupyter_core                     5.3.0
jupyter-server                   1.23.6
jupyterlab                       3.5.3
jupyterlab-pygments              0.2.2
jupyterlab_server                2.15.2
kestrel-jupyter                  1.0.7
kestrel-lang                     1.5.12
lark                             1.1.5
lark-parser                      0.12.0
lxml                             4.9.2
MarkupSafe                       2.1.2
matplotlib-inline                0.1.6
mistune                          2.0.5
nbclassic                        0.5.5
nbclient                         0.7.3
nbconvert                        7.3.1
nbformat                         5.8.0
nest-asyncio                     1.5.6
notebook                         6.5.4
notebook_shim                    0.2.2
numpy                            1.24.2
packaging                        23.1
pandas                           2.0.0
pandocfilters                    1.5.0
parso                            0.8.3
pexpect                          4.8.0
pickleshare                      0.7.5
pip                              23.0.1
platformdirs                     3.2.0
prometheus-client                0.16.0
prompt-toolkit                   3.0.38
psutil                           5.9.4
ptyprocess                       0.7.0
pure-eval                        0.2.2
pyarrow                          11.0.0
pycparser                        2.21
Pygments                         2.15.0
PyJWT                            2.6.0
pyOpenSSL                        23.0.0
pyrsistent                       0.19.3
python-dateutil                  2.8.2
pytz                             2023.3
PyYAML                           6.0
pyzmq                            25.0.2
requests                         2.28.2
requests-cache                   1.0.1
requests-toolbelt                0.10.1
s3transfer                       0.6.0
Send2Trash                       1.8.0
setuptools                       67.6.1
simplejson                       3.19.1
six                              1.16.0
sniffio                          1.3.0
soupsieve                        2.4
stack-data                       0.6.2
stix-shifter                     4.6.2
stix-shifter-modules-elastic-ecs 4.6.2
stix-shifter-utils               4.6.2
stix2-matcher                    3.0.0
stix2-patterns                   1.3.2
stix2-validator                  3.0.2
tabulate                         0.9.0
terminado                        0.17.1
tinycss2                         1.2.1
tomli                            2.0.1
tornado                          6.2
traitlets                        5.9.0
typeguard                        3.0.2
typer                            0.7.0
typing_extensions                4.5.0
tzdata                           2023.3
ujson                            5.7.0
url-normalize                    1.4.3
urllib3                          1.26.15
wcwidth                          0.2.6
webencodings                     0.5.1
websocket-client                 1.5.1
Werkzeug                         2.2.3
wheel                            0.40.0
xmltodict                        0.13.0
zipp                             3.15.0
pcoccoli commented 1 year ago

I think the problem involves stix-shifter mappings that map multiple native result fields to the same STIX property; I will need to handle that in the ingest() function. Maybe combining the duplicate columns into 1 (taking the first non-null value). I noticed the problem doesn't happen with stix-shifter 4.6.1 but it does with 4.6.2. I can't explain why though; the mappings in question don't seem to have changed between those versions.

pcoccoli commented 1 year ago

Fixed by 464f7d0380f7fd798049bd580d36140da6435a39