search

muchdogesec / cve2stix

A command line tool that turns NVD CVE records into STIX 2.1 Objects.
https://www.dogesec.com/
Apache License 2.0
4 stars 0 forks source link

Escape characters appear to be causing problems #17

Closed himynamesdave closed 1 month ago

himynamesdave commented 2 months ago 
2024-08-26 06:18:35,620 | INFO | CVE-> CVE-2008-5367
2024-08-26 06:18:35,780 | ERROR | Chord 'bc7ecede-8640-43f3-830c-726a7a1f983a' raised: ChordError('Dependency 405e4ead-c5fb-4557-ab9c-96f2d674d583 raised ObjectConfigurationError()')
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cve2stix/cve2stix/celery.py", line 35, in cve_syncing_task
    fetch_data(start, end, config)
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cve2stix/cve2stix/main.py", line 27, in fetch_data
    return fetch_url(urlunsplit(uri), config, parse_cve_api_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cve2stix/cve2stix/utils.py", line 44, in fetch_url
    response = callback(content, config)
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cve2stix/cve2stix/parse_api_response.py", line 239, in parse_cve_api_response
    config.fs.add(parse_cve_indicator(cve, vulnerability, config))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cve2stix/cve2stix/parse_api_response.py", line 189, in parse_cve_indicator
    indicator = Indicator(**indicator_dict)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/stix2/v21/sdo.py", line 250, in __init__
    super(Indicator, self).__init__(*args, **kwargs)
  File "/opt/homebrew/lib/python3.11/site-packages/stix2/base.py", line 232, in __init__
    self._check_object_constraints()
  File "/opt/homebrew/lib/python3.11/site-packages/stix2/v21/sdo.py", line 270, in _check_object_constraints
    raise InvalidValueError(self.__class__, 'pattern', str(errors[0]))
stix2.exceptions.InvalidValueError: Invalid value for Indicator 'pattern': FAIL: Error found at line 1:34. mismatched input 'itri' expecting ']'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/celery/backends/redis.py", line 528, in on_chord_part_return
    resl = [unpack(tup, decode) for tup in resl]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/celery/backends/redis.py", line 528, in <listcomp>
    resl = [unpack(tup, decode) for tup in resl]
            ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/celery/backends/redis.py", line 434, in _unpack_chord_result
    raise ChordError(f'Dependency {tid} raised {retval!r}')
celery.exceptions.ChordError: Dependency 405e4ead-c5fb-4557-ab9c-96f2d674d583 raised ObjectConfigurationError()
2024-08-26 06:18:35,789 | ERROR | Task cve2stix.celery.cve_syncing_task[405e4ead-c5fb-4557-ab9c-96f2d674d583] raised unexpected: ObjectConfigurationError()
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cve2stix/cve2stix/celery.py", line 35, in cve_syncing_task
    fetch_data(start, end, config)
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cve2stix/cve2stix/main.py", line 27, in fetch_data
    return fetch_url(urlunsplit(uri), config, parse_cve_api_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cve2stix/cve2stix/utils.py", line 44, in fetch_url
    response = callback(content, config)
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cve2stix/cve2stix/parse_api_response.py", line 239, in parse_cve_api_response
    config.fs.add(parse_cve_indicator(cve, vulnerability, config))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cve2stix/cve2stix/parse_api_response.py", line 189, in parse_cve_indicator
    indicator = Indicator(**indicator_dict)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/stix2/v21/sdo.py", line 250, in __init__
    super(Indicator, self).__init__(*args, **kwargs)
  File "/opt/homebrew/lib/python3.11/site-packages/stix2/base.py", line 232, in __init__
    self._check_object_constraints()
  File "/opt/homebrew/lib/python3.11/site-packages/stix2/v21/sdo.py", line 270, in _check_object_constraints
    raise InvalidValueError(self.__class__, 'pattern', str(errors[0]))
stix2.exceptions.ObjectConfigurationError
2024-08-26 06:18:35,790 | ERROR | 
Traceback (most recent call last):
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cxe2stix_helper.py", line 197, in <module>
    main()
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cxe2stix_helper.py", line 155, in main
    celery_task.get() # wait for it
    ^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/celery/result.py", line 251, in get
    return self.backend.wait_for_pending(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/celery/backends/asynchronous.py", line 221, in wait_for_pending
    for _ in self._wait_for_pending(result, **kwargs):
  File "/opt/homebrew/lib/python3.11/site-packages/celery/backends/asynchronous.py", line 287, in _wait_for_pending
    for _ in self.drain_events_until(
  File "/opt/homebrew/lib/python3.11/site-packages/celery/backends/asynchronous.py", line 58, in drain_events_until
    on_interval()
  File "/opt/homebrew/lib/python3.11/site-packages/vine/promises.py", line 161, in __call__
    return self.throw()
           ^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/vine/promises.py", line 158, in __call__
    retval = fun(*final_args, **final_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/celery/result.py", line 264, in _maybe_reraise_parent_error
    node.maybe_throw()
  File "/opt/homebrew/lib/python3.11/site-packages/celery/result.py", line 634, in maybe_throw
    result.maybe_throw(callback=callback, propagate=propagate)
  File "/opt/homebrew/lib/python3.11/site-packages/celery/result.py", line 365, in maybe_throw
    self.throw(value, self._to_remote_traceback(tb))
  File "/opt/homebrew/lib/python3.11/site-packages/celery/result.py", line 358, in throw
    self.on_ready.throw(*args, **kwargs)
  File "/opt/homebrew/lib/python3.11/site-packages/vine/promises.py", line 235, in throw
    reraise(type(exc), exc, tb)
  File "/opt/homebrew/lib/python3.11/site-packages/vine/utils.py", line 27, in reraise
    raise value
stix2.exceptions.ObjectConfigurationError
2024-08-26 06:18:35,790 | ERROR | 
Traceback (most recent call last):
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cxe2stix_helper.py", line 197, in <module>
    main()
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cxe2stix_helper.py", line 155, in main
    celery_task.get() # wait for it
    ^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/celery/result.py", line 251, in get
    return self.backend.wait_for_pending(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/celery/backends/asynchronous.py", line 221, in wait_for_pending
    for _ in self._wait_for_pending(result, **kwargs):
  File "/opt/homebrew/lib/python3.11/site-packages/celery/backends/asynchronous.py", line 287, in _wait_for_pending
    for _ in self.drain_events_until(
  File "/opt/homebrew/lib/python3.11/site-packages/celery/backends/asynchronous.py", line 58, in drain_events_until
    on_interval()
  File "/opt/homebrew/lib/python3.11/site-packages/vine/promises.py", line 161, in __call__
    return self.throw()
           ^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/vine/promises.py", line 158, in __call__
    retval = fun(*final_args, **final_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/celery/result.py", line 264, in _maybe_reraise_parent_error
    node.maybe_throw()
  File "/opt/homebrew/lib/python3.11/site-packages/celery/result.py", line 634, in maybe_throw
    result.maybe_throw(callback=callback, propagate=propagate)
  File "/opt/homebrew/lib/python3.11/site-packages/celery/result.py", line 365, in maybe_throw
    self.throw(value, self._to_remote_traceback(tb))
  File "/opt/homebrew/lib/python3.11/site-packages/celery/result.py", line 358, in throw
    self.on_ready.throw(*args, **kwargs)
  File "/opt/homebrew/lib/python3.11/site-packages/vine/promises.py", line 235, in throw
    reraise(type(exc), exc, tb)
  File "/opt/homebrew/lib/python3.11/site-packages/vine/utils.py", line 27, in reraise

The CVE patten has 

                "configurations": [
                    {
                        "nodes": [
                            {
                                "operator": "OR",
                                "negate": false,
                                "cpeMatch": [
                                    {
                                        "vulnerable": true,
                                        "criteria": "cpe:2.3:a:marco_d\\'itri:ppp-udeb:2.4.4:*:*:*:*:*:*:*",
                                        "matchCriteriaId": "108EBDF6-3D8A-434C-ADB9-02A4C64C67ED"
                                    }
                                ]
                            }
himynamesdave commented 1 month ago

Now processes above CVE (CVE-2008-5367) fine

Now fails on CVE-2013-0708

                "configurations": [
                    {
                        "nodes": [
                            {
                                "operator": "OR",
                                "negate": false,
                                "cpeMatch": [
                                    {
                                        "vulnerable": true,
                                        "criteria": "cpe:2.3:a:bayashi:dopvcomet\\*:0009:b:*:*:*:*:*:*",
                                        "matchCriteriaId": "60B3F93C-C271-4509-AEFF-F68E4B6445F6"
                                    }
                                ]
                            }
                        ]
                    }
                ],
2024-08-28 06:26:06,752 | ERROR | Chord '645e1a3f-83df-44f6-aad6-15b96acf7091' raised: ChordError('Dependency 52362353-0453-4164-ac10-522d4b0de452 raised ObjectConfigurationError()')
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cve2stix/cve2stix/celery.py", line 35, in cve_syncing_task
    fetch_data(start, end, config)
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cve2stix/cve2stix/main.py", line 28, in fetch_data
    return fetch_url(urlunsplit(uri), config, parse_cve_api_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cve2stix/cve2stix/utils.py", line 44, in fetch_url
    response = callback(content, config)
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cve2stix/cve2stix/parse_api_response.py", line 240, in parse_cve_api_response
    config.fs.add(parse_cve_indicator(cve, vulnerability, config))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cve2stix/cve2stix/parse_api_response.py", line 190, in parse_cve_indicator
    indicator = Indicator(**indicator_dict)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/stix2/v21/sdo.py", line 250, in __init__
    super(Indicator, self).__init__(*args, **kwargs)
  File "/opt/homebrew/lib/python3.11/site-packages/stix2/base.py", line 232, in __init__
    self._check_object_constraints()
  File "/opt/homebrew/lib/python3.11/site-packages/stix2/v21/sdo.py", line 270, in _check_object_constraints
    raise InvalidValueError(self.__class__, 'pattern', str(errors[0]))
stix2.exceptions.InvalidValueError: Invalid value for Indicator 'pattern': FAIL: Error found at line 1:15. mismatched input ''' expecting {IntNegLiteral, IntPosLiteral, FloatNegLiteral, FloatPosLiteral, HexLiteral, BinaryLiteral, StringLiteral, BoolLiteral, TimestampLiteral}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/celery/backends/redis.py", line 528, in on_chord_part_return
    resl = [unpack(tup, decode) for tup in resl]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/celery/backends/redis.py", line 528, in <listcomp>
    resl = [unpack(tup, decode) for tup in resl]
            ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/celery/backends/redis.py", line 434, in _unpack_chord_result
    raise ChordError(f'Dependency {tid} raised {retval!r}')
celery.exceptions.ChordError: Dependency 52362353-0453-4164-ac10-522d4b0de452 raised ObjectConfigurationError()
2024-08-28 06:26:06,755 | ERROR | Task cve2stix.celery.cve_syncing_task[52362353-0453-4164-ac10-522d4b0de452] raised unexpected: ObjectConfigurationError()
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cve2stix/cve2stix/celery.py", line 35, in cve_syncing_task
    fetch_data(start, end, config)
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cve2stix/cve2stix/main.py", line 28, in fetch_data
    return fetch_url(urlunsplit(uri), config, parse_cve_api_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cve2stix/cve2stix/utils.py", line 44, in fetch_url
    response = callback(content, config)
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cve2stix/cve2stix/parse_api_response.py", line 240, in parse_cve_api_response
    config.fs.add(parse_cve_indicator(cve, vulnerability, config))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cve2stix/cve2stix/parse_api_response.py", line 190, in parse_cve_indicator
    indicator = Indicator(**indicator_dict)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/stix2/v21/sdo.py", line 250, in __init__
    super(Indicator, self).__init__(*args, **kwargs)
  File "/opt/homebrew/lib/python3.11/site-packages/stix2/base.py", line 232, in __init__
    self._check_object_constraints()
  File "/opt/homebrew/lib/python3.11/site-packages/stix2/v21/sdo.py", line 270, in _check_object_constraints
    raise InvalidValueError(self.__class__, 'pattern', str(errors[0]))
stix2.exceptions.ObjectConfigurationError
2024-08-28 06:26:06,755 | ERROR | 
Traceback (most recent call last):
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cxe2stix_helper.py", line 197, in <module>
    main()
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cxe2stix_helper.py", line 155, in main
    celery_task.get() # wait for it
    ^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/celery/result.py", line 250, in get
    return self.backend.wait_for_pending(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/celery/backends/asynchronous.py", line 221, in wait_for_pending
    for _ in self._wait_for_pending(result, **kwargs):
  File "/opt/homebrew/lib/python3.11/site-packages/celery/backends/asynchronous.py", line 287, in _wait_for_pending
    for _ in self.drain_events_until(
  File "/opt/homebrew/lib/python3.11/site-packages/celery/backends/asynchronous.py", line 58, in drain_events_until
    on_interval()
  File "/opt/homebrew/lib/python3.11/site-packages/vine/promises.py", line 161, in __call__
    return self.throw()
           ^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/vine/promises.py", line 158, in __call__
    retval = fun(*final_args, **final_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/celery/result.py", line 263, in _maybe_reraise_parent_error
    node.maybe_throw()
  File "/opt/homebrew/lib/python3.11/site-packages/celery/result.py", line 633, in maybe_throw
    result.maybe_throw(callback=callback, propagate=propagate)
  File "/opt/homebrew/lib/python3.11/site-packages/celery/result.py", line 364, in maybe_throw
    self.throw(value, self._to_remote_traceback(tb))
  File "/opt/homebrew/lib/python3.11/site-packages/celery/result.py", line 357, in throw
    self.on_ready.throw(*args, **kwargs)
  File "/opt/homebrew/lib/python3.11/site-packages/vine/promises.py", line 235, in throw
    reraise(type(exc), exc, tb)
  File "/opt/homebrew/lib/python3.11/site-packages/vine/utils.py", line 27, in reraise
    raise value
stix2.exceptions.ObjectConfigurationError
2024-08-28 06:26:06,755 | ERROR | 
Traceback (most recent call last):
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cxe2stix_helper.py", line 197, in <module>
    main()
  File "/Users/dgreenwood/Documents/repos/dogesec/cxe2stix_helper/cxe2stix_helper.py", line 155, in main
    celery_task.get() # wait for it
    ^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/celery/result.py", line 250, in get
    return self.backend.wait_for_pending(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/celery/backends/asynchronous.py", line 221, in wait_for_pending
    for _ in self._wait_for_pending(result, **kwargs):
  File "/opt/homebrew/lib/python3.11/site-packages/celery/backends/asynchronous.py", line 287, in _wait_for_pending
    for _ in self.drain_events_until(
  File "/opt/homebrew/lib/python3.11/site-packages/celery/backends/asynchronous.py", line 58, in drain_events_until
    on_interval()
  File "/opt/homebrew/lib/python3.11/site-packages/vine/promises.py", line 161, in __call__
    return self.throw()
           ^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/vine/promises.py", line 158, in __call__
    retval = fun(*final_args, **final_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/celery/result.py", line 263, in _maybe_reraise_parent_error
    node.maybe_throw()
  File "/opt/homebrew/lib/python3.11/site-packages/celery/result.py", line 633, in maybe_throw
    result.maybe_throw(callback=callback, propagate=propagate)
  File "/opt/homebrew/lib/python3.11/site-packages/celery/result.py", line 364, in maybe_throw
    self.throw(value, self._to_remote_traceback(tb))
  File "/opt/homebrew/lib/python3.11/site-packages/celery/result.py", line 357, in throw
    self.on_ready.throw(*args, **kwargs)
  File "/opt/homebrew/lib/python3.11/site-packages/vine/promises.py", line 235, in throw
    reraise(type(exc), exc, tb)
  File "/opt/homebrew/lib/python3.11/site-packages/vine/utils.py", line 27, in reraise
    raise value
stix2.exceptions.ObjectConfigurationError
2024-08-28 06:26:06,757 | INFO | Killing all child processes
2024-08-28 06:26:06,757 | INFO | Killing all child processes