search

jasonish / py-idstools

idstools: Snort and Suricata Rule and Event Utilities in Python (Including a Rule Update Tool)
Other
277 stars 85 forks source link

New release to support python 3.13 #95

Open marcindulak opened 1 year ago

marcindulak commented 1 year ago

Originally reported at https://bugzilla.redhat.com/show_bug.cgi?id=2245828

Example error from pytest:

______________________ AggregatorTestCase.test_aggregator ______________________

self = <test_unified2.AggregatorTestCase testMethod=test_aggregator>

    def test_aggregator(self):

>       aggregator = unified2.Aggregator()

tests/test_unified2.py:201: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <idstools.unified2.Aggregator object at 0x7f3c78cff500>

    def __init__(self):
>       LOG.warn("idstools.unified2.Aggregator has been deprecated")
E       AttributeError: 'Logger' object has no attribute 'warn'

../../BUILDROOT/python-idstools-0.6.4-11.fc40.x86_64/usr/lib/python3.13/site-packages/idstools/unified2.py:433: AttributeError

Python 3.13 removes logging.warn https://docs.python.org/3.13/whatsnew/3.13.html#logging

This change https://github.com/jasonish/py-idstools/commit/6a74ca15b930cbd05e952b2de652e6b2ee214209 implemented the fix, but it has not been released yet.

Additionally, it may be also helpful to review the SyntaxError: invalid escape sequence warnings. I think those are related to https://github.com/python/cpython/issues/77093. They are not treated as errors yet, but may become one day.

Here is a Dockerfile to trigger them by running pytest: docker build --progress=plain -t py-idstools:latest .

FROM python:3.13-rc-bookworm

RUN set -x \
    && apt-get update \
    && apt-get install -y git

RUN set -x \
    && pip install pytest

RUN set -x \
    && git clone https://github.com/jasonish/py-idstools/ \
    && cd py-idstools \
    && PYTHONPATH=. pytest -v

CMD ["/bin/bash"]

Output

#4 3.499 + PYTHONPATH=. pytest -v
#4 3.880 ============================= test session starts ==============================
#4 3.880 platform linux -- Python 3.13.0a1, pytest-7.4.2, pluggy-1.3.0 -- /usr/local/bin/python
...
#4 8.607 =============================== warnings summary ===============================
#4 8.607 tests/test_rule.py:40
#4 8.607   /py-idstools/tests/test_rule.py:40: SyntaxWarning: invalid escape sequence '\/'
#4 8.607     rule = idstools.rule.parse("""alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET CURRENT_EVENTS Request to .in FakeAV Campaign June 19 2012 exe or zip"; flow:established,to_server; content:"setup."; fast_pattern:only; http_uri; content:".in|0d 0a|"; flowbits:isset,somebit; flowbits:unset,otherbit; http_header; pcre:"/\/[a-f0-9]{16}\/([a-z0-9]{1,3}\/)?setup\.(exe|zip)$/U"; pcre:"/^Host\x3a\s.+\.in\r?$/Hmi"; metadata:stage,hostile_download; reference:url,isc.sans.edu/diary/+Vulnerabilityqueerprocessbrittleness/13501; classtype:trojan-activity; sid:2014929; rev:1;)""")
#4 8.607 
#4 8.607 tests/test_rule.py:152
#4 8.607   /py-idstools/tests/test_rule.py:152: SyntaxWarning: invalid escape sequence '\/'
#4 8.607     rule_string = """alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET CURRENT_EVENTS Request to .in FakeAV Campaign June 19 2012 exe or zip"; flow:established,to_server; content:"setup."; fast_pattern:only; http_uri; content:".in|0d 0a|"; flowbits:isset,somebit; flowbits:unset,otherbit; http_header; pcre:"/\/[a-f0-9]{16}\/([a-z0-9]{1,3}\/)?setup\.(exe|zip)$/U"; pcre:"/^Host\x3a\s.+\.in\r?$/Hmi"; metadata:stage,hostile_download; reference:url,isc.sans.edu/diary/+Vulnerabilityqueerprocessbrittleness/13501; classtype:trojan-activity; sid:2014929; rev:1;)"""
#4 8.607 
#4 8.607 tests/test_rule.py:172
#4 8.607   /py-idstools/tests/test_rule.py:172: SyntaxWarning: invalid escape sequence '\;'
#4 8.607     rule_string = u"""alert ip any any -> any any (msg:"TEST RULE\; and some"; content:"uid=0|28|root|29|"; tag:session,5,packets; classtype:bad-unknown; sid:10000000; rev:1;)"""
#4 8.607 
#4 8.607 tests/test_rule.py:175
#4 8.607   /py-idstools/tests/test_rule.py:175: SyntaxWarning: invalid escape sequence '\;'
#4 8.607     self.assertEqual(rule.msg, "TEST RULE\; and some")
#4 8.607 
#4 8.607 tests/test_rulecat.py:173
#4 8.607   /py-idstools/tests/test_rulecat.py:173: SyntaxWarning: invalid escape sequence '\|'
#4 8.607     line = '2020757 "\|0d 0a\|" "|ff ff|"'
#4 8.607 
#4 8.607 tests/test_rulecat.py:184
#4 8.607   /py-idstools/tests/test_rulecat.py:184: SyntaxWarning: invalid escape sequence '\|'
#4 8.607     line = 're:classtype:trojan-activity "\|0d 0a\|" "|ff ff|"'
#4 8.607 
#4 8.607 idstools/scripts/rulecat.py:225
#4 8.607   /py-idstools/idstools/scripts/rulecat.py:225: SyntaxWarning: invalid escape sequence '\$'
#4 8.607     b = re.sub("\$\{(\d+)\}", "\\\\\\1", b)
#4 8.607 
#4 8.607 idstools/scripts/rulecat.py:247
#4 8.607   /py-idstools/idstools/scripts/rulecat.py:247: SyntaxWarning: invalid escape sequence '\w'
#4 8.607     drop_rule = idstools.rule.parse(re.sub("^\w+", "drop", rule.raw))
#4 8.607 
#4 8.607 idstools/scripts/rulecat.py:595
#4 8.607   /py-idstools/idstools/scripts/rulecat.py:595: SyntaxWarning: invalid escape sequence '\s'
#4 8.607     re.compile("\s+(re:\"(.*)\")"),
#4 8.607 
#4 8.607 idstools/scripts/rulecat.py:596
#4 8.607   /py-idstools/idstools/scripts/rulecat.py:596: SyntaxWarning: invalid escape sequence '\s'
#4 8.607     re.compile("\s+(re:(.*?)),.*"),
#4 8.607 
#4 8.607 idstools/scripts/rulecat.py:597
#4 8.607   /py-idstools/idstools/scripts/rulecat.py:597: SyntaxWarning: invalid escape sequence '\s'
#4 8.607     re.compile("\s+(re:(.*))"),
#4 8.607 
#4 8.607 idstools/suricata.py:51
#4 8.607   /py-idstools/idstools/suricata.py:51: SyntaxWarning: invalid escape sequence '\d'
#4 8.607     m = re.search("((\d+)\.(\d+)(\.(\d+))?(\w+)?)", str(buf).strip())
#4 8.607 
#4 8.607 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
#4 8.607 ================== 76 passed, 2 skipped, 12 warnings in 4.73s ==================
jasonish commented 1 year ago

Staging this for release: https://github.com/jasonish/py-idstools/pull/96

jasonish commented 1 year ago

0.6.5 released.

marcindulak commented 1 year ago

Have you tested with the Dockerfile build docker build --progress=plain --no-cache --pull -t py-idstools:latest .

FROM python:3.13-rc-bookworm

RUN set -x \
    && apt-get update \
    && apt-get install -y git

RUN set -x \
    && pip install pytest

RUN set -x \
    && git clone https://github.com/jasonish/py-idstools/ \
    && cd py-idstools \
    && git checkout 51aebc019b5f3132aa6d63b5dc3c17b03a2f5c8c \
    && PYTHONPATH=. pytest -Werror -v

CMD ["/bin/bash"]

I'm still gettting some "invalid escape sequence" with https://github.com/jasonish/py-idstools/commit/51aebc019b5f3132aa6d63b5dc3c17b03a2f5c8c

#4 2.751 HEAD is now at 51aebc0 version: 0.6.5
#4 2.752 + PYTHONPATH=. pytest -Werror -v
#4 3.185 ============================= test session starts ==============================
#4 3.185 platform linux -- Python 3.13.0a1, pytest-7.4.3, pluggy-1.3.0 -- /usr/local/bin/python
#4 3.185 cachedir: .pytest_cache
#4 3.185 rootdir: /py-idstools
#4 3.185 collecting ... collected 33 items / 3 errors
#4 3.989 
#4 3.989 ==================================== ERRORS ====================================
#4 3.989 _____________________ ERROR collecting tests/test_rule.py ______________________
#4 3.989 /usr/local/lib/python3.13/site-packages/_pytest/python.py:617: in _importtestmodule
#4 3.989     mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
#4 3.989 /usr/local/lib/python3.13/site-packages/_pytest/pathlib.py:567: in import_path
#4 3.989     importlib.import_module(module_name)
#4 3.989 /usr/local/lib/python3.13/importlib/__init__.py:88: in import_module
#4 3.989     return _bootstrap._gcd_import(name[level:], package, level)
#4 3.989 <frozen importlib._bootstrap>:1381: in _gcd_import
#4 3.989     ???
#4 3.989 <frozen importlib._bootstrap>:1354: in _find_and_load
#4 3.989     ???
#4 3.989 <frozen importlib._bootstrap>:1325: in _find_and_load_unlocked
#4 3.989     ???
#4 3.989 <frozen importlib._bootstrap>:929: in _load_unlocked
#4 3.989     ???
#4 3.989 /usr/local/lib/python3.13/site-packages/_pytest/assertion/rewrite.py:177: in exec_module
#4 3.989     source_stat, co = _rewrite_test(fn, self.config)
#4 3.989 /usr/local/lib/python3.13/site-packages/_pytest/assertion/rewrite.py:359: in _rewrite_test
#4 3.989     tree = ast.parse(source, filename=strfn)
#4 3.989 /usr/local/lib/python3.13/ast.py:54: in parse
#4 3.989     return compile(source, filename, mode, flags,
#4 3.989 E     File "/py-idstools/tests/test_rule.py", line 40
#4 3.989 E       rule = idstools.rule.parse("""alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET CURRENT_EVENTS Request to .in FakeAV Campaign June 19 2012 exe or zip"; flow:established,to_server; content:"setup."; fast_pattern:only; http_uri; content:".in|0d 0a|"; flowbits:isset,somebit; flowbits:unset,otherbit; http_header; pcre:"/\/[a-f0-9]{16}\/([a-z0-9]{1,3}\/)?setup\.(exe|zip)$/U"; pcre:"/^Host\x3a\s.+\.in\r?$/Hmi"; metadata:stage,hostile_download; reference:url,isc.sans.edu/diary/+Vulnerabilityqueerprocessbrittleness/13501; classtype:trojan-activity; sid:2014929; rev:1;)""")
#4 3.989 E                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#4 3.989 E   SyntaxError: invalid escape sequence '\/'
#4 3.989 ____________________ ERROR collecting tests/test_rulecat.py ____________________
#4 3.989 /usr/local/lib/python3.13/site-packages/_pytest/python.py:617: in _importtestmodule
#4 3.989     mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
#4 3.989 /usr/local/lib/python3.13/site-packages/_pytest/pathlib.py:567: in import_path
#4 3.989     importlib.import_module(module_name)
#4 3.989 /usr/local/lib/python3.13/importlib/__init__.py:88: in import_module
#4 3.989     return _bootstrap._gcd_import(name[level:], package, level)
#4 3.989 <frozen importlib._bootstrap>:1381: in _gcd_import
#4 3.989     ???
#4 3.989 <frozen importlib._bootstrap>:1354: in _find_and_load
#4 3.989     ???
#4 3.989 <frozen importlib._bootstrap>:1325: in _find_and_load_unlocked
#4 3.989     ???
#4 3.989 <frozen importlib._bootstrap>:929: in _load_unlocked
#4 3.989     ???
#4 3.989 /usr/local/lib/python3.13/site-packages/_pytest/assertion/rewrite.py:186: in exec_module
#4 3.989     exec(co, module.__dict__)
#4 3.989 tests/test_rulecat.py:37: in <module>
#4 3.989     from idstools.scripts import rulecat
#4 3.989 idstools/scripts/rulecat.py:48: in <module>
#4 3.989     import idstools.suricata
#4 3.989 E     File "/py-idstools/idstools/suricata.py", line 51
#4 3.989 E       m = re.search("((\d+)\.(\d+)(\.(\d+))?(\w+)?)", str(buf).strip())
#4 3.989 E                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#4 3.989 E   SyntaxError: invalid escape sequence '\d'
#4 3.989 ___________________ ERROR collecting tests/test_suricata.py ____________________
#4 3.989 /usr/local/lib/python3.13/site-packages/_pytest/python.py:617: in _importtestmodule
#4 3.989     mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
#4 3.989 /usr/local/lib/python3.13/site-packages/_pytest/pathlib.py:567: in import_path
#4 3.989     importlib.import_module(module_name)
#4 3.989 /usr/local/lib/python3.13/importlib/__init__.py:88: in import_module
#4 3.989     return _bootstrap._gcd_import(name[level:], package, level)
#4 3.989 <frozen importlib._bootstrap>:1381: in _gcd_import
#4 3.989     ???
#4 3.989 <frozen importlib._bootstrap>:1354: in _find_and_load
#4 3.989     ???
#4 3.989 <frozen importlib._bootstrap>:1325: in _find_and_load_unlocked
#4 3.989     ???
#4 3.989 <frozen importlib._bootstrap>:929: in _load_unlocked
#4 3.989     ???
#4 3.989 /usr/local/lib/python3.13/site-packages/_pytest/assertion/rewrite.py:186: in exec_module
#4 3.989     exec(co, module.__dict__)
#4 3.989 tests/test_suricata.py:36: in <module>
#4 3.989     from idstools import suricata
#4 3.989 E     File "/py-idstools/idstools/suricata.py", line 51
#4 3.989 E       m = re.search("((\d+)\.(\d+)(\.(\d+))?(\w+)?)", str(buf).strip())
#4 3.989 E                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#4 3.989 E   SyntaxError: invalid escape sequence '\d'
#4 3.989 =========================== short test summary info ============================
#4 3.989 ERROR tests/test_rule.py
#4 3.989 ERROR tests/test_rulecat.py
#4 3.989 ERROR tests/test_suricata.py
#4 3.989 !!!!!!!!!!!!!!!!!!! Interrupted: 3 errors during collection !!!!!!!!!!!!!!!!!!!!
#4 3.989 ============================== 3 errors in 0.80s ===============================

When running pytest without -Werror tests succeed, with some SyntaxWarning: invalid escape sequence

#7 2.873 + PYTHONPATH=. pytest -v
#7 3.240 ============================= test session starts ==============================
#7 3.240 platform linux -- Python 3.13.0a1, pytest-7.4.3, pluggy-1.3.0 -- /usr/local/bin/python
#7 3.240 cachedir: .pytest_cache
#7 3.240 rootdir: /py-idstools
#7 3.240 collecting ... collected 78 items
...
#7 8.365 =============================== warnings summary ===============================
#7 8.365 tests/test_rule.py:40
#7 8.365   /py-idstools/tests/test_rule.py:40: SyntaxWarning: invalid escape sequence '\/'
#7 8.365     rule = idstools.rule.parse("""alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET CURRENT_EVENTS Request to .in FakeAV Campaign June 19 2012 exe or zip"; flow:established,to_server; content:"setup."; fast_pattern:only; http_uri; content:".in|0d 0a|"; flowbits:isset,somebit; flowbits:unset,otherbit; http_header; pcre:"/\/[a-f0-9]{16}\/([a-z0-9]{1,3}\/)?setup\.(exe|zip)$/U"; pcre:"/^Host\x3a\s.+\.in\r?$/Hmi"; metadata:stage,hostile_download; reference:url,isc.sans.edu/diary/+Vulnerabilityqueerprocessbrittleness/13501; classtype:trojan-activity; sid:2014929; rev:1;)""")
#7 8.365 
#7 8.365 tests/test_rule.py:152
#7 8.365   /py-idstools/tests/test_rule.py:152: SyntaxWarning: invalid escape sequence '\/'
#7 8.365     rule_string = """alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET CURRENT_EVENTS Request to .in FakeAV Campaign June 19 2012 exe or zip"; flow:established,to_server; content:"setup."; fast_pattern:only; http_uri; content:".in|0d 0a|"; flowbits:isset,somebit; flowbits:unset,otherbit; http_header; pcre:"/\/[a-f0-9]{16}\/([a-z0-9]{1,3}\/)?setup\.(exe|zip)$/U"; pcre:"/^Host\x3a\s.+\.in\r?$/Hmi"; metadata:stage,hostile_download; reference:url,isc.sans.edu/diary/+Vulnerabilityqueerprocessbrittleness/13501; classtype:trojan-activity; sid:2014929; rev:1;)"""
#7 8.365 
#7 8.365 tests/test_rule.py:172
#7 8.365   /py-idstools/tests/test_rule.py:172: SyntaxWarning: invalid escape sequence '\;'
#7 8.365     rule_string = u"""alert ip any any -> any any (msg:"TEST RULE\; and some"; content:"uid=0|28|root|29|"; tag:session,5,packets; classtype:bad-unknown; sid:10000000; rev:1;)"""
#7 8.365 
#7 8.365 tests/test_rule.py:175
#7 8.365   /py-idstools/tests/test_rule.py:175: SyntaxWarning: invalid escape sequence '\;'
#7 8.365     self.assertEqual(rule.msg, "TEST RULE\; and some")
#7 8.365 
#7 8.365 idstools/suricata.py:51
#7 8.365   /py-idstools/idstools/suricata.py:51: SyntaxWarning: invalid escape sequence '\d'
#7 8.365     m = re.search("((\d+)\.(\d+)(\.(\d+))?(\w+)?)", str(buf).strip())
#7 8.365 
#7 8.365 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
#7 8.365 ================== 76 passed, 2 skipped, 5 warnings in 5.13s ===================
jasonish commented 1 year ago

Ah, missed actually testing with 3.13 as I typically don't bother with non-released versions, as I'm very much not in the Python world anymore. Looks like easy fixes I can get to soon.

marcindulak commented 1 year ago

OK, the good thing is there is no need for new release, as SyntaxWarning: invalid escape sequence in Python 3.13 are still warnings and not errors.

jasonish commented 1 year ago

Will re-open and fix in the near future. Thanks.