Continuation \ lines issue for quoted args (in pre-3.3 CRS rules)

[prof-milki]

type: bug
category: yacc
severity: minor
priority: obsolete

So, I ran into this bug for processing REQUEST-901-INITIALIZATION.conf from CRS 3.2.x. MSCParser stumbles over the closing quote for the continued argument/line.

import msc_pyparser
msc_pyparser.MSCParser().parser.parse(r'''
SecRule TX:sampling_rnd100 "!@lt %{tx.sampling_percentage}" \
    "id:901450,\
    phase:1,\
    pass,\
    log,\
    noauditlog,\
    msg:'Sampling: Disable the rule engine based on sampling_percentage \
%{TX.sampling_percentage} and random number %{TX.sampling_rnd100}.',\
    ctl:ruleEngine=Off"
''')

The stack trace points to the closing ' at least. But I'd assume it's because the \\<LF> got overlooked. So likely an issue with the precedence or in Ply itself? (Just guessing.)

  File "/usr/lib/python3/dist-packages/ply/yacc.py", line 192, in call_errorfunc
    r = errorfunc(token)
  File "/home/mario/.local/lib/python3.8/site-packages/msc_pyparser.py", line 1038, in p_error
    raise Exception(output)
Exception: Parser error: syntax error in line 9 at pos 272, column 66
%{TX.sampling_percentage} and random number %{TX.sampling_rnd100}.',\
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^

Anyway, it seems that's fixed already in CRS 3.3, and I haven't noticed any similar hiccups otherwise. So really just pertains older releases (Debian still ships 3.2; but doubly obsolete now given the new mod_sec/corerule repository.)

---

In other news, there's now a basic msc_pyparser frontend in modseccfg.

Made it somewhat less featureful, and just packs a handful of examples after all.

[airween]

Hi @prof-milki,

so many thanks for report - and special thanks for using msc_pyparser :).

I'm afraid you ran into a problem which was already fixed in 609cd5.

Are you sure you use v1.1?

Now I checked it, and works for me:

$ cat issue15.conf 
SecRule TX:sampling_rnd100 "!@lt %{tx.sampling_percentage}" \
    "id:901450,\
    phase:1,\
    pass,\
    log,\
    noauditlog,\
    msg:'Sampling: Disable the rule engine based on sampling_percentage \
%{TX.sampling_percentage} and random number %{TX.sampling_rnd100}.',\
    ctl:ruleEngine=Off"
$ ./test_parser_crs.py issue15.conf 
Config: issue15.conf

issue15.yml
$ ./test_writer.py issue15.yml 
Parsing CRS structure: issue15.yml
$ diff issue15_out.conf issue15.conf 
[EMPTY]

I think PyPI version also contains this fix.

Please check your version and let me know!

[prof-milki]

Hey @airween

Right! Still got version 1.0 installed indeed.

I sorta ran into trouble with the PyPI package though. msc_pyparser-1.1-py3.8.egg seems to be an Egg bound to Python 3.8, whereas 1.0 was a wheel for any >=3.0. Could you do a bdist_wheel upload to see if that results in the usual package?

(And I should properly add a dependency on >= 1.1 then.)

[airween]

hi,

I sorta ran into trouble with the PyPI package though. msc_pyparser-1.1-py3.8.egg seems to be an Egg bound to Python 3.8, whereas 1.0 was a wheel for any >=3.0.

indeed. But I have no idea why is that bounded for 3.8 - PyPI page shows the requested Python version is >= 3, and my config doesn't contains any dependencies for 3.8. I have to investigate this.

Could you do a bdist_wheel upload to see if that results in the usual package?

(And I should properly add a dependency on >= 1.1 then.)

As I know PyPI doesn't allow overwriting the package with same version. I just can upload (when I'll find the issue) with 1.1.1, or 1.1a... or similar.

[prof-milki]

Ah, it's probably not the tag in the package name. (Just did a bit of googling, the markers only seem to be honored for wheels. And also, I already was on Python 3.8.x. So, there's that.)

It seems to be a pip issue (tried 20.1 and current 21.0). And seemingly egg support isn't there anymore (if it ever was):
> pip3 install -v msc_pyparser==1.1

…
Fetching project page and analyzing links: https://pypi.org/simple/msc-pyparser/
…
  Skipping link: unsupported archive format: .egg: https://files.pythonhosted.org/packages/9a/d3/5134677b4e31f356f87a2c0f77c753a861c1f8cd45498a8305004e7f3d5d/msc_pyparser-1.1-py3.8.egg#sha256=c1b2b5c69355f28622ce382bdfe96d38fc5b37863c50f3f6bde33d1d465aaa7c (from https://pypi.org/simple/msc-pyparser/) (requires-python:>=3)
Given no hashes to check 0 links for project 'msc-pyparser': discarding no candidates
…
ERROR: Could not find a version that satisfies the requirement msc_pyparser==1.1 (from versions: 0.2, 0.3, 1.0)

That being said, it works with a vcs install:
pip3 install git+https://github.com/digitalwave/msc_pyparser/
(Which obviously solved my original hiccup.)

But yes, if you could do a patch release (1.1-2 or whatever), that still might make sense for others. (PyPI blocking versions is somewhat questionable indeed.) Thanks!

status: closed

[airween]

Version 1.1.1 has been uploaded to PyPI, so I'm going to close this issue - thanks again for your notes.

This uploaded package contains the new dependency version for PyYAML.