search

geographika / mappyfile

A pure Python MapFile parser for working with MapServer
MIT License
71 stars 20 forks source link

master / python 3.7.2 final tests errors and (too much warnings) #72

Closed tigerfoot closed 5 years ago

tigerfoot commented 5 years ago

Trying to run tests (excluding docs which failed issue #71 ) Goes to have one failure + tons of warnings I guess some new way of doing things in 3.7 and perhaps related to the presence of new lark_parser 0.6.6

py.test3 --ignore=tests/mapfiles --cov mappyfile --cov-report term-missing misc/ tests/

======================================================================== test session starts ========================================================================
platform linux -- Python 3.7.2, pytest-3.10.1, py-1.7.0, pluggy-0.8.0
rootdir: /tmp/mappyfile, inifile:
plugins: mock-1.10.0, cov-2.6.0, attrib-0.1.3
collected 201 items                                                                                                                                                 

misc/test_docs_parser.py ....x...                                                                                                                             [  3%]
tests/test_cli.py .                                                                                                                                           [  4%]
tests/test_comments.py ..........                                                                                                                             [  9%]
tests/test_errors.py ..Fx                                                                                                                                     [ 11%]
tests/test_expressions.py .................x......                                                                                                            [ 23%]
tests/test_includes.py ........                                                                                                                               [ 27%]
tests/test_map_collection.py .                                                                                                                                [ 27%]
tests/test_ordereddict.py .......                                                                                                                             [ 31%]
tests/test_pprint.py .......................                                                                                                                  [ 42%]
tests/test_sample_maps.py .......                                                                                                                             [ 46%]
tests/test_snippets.py ........................................x...............                                                                               [ 74%]
tests/test_symbolset.py ..                                                                                                                                    [ 75%]
tests/test_transformer.py ..............x...                                                                                                                  [ 84%]
tests/test_utils.py ................                                                                                                                          [ 92%]
tests/test_validation.py ................                                                                                                                     [100%]

============================================================================= FAILURES ==============================================================================
_________________________________________________________________________ test_missing_end __________________________________________________________________________

token = Token($END, '')

    def get_action(token):
        state = state_stack[-1]
        try:
>           return states[state][token.type]
E           KeyError: '$END'

/usr/lib/python3.7/site-packages/lark/parsers/lalr_parser.py:46: KeyError

During handling of the above exception, another exception occurred:

    def test_missing_end():
        """
        Check an invalid keyword throws a schema validation
        error
        """
        s = """MAP
    LAYER
    NAME "Test"
    LAYER
    NAME "Test2"
    END
    END"""

        p = Parser()
        try:
>           p.parse(s)

tests/test_errors.py:79: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <mappyfile.parser.Parser object at 0x7f3e4b87f710>, text = 'MAP\nLAYER\nNAME "Test"\nLAYER\nNAME "Test2"\nEND\nEND', fn = None

    def parse(self, text, fn=None):
        """
        Parse the Mapfile
        """

        text = str(text)
        if self.expand_includes:
            text = self.load_includes(text, fn=fn)

        try:
            self._comments[:] = []  # clear any comments from a previous parse
>           tree = self.lalr.parse(text)

mappyfile/parser.py:195: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = Lark(open('<string>'), parser='lalr', lexer='contextual', ...), text = 'MAP\nLAYER\nNAME "Test"\nLAYER\nNAME "Test2"\nEND\nEND'

    def parse(self, text):
        "Parse the given text, according to the options provided. Returns a tree, unless specified otherwise."
>       return self.parser.parse(text)

/usr/lib/python3.7/site-packages/lark/lark.py:228: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <lark.parser_frontends.LALR_ContextualLexer object at 0x7f3e4b87dda0>, text = 'MAP\nLAYER\nNAME "Test"\nLAYER\nNAME "Test2"\nEND\nEND'

    def parse(self, text):
        token_stream = self.lex(text)
        sps = self.lexer.set_parser_state
>       return self.parser.parse(token_stream, *[sps] if sps is not NotImplemented else [])

/usr/lib/python3.7/site-packages/lark/parser_frontends.py:38: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <lark.parsers.lalr_parser._Parser object at 0x7f3e4b628358>, seq = <generator object ContextualLexer.lex at 0x7f3e4b7230c0>
set_state = <bound method ContextualLexer.set_parser_state of <lark.lexer.ContextualLexer object at 0x7f3e4b887860>>

    def parse(self, seq, set_state=None):
        token = None
        stream = iter(seq)
        states = self.states

        state_stack = [self.start_state]
        value_stack = []

        if set_state: set_state(self.start_state)

        def get_action(token):
            state = state_stack[-1]
            try:
                return states[state][token.type]
            except KeyError:
                expected = [s for s in states[state].keys() if s.isupper()]
                raise UnexpectedToken(token, expected, state=state)

        def reduce(rule):
            size = len(rule.expansion)
            if size:
                s = value_stack[-size:]
                del state_stack[-size:]
                del value_stack[-size:]
            else:
                s = []

            value = self.callbacks[rule](s)

            _action, new_state = states[state_stack[-1]][rule.origin.name]
            assert _action is Shift
            state_stack.append(new_state)
            value_stack.append(value)

        # Main LALR-parser loop
        for token in stream:
            while True:
                action, arg = get_action(token)
                assert arg != self.end_state

                if action is Shift:
                    state_stack.append(arg)
                    value_stack.append(token)
                    if set_state: set_state(arg)
                    break # next token
                else:
                    reduce(arg)

        token = Token.new_borrow_pos('$END', '', token) if token else Token('$END', '', 0, 1, 1)
        while True:
>           _action, arg = get_action(token)

/usr/lib/python3.7/site-packages/lark/parsers/lalr_parser.py:83: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

token = Token($END, '')

    def get_action(token):
        state = state_stack[-1]
        try:
            return states[state][token.type]
        except KeyError:
            expected = [s for s in states[state].keys() if s.isupper()]
>           raise UnexpectedToken(token, expected, state=state)
E           lark.exceptions.UnexpectedToken: Unexpected token Token($END, '') at line 7, column 1.
E           Expected one of: 
E               * UNQUOTED_STRING
E               * STYLE
E               * QUERYMAP
E               * _END
E               * LAYER
E               * MAP
E               * CONFIG
E               * POINTS
E               * LEADER
E               * FEATURE
E               * LEGEND
E               * COMPOSITE
E               * LABEL
E               * CLUSTER
E               * VALIDATION
E               * PROJECTION
E               * CLASS
E               * METADATA
E               * REFERENCE
E               * WEB
E               * OUTPUTFORMAT
E               * SCALETOKEN
E               * GRID
E               * JOIN
E               * PATTERN
E               * VALUES
E               * SYMBOL
E               * SCALEBAR

/usr/lib/python3.7/site-packages/lark/parsers/lalr_parser.py:49: UnexpectedToken

During handling of the above exception, another exception occurred:

    def test_missing_end():
        """
        Check an invalid keyword throws a schema validation
        error
        """
        s = """MAP
    LAYER
    NAME "Test"
    LAYER
    NAME "Test2"
    END
    END"""

        p = Parser()
        try:
            p.parse(s)
        except UnexpectedToken as ex:
            print(ex.__dict__)
            assert(ex.line == 7)
            assert(ex.column == 1)
>           assert(str(ex.token) == 'END')
E           AssertionError: assert '' == 'END'
E             + END

tests/test_errors.py:84: AssertionError
----------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------
{'token': Token($END, ''), 'expected': ['UNQUOTED_STRING', 'STYLE', 'QUERYMAP', '_END', 'LAYER', 'MAP', 'CONFIG', 'POINTS', 'LEADER', 'FEATURE', 'LEGEND', 'COMPOSITE', 'LABEL', 'CLUSTER', 'VALIDATION', 'PROJECTION', 'CLASS', 'METADATA', 'REFERENCE', 'WEB', 'OUTPUTFORMAT', 'SCALETOKEN', 'GRID', 'JOIN', 'PATTERN', 'VALUES', 'SYMBOL', 'SCALEBAR'], 'line': 7, 'column': 1, 'considered_rules': None, 'state': 51, 'pos_in_stream': 45}
------------------------------------------------------------------------- Captured log call -------------------------------------------------------------------------
parser.py                  203 ERROR    Parsing of Mapfile unsuccessful

----------- coverage: platform linux, python 3.7.2-final-0 -----------
Name                       Stmts   Miss  Cover   Missing
--------------------------------------------------------
mappyfile/__init__.py         19      5    74%   19, 25-28
mappyfile/cli.py              63     32    49%   28-29, 45-48, 79-85, 110-135
mappyfile/linter.py            2      2     0%   11-12
mappyfile/ordereddict.py      66     12    82%   8, 20, 31-32, 36, 42, 48, 51, 54, 87, 90, 96
mappyfile/parser.py          135      9    93%   10, 69-71, 168, 175-178
mappyfile/pprint.py          298     11    96%   13, 35, 65, 68, 133-137, 362, 365
mappyfile/tokens.py            4      0   100%
mappyfile/transformer.py     416     34    92%   21, 44-47, 60-65, 76-82, 132, 196, 237-243, 301, 309, 441-443, 497, 500, 525, 528-529, 567-568
mappyfile/utils.py            95     15    84%   23-31, 246-248, 536, 558, 574-575, 579-580
mappyfile/validator.py       108      7    94%   14, 52, 67-69, 114-115
--------------------------------------------------------
TOTAL                       1206    127    89%

========================================================================= warnings summary ==========================================================================
mappyfile/ordereddict.py:1
  /tmp/mappyfile/mappyfile/ordereddict.py:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import OrderedDict, Callable

/usr/lib/python3.7/site-packages/jsonschema/compat.py:6
  /usr/lib/python3.7/site-packages/jsonschema/compat.py:6: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import MutableMapping, Sequence  # noqa
  /usr/lib/python3.7/site-packages/jsonschema/compat.py:6: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import MutableMapping, Sequence  # noqa

tests/test_comments.py::test_header_comment2
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))
  /usr/lib/python3.7/site-packages/lark/lexer.py:217: DeprecationWarning: Flags not at the start of the expression '(?P<CCOMMENT>(?s)\\/[' (truncated)
    mre = re.compile(u'|'.join(u'(?P<%s>%s)'%(t.name, t.pattern.to_regexp()+postfix) for t in terminals[:max_size]))

tests/test_map_collection.py::test_maps
  /usr/lib/python3.7/site-packages/glob2/fnmatch.py:80: DeprecationWarning: Flags not at the start of the expression '(.*)\\Z(?ms)'
    return re.compile(res, flags).match
  /usr/lib/python3.7/site-packages/glob2/fnmatch.py:80: DeprecationWarning: Flags not at the start of the expression '(.*)\\.map\\Z(?ms)'
    return re.compile(res, flags).match

-- Docs: https://docs.pytest.org/en/latest/warnings.html
================================================== 1 failed, 195 passed, 5 xfailed, 35 warnings in 157.00 seconds ===================================================
geographika commented 5 years ago

@tigerfoot - looks like lots of DeprecationWarnings were added in Python 3.7.2. I think I've fixed all the ones in mappyfile, but there are a few upstream in jsonschema and glob2. Looks like they should be fixed in future releases - https://github.com/Julian/jsonschema/issues/487

geographika commented 5 years ago

0.8.x release updated to work with latest versions of Lark, and warnings fixed.