error when parsing mako template

[tisdall]

I've tried parsing my mako files with the mako extractor and I get:

Aborting due to parse error in ./mainserver/templates/menu.mako[24]: else: pass

That seems to be coming from the lingua/extractors/python.py as that's the only one giving a line number. I have no idea where the "else: pass" is coming from as here's that section of that template (line 24 is ${name}\):

% if text is not None:
${text}\
% else:
${name}\
% endif

Is this extractor going over the compiled version of the mako template or is somehow using the wrong extractor? My config is:

[extensions]
.mako = mako
.pt = chameleon
.py = python

[extractor:babel-mako]
comment-tags = TRANSLATORS:

I tried with the babel-mako and I get:

$ env/bin/pot-create -c lingua.cfg -d mainserver -o mainserver/locale/mainserver.pot mainserver
Traceback (most recent call last):
  File "/sites/metrics_dev/env/lib/python3.4/site-packages/mako/lexer.py", line 201, in decode_raw_stream
    text = text.decode(parsed_encoding)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 3891: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "env/bin/pot-create", line 11, in <module>
    sys.exit(main())
  File "/sites/metrics_dev/env/lib/python3.4/site-packages/lingua/extract.py", line 262, in main
    for message in extractor(real_filename, options):
  File "/sites/metrics_dev/env/lib/python3.4/site-packages/lingua/extractors/babel.py", line 35, in __call__
    comment_tags, self.config):
  File "/sites/metrics_dev/env/lib/python3.4/site-packages/mako/ext/babelplugin.py", line 48, in extract
    for message in extractor(fileobj):
  File "/sites/metrics_dev/env/lib/python3.4/site-packages/mako/ext/extract.py", line 11, in process_file
    input_encoding=self.config['encoding']).parse()
  File "/sites/metrics_dev/env/lib/python3.4/site-packages/mako/lexer.py", line 215, in parse
    self.filename,)
  File "/sites/metrics_dev/env/lib/python3.4/site-packages/mako/lexer.py", line 207, in decode_raw_stream
    0, 0, filename)
mako.exceptions.CompileException: Unicode decode operation of encoding 'ascii' failed at line: 0 char: 0

[tisdall]

Ah.. I think I found the code at https://bitbucket.org/zzzeek/mako/src/f07e53dc08dcd8ed4a991539185c3e983413de06/mako/ext/linguaplugin.py?at=master

Line 29 is what's tossing that " pass" on there. And that code seems to be what's using the python extractor.

[tisdall]

After a bunch of playing around, it seems that the "mako" extractor fails every time it run across a % else:. If I remove the one it's erring on it just fails on the next one it finds. It also always reports the line number directly after the % else:.

[wichert]

That definitely sounds like a bug in the mako extractor. I am wondering if this pull request fixes your bug. Is it possible for you to try that?

[tisdall]

It seems to be this issue: https://bitbucket.org/zzzeek/mako/pull-request/18/continuation-partial-statements-else-elif/diff

[tisdall]

^_^ yeah.. that's it.

[ldaverio]

I think it's exactly the same problem :)

NB: lingua configuration file is not required to make extraction work :)

[tisdall]

I get this traceback:

Traceback (most recent call last):
  File "env/bin/pot-create", line 11, in <module>
    sys.exit(main())
  File "/sites/metrics_dev/env/lib/python3.4/site-packages/lingua/extract.py", line 262, in main
    for message in extractor(real_filename, options):
  File "/sites/metrics_dev/env/lib/python3.4/site-packages/mako/ext/extract.py", line 12, in process_file
    for extracted in self.extract_nodes(template_node.get_children()):
  File "/sites/metrics_dev/env/lib/python3.4/site-packages/mako/ext/extract.py", line 84, in extract_nodes
    code, node.lineno, translator_strings):
  File "/sites/metrics_dev/env/lib/python3.4/site-packages/mako/ext/linguaplugin.py", line 29, in process_python
    if source in ('try:', 'else:') or source.startswith('except'):
TypeError: startswith first arg must be bytes or a tuple of bytes, not str

[ldaverio]

@tisdall, sorry, I'm still not fully used to Python3. Give me a few minutes, and I'll fix it. Maybe just a matter of writing source.startswith(b'except')?

[tisdall]

I got it to finish with this:

        if source.endswith(compat.b(':')):
            if source in (compat.b('try:'), compat.b('else:')) or source.startswith(compat.b('except')):
                source = compat.b('') # Ignore try/except and else
            elif source.startswith(compat.b('elif')):
                source = source[2:] # Replace "elif" with "if"
            source += compat.b('pass')
            code = io.BytesIO(source)

[tisdall]

Looking at the compat.py file, this seems to be the correct way of handling both py2 and py3. compat.b does nothing in py2 and in py3 it converts it to a byte.

[tisdall]

@ldaverio if I don't use a .cfg file it seems to default to the mako extractor. I was using the .cfg to try using the babel-mako extractor but that didn't seem to work either. The error I get with it doesn't even specify what file it's looking at so I have no idea where to look for a solution there... However, now that I have the mako one working, I guess it doesn't matter too much to me.

[ldaverio]

@tisdall, for my last commit, I seem to have run the unit tests in a virtualenv where lingua was not installed. So, the tests were skipped and the error didn't show up. Oops :(

[tisdall]

@ldaverio eh.. whatever.. it fixed things, just had issues with py3k compatibility.

[wichert]

Since the issue and fix are on the Mako side of things, can we close this ticket?

[ldaverio]

Fine for me, Wichert. Thanks for crediting me :)

[tisdall]

woot!