nesaro / pydsl

Python Domain Specific Language Tools
http://nestorarocha.com/pydsl/
GNU General Public License v3.0
83 stars 11 forks source link

Using with python grammar on string (not file) #57

Closed ibizaman closed 11 years ago

ibizaman commented 11 years ago

Hi,

I'd like to use pydsl to validate a python file. I saw that there is a PythonGrammar object in pydsl.Grammar.Definition, is it working ?

I tried using it like this with no luck:

from sys import argv

from pydsl.Grammar.Definition import PythonGrammar
from pydsl.Validate import validate

def getCodeFromFile(source_file):
    with open(source_file, 'r') as f:
        return f.read()

if __name__ == '__main__':
    filename = argv[1]

    grammar = PythonGrammar()
    code = getCodeFromFile(filename)

    validate(grammar, code)

It gives me:

Traceback (most recent call last):
  File "validate.py", line 24, in <module>
    print(validate(grammar, code))
  File "...\pydsl-0.3.0-py3.3.egg\pydsl\Validate.py", line 58, in validate
  File "...\pydsl-0.3.0-py3.3.egg\pydsl\Validate.py", line 54, in validator_factory
ValueError: {}

Which indicates that the grammar is somewhat invalid.

I then also tried summary_python_file but with no luck either:

from sys import argv

from pydsl.File.Python import summary_python_file
from pydsl.Validate import validate

if __name__ == '__main__':
    filename = argv[1]

    print(summary_python_file(filename))

It gives me:

Traceback (most recent call last):
  File "validate.py", line 19, in <module>
    print(summary_python_file(filename))
  File "c:\Python33\lib\site-packages\pydsl-0.3.0-py3.3.egg\pydsl\File\Python.py", line 63, in summary_python_file
AttributeError: 'module' object has no attribute 'iclass'

I dived into the internals a little bit but I don't understand this iclass thing. And for the first error, I have no idea. Somebody knows what I'm doing wrong ?

Thanks, Ibiz

nesaro commented 11 years ago

Hi Ibiz,

I have written some documentation for the functions you've used in your scripts.

PythonGrammar is a class that defines a grammar using a Python function. There is an example in the contrib folder:

iclass="PythonGrammar"
def matchFun(inputstr):
    try:
        str(inputstr)
    except UnicodeDecodeError:
        return False
return True

that file is read by pydsl and converted into a PythonGrammar, which then can be used as any other grammar. the iclass line is useful for pydsl to know how to read the file. In the context of this example, it means "please convert me into a PythonGrammar instance"

The function _summary_pythonfile reads a pydsl file and produces a dictionary describing the file. Applying the function _summary_pythonfile to the previous example (pydsl/contrib/grammar/cstring.py) returns the following:

{'iclass': 'PythonGrammar', 'identifier': 'cstring', 'filepath': 'pydsl/contrib/grammar/cstring.py'}

About parsing python, pydsl includes 4 types of parsers:

  1. Backtracing Recusive Descent Parser
  2. Weighted Backtracing Recusive Descent Parser
  3. LR0 Parser
  4. LL1 Recursive Descent Parser

more info here

Unfortunately, none of them is powerful/well implemented enough to parse python grammar. I'm working on a better parser at the moment ( #54 ). But if you want to parse python grammar, I think the best option is to use the python parser via the ast module.

import ast
try:
    ast.dump(ast.parse('1c.2'))
except SyntaxError as e:
    print(e.offset, e.text)
....
(2, '1c.2\n')

more info about ast here and here

another option is to use a different parser implementation. plyplus is a good example and it includes a python parser!

Nestor

ibizaman commented 11 years ago

Hi Nestor,

Thank you very much for this thorough explanation and all those links. So I see I used those functions wrongly. I knew there were a catch :D

I'll then check the ast module and plyplus.

Nonetheless, I love the features implemented in pydsl. I hope you'll be able to make it go as far as it deserves it.

Ibiz