Report comments via AST

[rkaminsk]

• [x] add comment ASTs to C API
• [x] add comment ASTs to C++ API + tests
• [x] add comment ASTs to Python API + tests
• [x] it is unnecessary to extract comments in clingo mode
• [x] do not print newline in astv2_str
• [x] ensure proper order of comment ASTs (addressing the issue below)
• think about option to ignore comment ASTs
  • To add optional comment parsing, I would have to break the API. Most applications are written to handle a known set of AST types. Thus, adding a new AST type should not break many applications. Furthermore, passing comments should not incur much overhead. So it seems a reasonable approach to unconditionally report comment ASTs.

The snippet below shows some issues that should still (or not) be addressed before merging:

>>> from clingo.ast import parse_string
>>> 
>>> PRG = """
... %* This program shows some issues that appear
...  * when directives with optional attributes are used.
...  * Changing this would require storing comments in the lexer.
...  *%
... #const x=10.
... % comment appears unexpectedly before `x=10`
... a.
... % comment appears before `y=10` as expected
... #const y=10. [override]
... b.
... #external a.
... % comment appears unexpectedly before `#external a`
... a.
... % comment appears before `#external b` as expected
... #external b. [true]
... """
>>> 
>>> parse_string(PRG, print)
#program base.
%* This program shows some issues that appear
 * when directives with optional attributes are used.
 * Changing this would require storing comments in the lexer.
 *%
% comment appears unexpectedly before `x=10`
#const x = 10.
a.
% comment appears before `y=10` as expected
#const y = 10. [override]
b.
% comment appears unexpectedly before `#external a`
#external a. [false]
a.
% comment appears before `#external b` as expected
#external b. [true]

[rkaminsk]

@teiesti, could you have a look if this suits your application?

[teiesti]

I am going to have a look.

[teiesti]

I was able to make the following work on my machine which looks very promising.

>>> parse_string("%comment\nfact.", lambda ast: print(repr(ast)))
ast.Program(Location(begin=Position(filename='<string>', line=1, column=1), end=Position(filename='<string>', line=1, column=1)), 'base', [])
ast.Comment(Location(begin=Position(filename='<string>', line=1, column=1), end=Position(filename='<string>', line=1, column=9)), '%comment', 0)
ast.Rule(Location(begin=Position(filename='<string>', line=2, column=1), end=Position(filename='<string>', line=2, column=6)), ast.Literal(Location(begin=Position(filename='<string>', line=2, column=1), end=Position(filename='<string>', line=2, column=5)), 0, ast.SymbolicAtom(ast.Function(Location(begin=Position(filename='<string>', line=2, column=1), end=Position(filename='<string>', line=2, column=5)), 'fact', [], 0))), [])

I can't wait to see this being stabilized. Thanks for the effort!

Anyhow, I would like to see the pdoc documentation. Is there an easy way to generate it?

@susuhahnml You may also want to look through this!

[rkaminsk]

I was able to make the following work on my machine which looks very promising.

>>> parse_string("%comment\nfact.", lambda ast: print(repr(ast)))
ast.Program(Location(begin=Position(filename='<string>', line=1, column=1), end=Position(filename='<string>', line=1, column=1)), 'base', [])
ast.Comment(Location(begin=Position(filename='<string>', line=1, column=1), end=Position(filename='<string>', line=1, column=9)), '%comment', 0)
ast.Rule(Location(begin=Position(filename='<string>', line=2, column=1), end=Position(filename='<string>', line=2, column=6)), ast.Literal(Location(begin=Position(filename='<string>', line=2, column=1), end=Position(filename='<string>', line=2, column=5)), 0, ast.SymbolicAtom(ast.Function(Location(begin=Position(filename='<string>', line=2, column=1), end=Position(filename='<string>', line=2, column=5)), 'fact', [], 0))), [])

I can't wait to see this being stabilized. Thanks for the effort!

Anyhow, I would like to see the pdoc documentation. Is there an easy way to generate it?

https://pdoc3.github.io/pdoc/doc/pdoc/#command-line-interface

There is not much to document though. It is just one additional type added to the AST.

[rkaminsk]

@teiesti, @susuhahnml, this should be ready. You can already try using the development packages on ubuntu/anaconda/pypi test. -R