Added the ability to read variables from a comment

[vadim-isakov]

Thank you for the wonderful library.

Files can contain variables as comments. I propose a function for reading variables.

[EmilStenstrom]

Hi! Thanks for making a pull request!

I have a couple of questions before deciding on this:

1. I don’t see any variables in the example you added to README.md. Did you add the wrong example?
2. Is there a specification somewhere of how variables work? I’d like to check that we don’t miss any corner cases.

[EmilStenstrom]

Ah, I see the second commit now. That answers the first question. Do you have an answer for the second?

[vadim-isakov]

Hi! There is a specification on the site http://universaldependencies.org/format.html in the following paragraphs: "Untokenized Text", "Sentence Boundaries and Comments", "Paragraph and Document Boundaries".

Important points:

• Lines starting with the # character and preceding a sentence are considered as carrying comments or metadata relevant to the following sentence.
• From v2 the following two comments are compulsory for every sentence: 'sent_id' and 'text' (and there must be just one comment of each kind per sentence).
• Not specified exactly how the comment should look with the variable (medatada). In the examples, this structure "[#][space]KEY[space][=][space]VALUE[new line]" ("# sent_id = 1").
• There is some specification for 'sent_id' - "identifier does not contain whitespace characters (while the comment line may contain whitespace around the sent_id keyword and the equals-to sign)".

It seems as comments can be without a variable so I made the new commit where program works, if there are comments without variables (metadata) but we don't extract them.

P.S. I often find conllu files with different variables.

Update: Made a new commit in which I replaced word 'variables' to 'metadata' it is so mentioned in the specification and corrected the example of the answer in the README for clarity.

[EmilStenstrom]

Thanks for the references! I do want this in the library. I’m a little sure about adding this as a separate entry point, instead of it being part of the parse and parse_tree functions. I’m currently away on vacation, but I’ll have a look at this in a couple of days when I’m back again.

[EmilStenstrom]

Hi! After thinking about this code a bit more, I don't like the fact that there is now two different entry points, parse and parse_with_comments. I'd like parse to include comments as well. But the list returned by parse isn't easily extended to include comments without breaking backwards compatibility, so that's the catch.

I've played around with the code in a separate branch a bit, and I think I have an idea, but I'll do that separately.

So, the plan is to include this PR first, and then separately work on a nicer API.

To get this merged, I wanted the tests in Travis to pass first. That required proper formatting according to flake8, so I fixed that, and 100% test coverage, so I added that too. As far as I'm concerned, this is ready to be merged.

[EmilStenstrom]

Conllu 0.11 is now released to PyPi: https://pypi.org/project/conllu/

[EmilStenstrom]

Just a heads up. I just released conllu 1.0, which implements the changes I described above. Now comments are always parsed when using the normal parse function, and you can get them by accessing the .metadata attribute on the returned TokenList or TokenTree. Hope I didn't mess too much with your current code :)