analyser using generators #11

[danilobellini]

• Solves issue #11 with generators
• Raises an UnknownToken instance with an useful error message, when it is the case (new class)
• String processing had some severe optimization (since strings are immutable) with a new line-based generator approach (new code_line_generator function)
• Both lexical and analyse now accepts both string (str or unicode) and file (StringIO, BytesIO or TextIOWrapper from io.open) as "code" input
• Tokenizing now is lazy (although syntax still doesn't use that for any optimization)

$ coverage run --branch setup.py test
  [...]
----------------------------------------------------------------------
Ran 342 tests in 18.430s

OK (skipped=1)

$ coverage report pynes/analyzer.py
Name             Stmts   Miss Branch BrMiss  Cover
--------------------------------------------------
pynes/analyzer      37      0     18      0   100%

[ellisonleao]

:+1:

[gutomaia]

@danilobellini could you send the pull request to the 0.1.x branch. Even better if you put it just set for the merge!

Thanks

[coveralls]

Coverage remained the same when pulling 37a464f402a1384afa176aa0d9a234667c6f97d7 on danilobellini:analyser into a43a3f2a3a5550f2978fa55bf08fa8c5d42e719b on gutomaia:master.

[danilobellini]

It seems that GitHub doesn't allow changing the branch of a pull-request [anymore]. So I can't change this pull-request to point to the 0.1.x branch. That would be another pull request. I think a sync without any commit like this would be enough (after merging with master):

git checkout 0.1.x
git merge master
git push origin 0.1.x

This would apply this pull-request to both master and 0.1.x branches. If you don't want to apply these changes to master, any collaborator can just merge this pull request directly to the 0.1.x branch:

git remote add danilo https://github.com/danilobellini/pyNES.git
git checkout -b analyser
git pull danilo analyser
git checkout 0.1.x
git merge analyser
git push origin 0.1.x

That's just to avoid opening a new issue for the same idea/contents/data.

That merge was simple, I spent more time to find why a commit you did 8 months ago conflicted with this pull request from a fork recently done hehehe...

[gutomaia]

No problem! I do it :+1: