update: Refactoring Code-class and new Custom pygments based formatter

OliverStrait commented 3 months ago

Overview: What does this pull request change?

Old Code parser is very convoluted and hard to understand code of custom python parser with poor naming, and jumpping and overlapping procedure. It used pygments HtmlFormatter as backend and after parsed html to list[tuple[str,str]] structure which is used for coloring.
New custom formatter skips html and transforms code directly to list-mapping using pygments lexers and tokenizer functions.
Better naming and capsulating responsibility to a logical chuncks.
New system is just about 2 to 3 times faster, but Mobject constructors are still performance bottlenecks

code_class-new_perf

If Paragraph could support color to text mapping we could remove whole block of re-coloring code after object-construction (little performance boost).
Tested with pygments 2.18.0 and python language There may be some odd cases when pygments Tokenizer sneak newlines to unexpected places. Testes with different syntax and languages would be good to perform. If uncatched newlines is passed to Paragraph inside of string literals it will broke re-coloring an line-numbers.

[ ] The PR title is descriptive enough for the changelog, and the PR is labeled correctly
[ ] If applicable: newly added non-private functions and classes have a docstring including a short summary and a PARAMETERS section
[ ] If applicable: newly added functions and classes are tested

JasonGrace2282 commented 3 months ago

This seems interesting - Code has been in need of a rework for a while. Hopefully, I'll get the chance to look at this in a bit :)

OliverStrait commented 2 months ago

I made parser more general to handle odd cases where tokenizer produce string literals with newlines hidden inside.
Somehow Paragraph class cannot handle empty newlines at end of string (therefore line numbers will missalign) so I added parser to remove those.
(also fixed messed up git history)