Support unicode tokens in standardizing line endings

tempestphp / highlight

🎨 Fast, extensible, server-side code highlighting for web and terminal

https://tempest.stitcher.io/highlight/01-getting-started

MIT License

640 stars 38 forks source link

Support unicode tokens in standardizing line endings #143

Closed nimah79 closed 2 months ago

nimah79 commented 2 months ago

This PR fixes rendering codes that contain UTF-8 tokens when using gutter or Ellison. For example, the following code is not currently being rendered correctly:

import نام کتابخانه

The highlighted output is currently something like this:

This PR uses the regular expression u flag to support Unicode characters. The result is now correct:

coveralls commented 2 months ago

Pull Request Test Coverage Report for Build 10555877509

Details

2 of 2 (100.0%) changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 92.977%

Totals
Change from base Build 10387601521:	0.0%
Covered Lines:	1549
Relevant Lines:	1666

💛 - Coveralls

brendt commented 2 months ago

Cool, thanks!

brendt commented 2 months ago

Tagged: https://github.com/tempestphp/highlight/releases/tag/2.10.0