Closed bmschmidt closed 7 years ago
This would explain the issues we had trying to build the Sefaria bookworm.
maybe this is a good starting point? Each line in the downloaded quran.txt should be a document. http://tanzil.net/download
Thanks for the bump here. That Quran's license forbids alteration, so I'm disinclined to chop it up for a test suite.
But I've added a test suite with just a little Arabic and a little Cherokee, so hopefully we can't accidentally lose unicode support again.
There needs to be a basic test that includes unicode characters. I just realized that most things outside of ASCII were being silently dropped. Maybe a Hebrew bible, or something.