Current updates from Migamake

[dmalkr]

There is almost all our changes.

• First significant change is speed improvement in src/Xeno/SAX.hs. I find out that most used function is s_index: https://github.com/ocramz/xeno/blob/a5f942d823ca6a16adbb319c43f3b131c4c1d19d/src/Xeno/SAX.hs#L252-L257 This function check for boundary crossing of n. But code analyze show that n is always non-negative, so we can remove first check.
  • Then, s_index used to check current or next to current character and always used to compare with some expectable character, i.e. in mode s_index inputString i == someExpectableChar. So if we place meaninless char (for example, \NUL) at the end of string, we can remove second check (n >= S.length ps). Our benchmark shows speed increasing (40-150%) because of this check removing.
  • But how to add this zero delimiter? For long strings this will lead to copying and decreasing of speed. I try to use memory mapping beginning of original string to new place and copy only last memory page of string where I can place zero at the end of string. But this work in progress, I have not finished it yet.
  • So for now I assume that user of library adds a trailing zero. For example, user can prepare buffer with additional place to \NUL and read file to this buffer. For this I introduce StringLike class, and StringLike ByteString instance and special data type for zero-terminated strings ByteStringZeroTerminated and StringLike ByteStringZeroTerminated instance. process function now process StringLike instances.
  • Other speed improvements are questionable and require additional investigation with modern GHCs versions (for example bit hacks: https://github.com/ocramz/xeno/blob/dc02b6fc5320df6618e7d01987b0243760f64abb/src/Xeno/SAX.hs#L423-L444). So I can remove it from PR.
  • You can view benchmarks comparison here: benchmark.md: https://github.com/ocramz/xeno/blob/dc02b6fc5320df6618e7d01987b0243760f64abb/benchmark.md
• For more comfortable use of process function it was introduced new data type Process: https://github.com/ocramz/xeno/blob/dc02b6fc5320df6618e7d01987b0243760f64abb/src/Xeno/SAX.hs#L69-L77 . This change don't affect speed/memory according to benchmarks.
• DOM processing:
  • Speed improvements for src/Xeno/DOM.hs. Main improvement here is using adoptable internal array growing: https://github.com/ocramz/xeno/blob/dc02b6fc5320df6618e7d01987b0243760f64abb/src/Xeno/DOM.hs#L140-L150 . We try predict final size to grows internal vector only once. Other improvements is use unsafe vector operations, because we know vector boundaries.
  • Refactoring, some parts of Xeno/DOM.hs moved to Xeno/DOM/Internal.hs.
  • src/Xeno/DOM/Robust.hs variant of DOM for more robust DOM processing.
• Benchmarks improvements:
  • Remove code duplication
  • Added benchmarks for ByteStringZeroTerminated: benchmarks run with original ByteString (input) and with strings to wich \NUL appended (inputz).
  • Added benchmarks for big files (bench/SpeedBigFiles.hs). We use some files from open sources (dumps from MediaWiki project). They must be downloaded before run benchmarks using data/download-ex-data-for-benchmarks.sh script. It downloads data to data/ex directory.
  • Benchmarks run non-threaded. We decide that benchmarks in threaded RTS is benchmark not only XML processing, but XML processing and threadings. So we decide to run benchmarks in more clean environment (but according to our benchmarks, improvements are still exists in threaded environment too, but not so expressed).

[ocramz]

@dmalkr @mgajda thank you, this is very impressive! For the record, which GHC versions have you tested this with?

[dmalkr]

@dmalkr @mgajda thank you, this is very impressive! For the record, which GHC versions have you tested this with?

Mainly with GHC 8.6.5; in other versions (8.*) we just check that tests and benchmarks still working.

[ocramz]

Hi @dmalkr @mgajda , hope you're doing great! Shall we finish this PR and publish a new release before handover?

[ocramz]

OK, merging and cutting a new release.