Introduce `Token` and `Span`

[Hywan]

Work in Progress.

I would like to thank @fflorent for the original idea and code. The goal is to declare a Span structure that can be used as an input for nom parsers. Each Span collects the offset, line, and column numbers for each slices computed by parsers. Then, the Token structure is used to attached a span to any “data”.

It works. We are using bytecount::count to count the number of lines. And we are using memchr::memrchr to count the number of columns. All counting are made relatively to the previous span, so we are not iterating over the whole input each time. The offset is computed based on the nom::Offset features.

We can compile tagua-parser as follows:

$ cargo build --release --features simd

or:

$ cargo build --release --features avx

to respectively enable SIMD or AVX optimisations. bytecount performance depends on these features.

Next step: See how it feets with the AST.

[coveralls]

Coverage increased (+0.2%) to 94.18% when pulling a5ac1bfc807fccbec3c2f50153576b8585f293da on Hywan:span into 7b4b15ad6f58784aa4029bb40136553a819d5af5 on tagua-vm:master.

[coveralls]

Coverage increased (+0.2%) to 94.18% when pulling a5ac1bfc807fccbec3c2f50153576b8585f293da on Hywan:span into 7b4b15ad6f58784aa4029bb40136553a819d5af5 on tagua-vm:master.

[coveralls]

Coverage increased (+0.2%) to 94.18% when pulling a5ac1bfc807fccbec3c2f50153576b8585f293da on Hywan:span into 7b4b15ad6f58784aa4029bb40136553a819d5af5 on tagua-vm:master.

[coveralls]

Coverage increased (+0.2%) to 94.18% when pulling a5ac1bfc807fccbec3c2f50153576b8585f293da on Hywan:span into 7b4b15ad6f58784aa4029bb40136553a819d5af5 on tagua-vm:master.

[coveralls]

Coverage increased (+0.2%) to 94.18% when pulling a5ac1bfc807fccbec3c2f50153576b8585f293da on Hywan:span into 7b4b15ad6f58784aa4029bb40136553a819d5af5 on tagua-vm:master.

[Hywan]

POC is working. Now I need to update all the parsers…

[coveralls]

Coverage increased (+0.2%) to 94.18% when pulling a5ac1bfc807fccbec3c2f50153576b8585f293da on Hywan:span into 7b4b15ad6f58784aa4029bb40136553a819d5af5 on tagua-vm:master.

[coveralls]

Coverage increased (+0.3%) to 94.319% when pulling 9a72cfd72fd9d1992a1e381cc5d74ef82f46c361 on Hywan:span into 7b4b15ad6f58784aa4029bb40136553a819d5af5 on tagua-vm:master.

[coveralls]

Coverage increased (+0.3%) to 94.319% when pulling 9a72cfd72fd9d1992a1e381cc5d74ef82f46c361 on Hywan:span into 7b4b15ad6f58784aa4029bb40136553a819d5af5 on tagua-vm:master.

[coveralls]

Coverage increased (+0.3%) to 94.319% when pulling 9a72cfd72fd9d1992a1e381cc5d74ef82f46c361 on Hywan:span into 7b4b15ad6f58784aa4029bb40136553a819d5af5 on tagua-vm:master.

[coveralls]

Coverage increased (+0.3%) to 94.319% when pulling 9a72cfd72fd9d1992a1e381cc5d74ef82f46c361 on Hywan:span into 7b4b15ad6f58784aa4029bb40136553a819d5af5 on tagua-vm:master.

[coveralls]

Coverage increased (+0.3%) to 94.319% when pulling 9a72cfd72fd9d1992a1e381cc5d74ef82f46c361 on Hywan:span into 7b4b15ad6f58784aa4029bb40136553a819d5af5 on tagua-vm:master.

[coveralls]

Coverage increased (+0.3%) to 94.319% when pulling 9a72cfd72fd9d1992a1e381cc5d74ef82f46c361 on Hywan:span into 7b4b15ad6f58784aa4029bb40136553a819d5af5 on tagua-vm:master.

[Hywan]

All parsers have been migrated, along with the 387 unit tests (calculating all span positions by hand was a very satisfying experience…)!

Next step: Update the documentation!

[coveralls]

Coverage increased (+2.6%) to 96.588% when pulling ff0ca1f618fcce0fc7453d893af0f1dc55f93009 on Hywan:span into 7b4b15ad6f58784aa4029bb40136553a819d5af5 on tagua-vm:master.

[coveralls]

Coverage increased (+2.6%) to 96.588% when pulling ff0ca1f618fcce0fc7453d893af0f1dc55f93009 on Hywan:span into 7b4b15ad6f58784aa4029bb40136553a819d5af5 on tagua-vm:master.