Open ornj opened 7 months ago
Now supports @xynicole's changes to enable tokenizing Lua.
Benchmarks:
❯ go test -benchmem -run=^$ -bench "^(BenchmarkLex|BenchmarkLexWithLua|BenchmarkScanner|BenchmarkScannerWithLua)$" github.com/nginxinc/nginx-go-crossplane -count=1
goos: darwin
goarch: arm64
pkg: github.com/nginxinc/nginx-go-crossplane
BenchmarkLex/simple-10 57963 17756 ns/op 103049 B/op 39 allocs/op
BenchmarkLex/with-comments-10 60025 20067 ns/op 103112 B/op 45 allocs/op
BenchmarkLex/messy-10 26170 47822 ns/op 104400 B/op 168 allocs/op
BenchmarkLex/quote-behavior-10 74510 17693 ns/op 102961 B/op 26 allocs/op
BenchmarkLex/quoted-right-brace-10 43134 27752 ns/op 103560 B/op 54 allocs/op
BenchmarkLex/comments-between-args-10 78271 14866 ns/op 102937 B/op 27 allocs/op
BenchmarkLexWithLua/lua-basic-10 46273 26012 ns/op 105499 B/op 53 allocs/op
BenchmarkLexWithLua/lua-block-simple-10 22514 54149 ns/op 108556 B/op 143 allocs/op
BenchmarkLexWithLua/lua-block-larger-10 25983 46605 ns/op 108403 B/op 59 allocs/op
BenchmarkLexWithLua/lua-block-tricky-10 33756 35067 ns/op 106684 B/op 66 allocs/op
BenchmarkScanner/simple-10 163138 7084 ns/op 4648 B/op 36 allocs/op
BenchmarkScanner/with-comments-10 144558 8100 ns/op 4712 B/op 42 allocs/op
BenchmarkScanner/messy-10 47570 25026 ns/op 6000 B/op 165 allocs/op
BenchmarkScanner/quote-behavior-10 222280 5083 ns/op 4560 B/op 23 allocs/op
BenchmarkScanner/quoted-right-brace-10 82656 14281 ns/op 5160 B/op 51 allocs/op
BenchmarkScanner/comments-between-args-10 225475 4872 ns/op 4536 B/op 24 allocs/op
BenchmarkScannerWithLua/lua-basic-10 93081 12833 ns/op 7866 B/op 66 allocs/op
BenchmarkScannerWithLua/lua-block-simple-10 31426 37989 ns/op 10924 B/op 156 allocs/op
BenchmarkScannerWithLua/lua-block-larger-10 37148 30723 ns/op 10770 B/op 72 allocs/op
BenchmarkScannerWithLua/lua-block-tricky-10 54890 22383 ns/op 9050 B/op 79 allocs/op
PASS
ok github.com/nginxinc/nginx-go-crossplane 29.969s
Proposed changes
Implemented
crossplane.Scanner
that follows the example of other "scanner" types implemented in the Go stdlib. The existingLex
uses concurrency to make tokens available to the caller while managing "state". I think this design queue was taken from Rob Pike's 2011 talk on Lexical Scanning in Go. If you look at examples from the Go stdlib-- such asbufio.Scanner
thatLex
depends on-- you'd find that this isn't the strategy being employed and instead there is a struct that manages the state of the scanner and a method that used by the caller to advance the scanner to obtain tokens.After a bit of Internet archeology, I found this post on
golang-nuts
from Rob Pike himself:An implementation of a "scanner" using the more "traditional" model-- much of the logic is the same or very close to
Lex
-- seems to support the above statement.This alternative to
Lex
is probably a micro-optimization for many use cases. As the size and number of NGINX configurations that need to be analyzed grows, optimization can be a good thing as well as an API that feels familiar to Go developers who might use this tool for their own purposes.Next steps:
Scanner
to "parse" NGINX configurations. I think this should be done in place so that the existing API works as is, but we should also expose a way to allow the caller to provide the scanner.Lex
in favor ofScanner
. If we leaveLex
in place then I don't think we would need av2
of the crossplane package (yet).Checklist
Before creating a PR, run through this checklist and mark each as complete.
CONTRIBUTING
documentREADME.md
)