subfinder / research

❄️ Research project for SubFinder core API V2
MIT License
35 stars 8 forks source link

Add crtsh source #37

Closed picatz closed 6 years ago

picatz commented 6 years ago

This PR aims to fix https://github.com/subfinder/research/issues/17

picatz commented 6 years ago

Note: I chose to implement a byte scanner that looks for the } character as the end of a JSON object. This requires the use of a secondary json buffer. This is because this source doesn't properly delimit their JSON objects with , characters.

picatz commented 6 years ago
$ go test -run=Crt -v -bench=.
=== RUN   TestCrtSh
--- PASS: TestCrtSh (1.58s)
=== RUN   TestCrtSh_MultiThreaded
--- PASS: TestCrtSh_MultiThreaded (60.00s)
=== RUN   ExampleCrtSh
--- PASS: ExampleCrtSh (1.26s)
=== RUN   ExampleCrtSh_MultiThreaded
--- PASS: ExampleCrtSh_MultiThreaded (60.00s)
goos: darwin
goarch: amd64
BenchmarkCertSpotterSingleThreaded-8               1    26228651368 ns/op
BenchmarkCertSpotterMultiThreaded-8                1    27422151807 ns/op
BenchmarkCrtShSingleThreaded-8                     1    60005432034 ns/op
BenchmarkCrtShMultiThreaded-8                      1    60002526401 ns/op
BenchmarkHackerTargetSingleThreaded-8              2     594651929 ns/op
BenchmarkHackerTargetMultiThreaded-8               2     616467708 ns/op
PASS
ok      _/Users/kentgruber/Desktop/Public_Projects/research/core/sources    300.438s