add sources

[picatz]

This PR adds an initial implementation of a custom source and adds a directory to house future sources.

[picatz]

This is fun

$ curl "https://api.hackertarget.com/hostsearch/?q=google.com"

API count exceeded - Increased quota available with membership

[picatz]

From the hacker target website:

There is a limit of 100 API calls per day from a single IP address. No guarantees are made for accuracy or reliability of this service.

[picatz]

Benchmark results after maxing out the API. So, these aren't really saying much.

$ go test -bench=. -v

=== RUN   TestIsOverFreeLimit
--- PASS: TestIsOverFreeLimit (0.35s)
    hacker_target_test.go:13: warning: the tests will be ran with API request limit hit
=== RUN   TestHackerTarget
--- PASS: TestHackerTarget (0.17s)
=== RUN   TestHackerTarget_MultiThreaded
--- PASS: TestHackerTarget_MultiThreaded (0.20s)
=== RUN   ExampleHackerTarget
--- PASS: ExampleHackerTarget (0.17s)
=== RUN   ExampleHackerTargetMultiThreaded
--- PASS: ExampleHackerTargetMultiThreaded (0.19s)

goos: darwin
goarch: amd64

BenchmarkHackerTargetSingleThreaded-8             10     182458031 ns/op
BenchmarkHackerTargetMultiThreaded-8               5     231854402 ns/op
``
PASS
ok      _/research/core/sources 4.462s

[picatz]

Now, with a different IP address, using the free google cloud shell VM (default mode):

$ go test -bench=. -v

=== RUN   TestIsOverFreeLimit
--- PASS: TestIsOverFreeLimit (0.31s)
=== RUN   TestHackerTarget
--- PASS: TestHackerTarget (0.54s)
=== RUN   TestHackerTarget_MultiThreaded
--- PASS: TestHackerTarget_MultiThreaded (0.57s)
=== RUN   ExampleHackerTarget
--- PASS: ExampleHackerTarget (0.51s)
=== RUN   ExampleHackerTargetMultiThreaded
--- PASS: ExampleHackerTargetMultiThreaded (0.56s)

goos: linux
goarch: amd64

BenchmarkHackerTargetSingleThreaded            2         502639714 ns/op
BenchmarkHackerTargetMultiThreaded             2         572807879 ns/op

PASS
ok      _/research/core/sources   5.746s

[picatz]

Default google cloud shell VM CPU:

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                1
On-line CPU(s) list:   0
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 45
Model name:            Intel(R) Xeon(R) CPU @ 2.60GHz
Stepping:              7
CPU MHz:               2600.000
BogoMIPS:              5200.00
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              20480K
NUMA node0 CPU(s):     0
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc r
ep_good nopl xtopology nonstop_tsc eagerfpu pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2
apic popcnt aes xsave avx hypervisor lahf_lm xsaveopt tsc_adjust

[picatz]

Once the API limit is hit for the google cloud shell VM:

goos: linux
goarch: amd64

BenchmarkHackerTargetSingleThreaded           10         137268038 ns/op
BenchmarkHackerTargetMultiThreaded            10         179668906 ns/op

[picatz]

Note: run on a single Intel(R) Xeon(R) CPU @ 2.60GHz

goos: linux
goarch: amd64

Round 1

BenchmarkHackerTargetSingleThreaded            2         508382563 ns/op
BenchmarkHackerTargetMultiThreaded             2         618926973 ns/op

Round 2

BenchmarkHackerTargetSingleThreaded            2         577797819 ns/op
BenchmarkHackerTargetMultiThreaded             2         633551437 ns/op

Round 3

BenchmarkHackerTargetSingleThreaded            2         534767271 ns/op
BenchmarkHackerTargetMultiThreaded             2         629714929 ns/op

Round 4

BenchmarkHackerTargetSingleThreaded            2         536905113 ns/op
BenchmarkHackerTargetMultiThreaded             2         513762424 ns/op

Round 5 (limit hit)

BenchmarkHackerTargetSingleThreaded           10         142827507 ns/op
BenchmarkHackerTargetMultiThreaded            10         159362388 ns/op