Gigantic files

[e3rd]

(see gitlab)

• [x] odstraň pak 30 GB file in 04_26 folder

[e3rd]

200 lines /s , conveying all, invalids true

50 000, 8 minut 100 000, 16 min

Řádky s porty byly vynechány, no mirror: 100 000, 6 minut 1000 000, 7 min 4,5 mil, 8 min 21, mil, 13 min 69 mil, 25 min 194 mil, 60 min když se vypíšu za méně než vteřinu, zpomalit výpis

[e3rd]

with implemented writing to 1000 files at once, this may speed it up a lot, very curious to see the functionality coming in the new version ( #21 )

[e3rd]

That's perfect. No lines were skipped. About 4 milion lines / minute. However it is difficult to compare with previous results since that's another SSD and we were not skipping lines.

First try:
Log lines processed: 6341855/350000000, 2 %, 0:05:41, 34968 lines / s
Log lines processed: 11338250/350000000, 4 %, 0:06:50, 29749 lines / s
Whois servers asked: general (962×)

Second try:
Log lines processed: 14818787/350000000, 5 %, 0:07:09, 39492 lines / s
Whois servers asked: general (1015×)
Log lines processed: 32503460/350000000, 10 %, 0:10:55, 56634 lines / s
Whois servers asked: general (1217×), descriptors open 53
Log lines processed: 44382750/350000000, 13 %, 0:13:20, 26974 lines / s
Whois servers asked: general (1287×), descriptors open 54
Log lines processed: 64728850/350000000, 19 %, 0:17:41, 31649 lines / s
Whois servers asked: general (1361×), descriptors open 54
Log lines processed: 83954200/350000000, 24 %, 0:21:23, 69149 lines / s
Whois servers asked: general (1423×), descriptors open 54