search

codingo / Interlace

Easily turn single threaded command line applications into a fast, multi-threaded application with CIDR and glob support.
GNU General Public License v3.0
1.21k stars 179 forks source link

Interlace hangs with large amount of IPs in a file #131

Closed 0xtavian closed 3 years ago

0xtavian commented 3 years ago

Get 50,000 random IPs from https://onlinerandomtools.com/generate-random-ip and save to file named ips.txt The following command hangs and never starts. CPU is maxed out. From what I can tell, this bug only impacts IPs in a list. For example: interlace -tL ips.txt -c "echo _target_ " -threads 5

Attempting with hostnames instead of IPs works as expected and is successful, example interlace -tL subdomains.txt -c "echo _target_ " -threads 5 (download hostnames from a target with more than 50k subdomains from https://chaos.projectdiscovery.io/#/). Attempting with CIDR expansion works as expected and is successful, example interlace -t 10.0.0.0/16 -c "echo _target_ " -threads 5

I've tried with Ubuntu 20.04 and Kali 2020.4.

prodigysml commented 3 years ago

Hey @0xtavian , this is a known issue. There is a cheeky workaround for it. Basically using a fake target (like -t a) and then passing a command list (-cL) with all the string replacements already done, will speed up the process significantly! Unfortunately, atm the string replacement is written really slow, which is why interlace starts off a bit slow. The command list workaround works well and I use it myself!

prodigysml commented 3 years ago

After some chit chat with @0xtavian, we think the issue may lie in the usage of the IP object (slows down everything). When we get around to fixing this, it's something we need to take a look at (#reminder)

pry0cc commented 3 years ago

So I did some quick n dirty forking and just removed the IP object piece altogether - ...and it works!

So maybe we just need an option to skip putting them into options? Or do we make it more performant?

https://github.com/pry0cc/Interlace/commit/64ef01783b7d7b2f0480ea195674de9506b720e6

0xtavian commented 3 years ago

I can confirm this fixes the issue as well.

prodigysml commented 3 years ago

Awesome! I'll try to take a look and push a patch soon! Might be a while as some other things are on fire atm. Thanks for being patient everyone!

pry0cc commented 3 years ago

Any movement on this issue?

prodigysml commented 3 years ago

Hey @pry0cc, tad busy with some work stuff, but I am working on it in the background :) So far my analysis is that we will need to rewrite that section (IP related and all operations around that).

prodigysml commented 3 years ago

I can confirm the patch is working a LOT better than before! It is easily compatible with 50k IPs now. Waiting on the reviewer to approve.