search

codingo / Interlace

Easily turn single threaded command line applications into a fast, multi-threaded application with CIDR and glob support.
GNU General Public License v3.0
1.19k stars 178 forks source link

Multithreading does not work on OSX #36

Closed amalmurali47 closed 5 years ago

amalmurali47 commented 5 years ago

Hi,

When I run like Interlace like this:

interlace -tL targets.txt -threads 20 -c "dirsearch -u _target_ -t 30 -e '' -w wordlist.txt > results.txt" -v

It feels more like:

while read target; do dirsearch [args] > output.txt; done < targets.txt

It seems to run everything sequentially. @hakluke confirms that he also faced the same issue.

prodigysml commented 5 years ago

Thanks for that! Trying to get my hands on a Mac to test this issue.

hakluke commented 5 years ago

Hey @amalmurali47 - thanks for making the issue. I did have the same issue on mac so it seems to be OSX related. Unfortunately I also no longer have a mac to test on.

Do you think you could just paste the output of the following as proof + testbench?

seq 1 5 > ./test.txt
time interlace -tL ./test.txt -threads 5 -c "sleep _target_"

When threading works properly, it should take around 5 seconds. If it is single threaded, it will take about 15.

xYantix commented 5 years ago

I have a Macbook. Testing isn't working properly for me tonight. Will try again tomorrow and let you know how I go!

xYantix commented 5 years ago
Interlace

xYantixs-MacBook-Pro:Interlace xyantix$ seq 1 5 > ./test.txt xYantixs-MacBook-Pro:Interlace xyantix$ time python interlace.py -tL ./test.txt -threads 5 -c "sleep target"

Interlace v1.3.4 by Michael Skelton (@codingo_)

[10:28:09] [THREAD] [sleep 4] Added to Queue [10:28:09] [THREAD] [sleep 5] Added to Queue [10:28:09] [THREAD] [sleep 2] Added to Queue [10:28:09] [THREAD] [sleep 3] Added to Queue [10:28:09] [THREAD] [sleep 1] Added to Queue

real 0m15.236s user 0m0.055s sys 0m0.083s

Added a picture because the formatting was funky. Looks single threaded though.

prodigysml commented 5 years ago

Awesome! Thanks mate! I'll try to debug what the issue is.

On Sun, 21 Apr. 2019, 12:34 pm xYantix, notifications@github.com wrote:

[image: Interlace] https://user-images.githubusercontent.com/3001014/56464665-ca766a80-6420-11e9-8a98-495219b3fab7.png `xYantixs-MacBook-Pro:Interlace xyantix$ seq 1 5 > ./test.txt xYantixs-MacBook-Pro:Interlace xyantix$ time python interlace.py -tL ./test.txt -threads 5 -c "sleep target" ============================================== Interlace v1.3.4 by Michael Skelton (@codingo_) ============================================== [10:28:09] [THREAD] [sleep 4] Added to Queue [10:28:09] [THREAD] [sleep 5] Added to Queue [10:28:09] [THREAD] [sleep 2] Added to Queue [10:28:09] [THREAD] [sleep 3] Added to Queue [10:28:09] [THREAD] [sleep 1] Added to Queue

real 0m15.236s user 0m0.055s sys 0m0.083s`

Added a picture because the formatting was funky. Looks single threaded though.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/codingo/Interlace/issues/36#issuecomment-485219505, or mute the thread https://github.com/notifications/unsubscribe-auth/AEBVTU7R4B4T5HUYZWPVLVTPRPHCLANCNFSM4HFXZ5JQ .

xYantix commented 5 years ago

No worries. Let me know if you need any help with testing. Catch me on here or on the SecTalks Slack :)

On Sun, Apr 21, 2019 at 10:44 AM Sajeeb Lohani notifications@github.com wrote:

Awesome! Thanks mate! I'll try to debug what the issue is.

On Sun, 21 Apr. 2019, 12:34 pm xYantix, notifications@github.com wrote:

[image: Interlace] < https://user-images.githubusercontent.com/3001014/56464665-ca766a80-6420-11e9-8a98-495219b3fab7.png

`xYantixs-MacBook-Pro:Interlace xyantix$ seq 1 5 > ./test.txt xYantixs-MacBook-Pro:Interlace xyantix$ time python interlace.py -tL ./test.txt -threads 5 -c "sleep target" ============================================== Interlace v1.3.4 by Michael Skelton (@codingo_) ============================================== [10:28:09] [THREAD] [sleep 4] Added to Queue [10:28:09] [THREAD] [sleep 5] Added to Queue [10:28:09] [THREAD] [sleep 2] Added to Queue [10:28:09] [THREAD] [sleep 3] Added to Queue [10:28:09] [THREAD] [sleep 1] Added to Queue

real 0m15.236s user 0m0.055s sys 0m0.083s`

Added a picture because the formatting was funky. Looks single threaded though.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/codingo/Interlace/issues/36#issuecomment-485219505, or mute the thread < https://github.com/notifications/unsubscribe-auth/AEBVTU7R4B4T5HUYZWPVLVTPRPHCLANCNFSM4HFXZ5JQ

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/codingo/Interlace/issues/36#issuecomment-485219954, or mute the thread https://github.com/notifications/unsubscribe-auth/AAW4VNXVMLS7I4EGGS7C4WDPRPIINANCNFSM4HFXZ5JQ .

amalmurali47 commented 5 years ago

@hakluke: Sure. Here's the output on my machine:

➜ seq 1 5 > ./test.txt
➜ time interlace -tL ./test.txt -threads 5 -c "sleep _target_"
==============================================
Interlace v1.3.3    by Michael Skelton (@codingo_)
==============================================
[13:52:39] [THREAD] [sleep 4] Added to Queue
[13:52:39] [THREAD] [sleep 5] Added to Queue
[13:52:39] [THREAD] [sleep 2] Added to Queue
[13:52:39] [THREAD] [sleep 3] Added to Queue
[13:52:39] [THREAD] [sleep 1] Added to Queue
interlace -tL ./test.txt -threads 5 -c "sleep _target_"  
0.17s user 
0.09s system 
1% cpu 
15.278 total
Emxm3 commented 5 years ago

Using @xYantix's macbook and confirmation from he himself, we've fixed the issue with the multi-threading on OSX.

Emxm3 commented 5 years ago

(Yay, my first pull request evarrrr!)

hakluke commented 5 years ago

Phwoar nice work dudes!

xYantix commented 5 years ago
Interlace

Working nicely on my machine now! Mac OS 10.14.3