Moving from threads to asyncio

maurosoria commented 3 years ago

Hello folks,

I am opening this issue because I do not have the time to do it myself, but I really think that now is the momentot move from threads to an asyncio based application (at least for requests and some callbacks).

I planned from the beginning to do this, but back then asyncio wasn't widely supported by most of linux distros and wasn't as mature like it is today.

This is a very important step, because it would open the door to add extra scanning capabilities to dirsearch, that now seems very difficult with a normal thread scheme.

What is the feature?

Moving from a regular thread-based system to an asynchronous based requests system would improve the performance.

Take into consideration

The idea is using an asyncio executor for processing Jobs.
It is important to follow pep8 guidelines and validate the code.
It might be a good idea considering a model who may, in the future, support scanning multiple hosts at the same time (like meg)
I guess the proper library for doing this may be aiohttp, but I guess for now it is important to stop using threads and keep using requests.
It can be very time consuming to code, but maybe using CPU bound executor for certain tasks could make a difference (maybe not).

Obviously, I will assist you if you want to cooperate in all I can, and I will review the code myself.

Thanks for cooperating, and spread the voice!

Regards, Mauro

shelld3v commented 3 years ago

Should we fully stop using threads and turn into async, or support both?

shelld3v commented 3 years ago

And the fact that it will be really hard to can control the speed if we use asyncio

maurosoria commented 3 years ago

it is possible to mix asyncio with threads, using the proper asyncio executor:

Python 3 ThreadPoolExecutor

dbfreem commented 9 months ago

@maurosoria I actually have a lot of experience moving code over to asyncio. One thing to keep in mind is it can grow legs really quickly and get complex. Do you have a suggestion on where would be a good section to pull off and move over to asyncio. I would expect to use aiohttp instead of request though, so you get the true benefit of asyncio.

Anyways, thought I would reach out and see if you wall were still moving forward with this, if so if you needed some help.

maurosoria commented 9 months ago

Hello @dbfreem ! I am very grateful for your interest !

IMO Fuzzer class is the best candidate for starting to move code. You can rewrite that class, create a super class BaseFuzzer, and just move the old class to a subclass (e.g. TheadingFuzzer) and you can write another (eg. AsyncioFuzzer).

In that way we can write universal unit tests for both fuzzers (and make other ones in the future!).

the other candidate (which I'm also looking forward to doing that) is the Controller class. But, in that case, is much more complex because you need to completely rewrite everything (report writing, user input, etc).

If you are really into this please do not hesitate in use this or another communication method you prefer (like twitter!)

dbfreem commented 9 months ago

Hey @maurosoria that sounds like a great plan. I plan on working on this over the holidays, so I will keep you up to date on the progress.

willjprice commented 7 months ago

Following

maurosoria commented 6 months ago

Hello! Any news about this ?

Prady18 commented 6 months ago

No

On Fri, Feb 2, 2024, 1:05 AM Mauro Soria @.***> wrote:

Hello! Any news about this ?

— Reply to this email directly, view it on GitHub https://github.com/maurosoria/dirsearch/issues/643#issuecomment-1922083804, or unsubscribe https://github.com/notifications/unsubscribe-auth/AR5BMYBGBR6EEJNX4ZAQ5PTYRPU7DAVCNFSM4UJ7WCNKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJSGIYDQMZYGA2A . You are receiving this because you are subscribed to this thread.Message ID: @.***>

DylanTiger commented 3 months ago

Hey what are you going to use asyncio and aiohttp or asyncio and httpx? or something else that i ignore? I think that with chatgpt help it won't need much to accomplish it :D

Prady18 commented 3 months ago

Good question! ⁉️

On Thu, May 16, 2024, 2:52 AM DylanTiger @.***> wrote:

Hey what are you going to use asyncio and aiohttp or asyncio and httpx? or something else that i ignore? I think that with chatgpt help it won't need much to accomplish it :D

— Reply to this email directly, view it on GitHub https://github.com/maurosoria/dirsearch/issues/643#issuecomment-2113477351, or unsubscribe https://github.com/notifications/unsubscribe-auth/AR5BMYGLINVZYVHH2RTFWVDZCPG2XAVCNFSM4UJ7WCNKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJRGM2DONZTGUYQ . You are receiving this because you commented.Message ID: @.***>

asadmins commented 1 month ago

Is there any progress on this project?

shelld3v commented 2 weeks ago

Is there any progress on this project?

Not yet

maurosoria / dirsearch

Moving from threads to asyncio #643

What is the feature?

Take into consideration