Source code refactoring and new features added

[jtof-fap]

Source code refactoring into a single block with the aim of providing a more solid base to add new features and bypass methods (to come).

Improved code and new features added:

• Bypasser class is now a full library that can be called in other projects;
• Improved the tool's portability: The tool works on Linux as well as on Windows (with python >= 3.7);
• Improved the logging level with the following arguments: -v, -d, -dd;
• Adding short options in doc_opt;
• Adding a bypass_mode argument (-m, --mode) to optionally select the desired bypass mode(s);
• Adding the support of multiple URLs (-u, --url);
• Adding proxy support (-p, --proxy);
• Adding a retry feature and a new argument (-r, --retry) to define the number of retry attempts in case of failed requests;
• Adding (-p, --spoofport) argument to inject custom ports in port-specific headers;
• Adding two options (--spoofip-replace and --spoofport-replace) to choose if the custom element adds or replaces the existing ones;
• Single quote (')support in header(s) arguments (-H).

Improved argument parsing:

• -u, --url, -s, --spoofip and -p, --spoofport arguments can now be a filename, a string, a comma-separated string list or a list (when Bypasser is used as a library);
• stdin support for these three arguments (with -). Ex: cat /path/urls | ./bypass-url-parser.py -u -

Improved tool's output and results saving:

• More precise "same responses" aggregation;
• The output now grepable and displays: [status_code] [content_type] [content_length] [lines_count] [word_counts] [title] [server] [redirect_url];
• Adding (-S, --save-level) to choose saving level of the results. From 0: disable to 3: save log file and each curl request/response in output directory.

More PR to come ;-)

[laluka]

Heya! PR received, testing will be done within a few days. We spend quite some time chatting but I want this information to be public anyway: The work done here is HUGE, this SHALL be merged! :heart:

[laluka]

Ok, so this PR doesn't seem to introduce any regression, and adds tons of cool features! BUT we have a few things left to do before we merge it:

Fix the escaping so we don't introduce an argument rce

This would be really bad for folks with automation, and can also break some urls

python bypass-url-parser.py -u "http://127.0.0.1/foo/bar';`touch /tmp/rce`" -d
# Results in /tmp/rce being created

Update the readme with the new usages

• cat /path/urls | ./bypass-url-parser.py -u -
• Document a bit the library usage, offer a sample file
• New flags, idealy one exemple per flag
• Paste the new -h output from docopt

Small log tweak

Display one line every 100 tests done to let the use know the tool is still running

Awesome work overall, looking forward to build from this one! :heart: :rose:

[laluka]

Don't mind my sample rce payload, it is missleading and erroneous, I'm still pretty sure this is vuln but this rce isn't introduced by the tool, I was definitely too tired.. :)

[jtof-fap]

Hi!

It's done! I pushed three extra commits to this pull request:

• Clean up arguments with shlex.quote() and switch to sub-process.Popen (with shell=False and no more sh -c) to prevent eventual rce problems;
• Adding a progress bar;
• Updated README detailing new features and arguments;

Enjoy ;-)