Incompatibility with `tee -a` and `>>` when using `-o cli`, and Need for Direct Stream to Output File Without Filtering

[Nishantbhagat57]

Issue

The -o cli option doesn't work properly with tee -a and >>

I am using xnLinkFinder with axiom-scan with 20 fleets/servers. Sometimes, the input file is so large that a full scan can take an entire day. To manage this, I use -o cli | tee -a output.txt so the results are directly streamed to the terminal and appended to a file using tee -a. This allows me to terminate the scan using Ctrl+C if it takes too long while still preserving the partial results.

However, this approach no longer seems to work. The results are not being saved when I use tee -a or >>. I haven't changed my xnLinkFinder axiom module; it worked before. I think that since the last 2-3 updates, it no longer functions as expected—or maybe it never worked, and I hadn't noticed.

Also, I think there should a flag to directly stream/yield/write result to output file without doing filtering when using -o output.txt. This way I can kill axiom-scan and the partial result will still be saved, something is better than nothing. I know if I use xnLinkFinder with -o output.txt flag, the default behavior is to save the partial result if the user interrupts with Ctrl+C. However, this process takes some time and doesn't work with axiom-scan. If a user interrupts axiom-scan with Ctrl+C, it terminates the running module and saves the results to the output file. But xnLinkFinder only saves the output file after the scan is fully completed, or if the user interrupts the program with Ctrl+C.

Example

Normal -o cli

-o cli with tee -a xntest.txt

xntest.txt is empty

@xnl-h4ck3r

[xnl-h4ck3r]

Hi @Nishantbhagat57. The output written to the output when using -v are not the actual results written to the file. I'm not sure if there was a change that has happened that would make this different, but I don't think so. You can just use the -o output.txt instead (I would also use the other output args for params, wordlists. and oos, as you may as well while running it anyway). If yoy press Ctrl-C to stop it then it will stop the crawling and then write the gathered results so far to the output files (unless you keep pressing Ctrl-C to not save any results). It does it this way (and alwasy has) so that it has a deduped list as a result rather than a potentially huge file with the same endpoints listed over and over. Does that make sense?

[Nishantbhagat57]

Thanks for the reply @xnl-h4ck3r

I understand what you mean. And yes the tool is totally functioning as it should be. Bit in my case I am using axiom-scan. As mentioned in the issue, the input urls file can sometimes be really big that I may have to close the scan so it can save the partial results. axiom-scan saves the partial result that is in the output file, it doesn't sends the Ctrl+C signal to the running tool/command, instead it just takes the output file and copies and sort -u all the resilt from different feelts into the local machine.

But in xnLinkFinder case, the output file will always remain empty unless: the program ran fully and xnLinkFinder have saved the result successfully, or if user presses Ctrl+C then it saves the partial result. But, it takes some time to do that. I haven't checked the code but I am guessing it's doing some filtration before saving the output.

So for this axiom-scan issue—which I know most users will never face unless they are scanning large files—I think there should be an argument that directly streams/yields/write the result to the output file, or maybe saves by filtering in chunks. In this way the data will be saved to output file and the axiom-scan can cat output* | sort -u.

I have already implemented a quick solution. I am not near my laptop right now, but I will share it tomorrow. Honestly, the solution wasn’t that good, but it did work.

[xnl-h4ck3r]

Sorry @Nishantbhagat57, I didn't fully understand your issue originally (my fault for not reading it properly!), but I think I do now. I assume you are only interested in getting links and not params, wordlist, etc? I'd be happy to take a look at the solutuion you implemented for this first. Thanks

[Nishantbhagat57]

@xnl-h4ck3r Honestly, I used GPT-4o to modify the code to add -do or --direct-output argument that will write all the found links/URLs to the output file or CLI. You can check the modified code here: https://raw.githubusercontent.com/Nishantbhagat57/OK-VPS/main/xnlinkfinder_direct.py

It was just a quick fix, and I'm further using pencode unicodedecode to solve encoding issues in the result file. But, the solution isn't perfect, and I haven't verified if it saves the same results when using the added -do flag as xnLinkFinder does normally.

I will look into it again to find a better solution. I think your point is also valid; I should be saving parameters and words instead of just URLs. I would need to modify my axiom module to use a directory structure "ext":"dir" instead of a text file "ext":"txt" to save all 3 result files (links,parms,words) inside a folder.

For the best solution, there should be a --stream or --direct-output flag that directly prints/writes the results or processes the URLs in chunks (e.g., 10k URLs at a time) and then prints the results. This process should repeat, processing URLs in chunks and printing the results each time.

[xnl-h4ck3r]

Hi @Nishantbhagat57, thanks for that. Having the direct output to the files sounds like a good option to provide, and will just write to the separate links, params, etc. files just as and when it finds them. This can result in files that are a lot larger and contain a lot of duplicate entries, but these can be sorted by the user afterwards anyway. You mentioned using pencode to solve unicode encoding issues in the result file... can you give me an example of en endpoint in the results file that was a problem? Did it have things like \u003D in there?

[Nishantbhagat57]

Hi @xnl-h4ck3r,

Here’s an example:

Note: I am using the modified xnLinkFinder code: https://raw.githubusercontent.com/Nishantbhagat57/OK-VPS/main/xnlinkfinder_direct.py

Suppose the original result file is xnlinkfinder.txt, generated using the command:

python3 xnlinkfinder_direct.py -i input.txt -insecure -v -t 10 -d 5 -sf hilton.com -o xnlinkfinder.txt -do

You can see that the file gives encoding errors:

Note: This is just a partial result for example purpose; hence, the file size is only 3.1 MB.

Then if I run:

cat xnlinkfinder.txt | while read -r line; do echo "$line" | pencode unicodedecode; echo ""; done | while IFS= read -r line; do echo "$line"; done >> pencode_xnlinkfinder.txt

As you can see, the encoding issues are resolved, and the file can now be opened normally, and the file size of pencode_xnlinkfinder.txt is just 36.7 KB:

Also see wc -l * and ls -l * results:

I can't confirm this rn, but I am fairly certain that the results will be the same for the normal (unmodified) xnLinkFinder.py.

[xnl-h4ck3r]

Oh I see! That is because it's the terminal output and not the raw output from the files, so that shouldn't be a problem if I do the change to just output as it finds it. Thanks for all the info

[xnl-h4ck3r]

Hi @Nishantbhagat57. I started looking at this, but I;m not sure if it already does what you need. For example, if I run xnLinkFinder -i hackerone.com -sf hackerone.com | tee xnlftest then it just displays the links to the terminal, and the same thing is written to the xnlftest file. It only does this for links though, not params and wordlist. Is that what you wanted it to do?

[Nishantbhagat57]

Hi @Nishantbhagat57. I started looking at this, but I;m not sure if it already does what you need. For example, if I run xnLinkFinder -i hackerone.com -sf hackerone.com | tee xnlftest then it just displays the links to the terminal, and the same thing is written to the xnlftest file. It only does this for links though, not params and wordlist. Is that what you wanted it to do?

Initially, yes, but as you pointed out, I can also save params and words at the same time. Now, I want this direct stream feature for all three functionalities (links, params, words).

[xnl-h4ck3r]

I'm not too sure how I'd provide links, params, words etc, all in the same output stream. Would you want something like:

[ LINK ] https://target.com
[ WORD ] blah
[ PARAM ] redirectUrl

What was your thinking on that?

[Nishantbhagat57]

I'm not too sure how I'd provide links, params, words etc, all in the same output stream. Would you want something like:

[ LINK ] https://target.com
[ WORD ] blah
[ PARAM ] redirectUrl

What was your thinking on that?

You can do that for -o cli, but I was referring to the case where I use these arguments --output rawlinks.txt --output-params params.txt --output-wordlist wordlist.txt. This would write all the particular results to the respective files. However, as the solution requires a functionality to directly save the results without filtering (or maybe filter in chunks of 10,000 results at a time and save them), you would have to introduce another flag/argument, such as --stream or --direct-output, and the result should be saved in the files mentioned by the user.

I hope this clarifies what I meant.

[Nishantbhagat57]

I'm not too sure how I'd provide links, params, words etc, all in the same output stream. Would you want something like:

[ LINK ] https://target.com
[ WORD ] blah
[ PARAM ] redirectUrl

What was your thinking on that?

BTW, this also looks good. I can simply use grep to grep links or params or words whatever I want :)

[Nishantbhagat57]

Any updates on this?

I'm not too sure how I'd provide links, params, words etc, all in the same output stream. Would you want something like:

[ LINK ] https://target.com
[ WORD ] blah
[ PARAM ] redirectUrl

What was your thinking on that?

As I mentioned previously, this approach looks promising. Please proceed with the implementation; I am eagerly waiting for the update :)