How can we debug axiom-scan when hanging?

[logspace-ai]

Hello pry0cc,

Thank you so much for this amazing tool!

I would like to know how could I debug axiom when a scan is hanging.

I've tried to use --debug but it will just print out the same "wait 30 sec ..." forever at some point.

As I often face this problem, I thought it could be useful for many to have a way to properly understand / reproduce the bug.

Example of two recent scans were it will fail to finish every time but where scans are actually done:

axiom-scan $AXIOM_INPUT_FILE --fleet $AXIOM_FLEET_NAME -m httpx -silent -timeout 30 -retries 2 -nc -sc -fr -title -rate-limit-minute 24 --debug -o $AXIOM_OUTPUT_FILE | tee save_logs.txt

$AXIOM_INPUT_FILE contains 2millions lines
$AXIOM_FLEET_NAME is a fleet of 20 vps

axiom-scan $AXIOM_INPUT_FILE -m katana -silent -headless -crawl-duration 60 -fs dn -depth 2 -concurrency 1 -parallelism 1 -xhr-extraction -omit-body -omit-raw -jsonl -o output_katana_urls.json --fleet javiskatana ---debug | tee save_logs.txt

$AXIOM_INPUT_FILE contains 100 alived hosts
$AXIOM_FLEET_NAME is a fleet of 2 vps

My version is up to date & has been rebuild, without mixing providers.

Take care!

[0xtavian]

@guav0s are you still having this issue? So when each instance is finished scanning it create a file name hostname in the remote scan working directory. https://github.com/pry0cc/axiom/blob/master/interact/axiom-scan#L1040. These files are downloaded to the controller to track progress.

Also, If you ssh into an instance that is hanging, you can attach to the tmux session and see the processes output in real time.

To see which scans are still running, take the scan-id from the creating scan working directory at : /home/op/scan/$scan-id line. The scan-id in this case is httpx+16941930731794

axiom-scan input -m httpx -silent -timeout 30 -retries 2 -nc -sc -fr -title -rate-limit-minute 24               
              _
  ____ __  __(_)___  ____ ___        ______________ _____
 / __ `/ |/_/ / __ \/ __ `__ \______/ ___/ ___/ __ `/ __ \
/ /_/ />  </ / /_/ / / / / / /_____(__  ) /__/ /_/ / / / /
\__,_/_/|_/_/\____/_/ /_/ /_/     /____/\___/\__,_/_/ /_/

                                    @pry0cc
                                 & @0xtavian

creating scan working directory at : /home/op/scan/httpx+169419313422574/
module: [ httpx ] | module args: [  -silent -timeout 30 -retries 2 -nc -sc -fr -title -rate-limit-minute 24 ] | input: [ 3 lines ] |
instances:  3  [ clarke01 clarke02 clarke03 ] |
command: [ cat input | /home/op/go/bin/httpx -o output -silent -timeout 30 -retries 2 -nc -sc -fr -title -rate-limit-minute 24 ] | ext: [ txt ] | threads: [ null ]
spliting and distributing input file...
[ OK ]
https://google.com [301,200] [Google] [https://www.google.com/]
https://tesla.com [403] [Access Denied]

You can also get a list of all running scan-ids via axiom-exec tmux ls. After you get the scan-id, run the following. This will show you the instances that are still running the scan.

diff ~/.axiom/tmp/httpx+169419313422574/hosts ~/.axiom/tmp/httpx+169419313422574/status/completed/hosts 
1,3d0
< clarke01
< clarke02
< clarke03

Then ssh into an instance and attach to the tmux session running the scan that might be hanging.

axiom-ssh clarke01

 ▄▄▄      ▒██   ██▒ ██▓ ▒█████   ███▄ ▄███▓
▒████▄    ▒▒ █ █ ▒░▓██▒▒██▒  ██▒▓██▒▀█▀ ██▒
▒██  ▀█▄  ░░  █   ░▒██▒▒██░  ██▒▓██    ▓██░
░██▄▄▄▄██  ░ █ █ ▒ ░██░▒██   ██░▒██    ▒██
 ▓█   ▓██▒▒██▒ ▒██▒░██░░ ████▓▒░▒██▒   ░██▒
 ▒▒   ▓▒█░▒▒ ░ ░▓ ░░▓  ░ ▒░▒░▒░ ░ ▒░   ░  ░
  ▒   ▒▒ ░░░   ░▒ ░ ▒ ░  ░ ▒ ▒░ ░  ░      ░

Public IP: [redacted]- Instance: clarke01

clarke01:~:% tmux ls
httpx+169419313422574: 1 windows (created Fri Sep  8 17:12:22 2023)
clarke01:~:% tmux a -t httpx+169419313422574  

Also for what its worth, your second example has three hyphens in the debug argument. it should be --debug, not ---debug. Hope this helps.

[logspace-ai]

Thank you very much!

The issue hasn't recurred for quite some time, so I haven't had the opportunity to test this further.

Could we consider adding these steps to the wiki?