Exception when executing a lot of SMB targets: IndexError: list index out of range

joaovarelas commented 1 month ago

Describe the bug The output contains a couple of exceptions and stacktraces but seem to work for some hosts.

To Reproduce Steps to reproduce the behavior i.e.: Command: netexec smb lots-of-hosts-list.txt -u username -p password Resulted in:

[16:32:13] ERROR    Exception while calling proto_flow() on target 10.1.21.11: list index out of range                                                                           connection.py:174
                    ╭─────────────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────────────╮                  
                    │ /home/user/.local/share/pipx/venvs/netexec/lib/python3.12/site-packages/nxc/connection.py:166 in __init__                                                  │                  
                    │                                                                                                                                                          │                  
                    │   163 │   │   self.logger.info(f"Socket info: host={self.host}, hostname={self.hostname},                                                                │                  
                    │       kerberos={self.kerberos}, ipv6={self.is_ipv6}, link-local                                                                                          │                  
                    │       ipv6={self.is_link_local_ipv6}")                                                                                                                   │                  
                    │   164 │   │                                                                                                                                              │                  
                    │   165 │   │   try:                                                                                                                                       │                  
                    │ ❱ 166 │   │   │   self.proto_flow()                                                                                                                      │                  
                    │   167 │   │   except Exception as e:                                                                                                                     │                  
                    │   168 │   │   │   if "ERROR_DEPENDENT_SERVICES_RUNNING" in str(e):                                                                                       │                  
                    │   169 │   │   │   │   self.logger.error(f"Exception while calling proto_flow() on target                                                                 │                  
                    │       {target}: {e}")                                                                                                                                    │                  
                    │                                                                                                                                                          │                  
                    │ /home/user/.local/share/pipx/venvs/netexec/lib/python3.12/site-packages/nxc/connection.py:232 in proto_flow                                                │                  
                    │                                                                                                                                                          │                  
                    │   229 │   │   else:                                                                                                                                      │                  
                    │   230 │   │   │   self.logger.debug("Created connection object")                                                                                         │                  
                    │   231 │   │   │   self.enum_host_info()                                                                                                                  │                  
                    │ ❱ 232 │   │   │   if self.print_host_info() and (self.login() or (self.username == "" and                                                                │                  
                    │       self.password == "")):                                                                                                                             │                  
                    │   233 │   │   │   │   if hasattr(self.args, "module") and self.args.module:                                                                              │                  
                    │   234 │   │   │   │   │   self.load_modules()                                                                                                            │                  
                    │   235 │   │   │   │   │   self.logger.debug("Calling modules")                                                                                           │                  
                    │                                                                                                                                                          │                  
                    │ /home/user/.local/share/pipx/venvs/netexec/lib/python3.12/site-packages/nxc/connection.py:560 in login                                                     │                  
                    │                                                                                                                                                          │                  
                    │   557 │   │   if not self.args.no_bruteforce:                                                                                                            │                  
                    │   558 │   │   │   for secr_index, secr in enumerate(secret):                                                                                             │                  
                    │   559 │   │   │   │   for user_index, user in enumerate(username):                                                                                       │                  
                    │ ❱ 560 │   │   │   │   │   if self.try_credentials(domain[user_index], user, owned[user_index],                                                           │                  
                    │       secr, cred_type[secr_index], data[secr_index]):                                                                                                    │                  
                    │   561 │   │   │   │   │   │   owned[user_index] = True                                                                                                   │                  
                    │   562 │   │   │   │   │   │   if not self.args.continue_on_success:                                                                                      │                  
                    │   563 │   │   │   │   │   │   │   return True                                                                                                            │                  
                    │                                                                                                                                                          │                  
                    │ /home/user/.local/share/pipx/venvs/netexec/lib/python3.12/site-packages/nxc/connection.py:495 in try_credentials                                           │                  
                    │                                                                                                                                                          │                  
                    │   492 │   │   │   │   │   return self.kerberos_login(domain, username, secret, "", "",                                                                   │                  
                    │       self.kdcHost, False)                                                                                                                               │                  
                    │   493 │   │   │   │   elif hasattr(self.args, "domain"):  # Some protocols don't use domain                                                              │                  
                    │       for login                                                                                                                                          │                  
                    │   494 │   │   │   │   │   self.logger.debug("Trying to authenticate using plaintext with                                                                 │                  
                    │       domain")                                                                                                                                           │                  
                    │ ❱ 495 │   │   │   │   │   return self.plaintext_login(domain, username, secret)                                                                          │                  
                    │   496 │   │   │   │   elif self.args.protocol == "ssh":                                                                                                  │                  
                    │   497 │   │   │   │   │   self.logger.debug("Trying to authenticate using plaintext over SSH")                                                           │                  
                    │   498 │   │   │   │   │   return self.plaintext_login(username, secret, data)                                                                            │                  
                    │                                                                                                                                                          │                  
                    │ /home/user/.local/share/pipx/venvs/netexec/lib/python3.12/site-packages/nxc/protocols/smb.py:423 in plaintext_login                                        │                  
                    │                                                                                                                                                          │                  
                    │    420 │   │   │   self.logger.debug(f"Adding credential:                                                                                                │                  
                    │        {domain}/{self.username}:{self.password}")                                                                                                        │                  
                    │    421 │   │   │   self.db.add_credential("plaintext", domain, self.username, self.password)                                                             │                  
                    │    422 │   │   │   user_id = self.db.get_credential("plaintext", domain, self.username,                                                                  │                  
                    │        self.password)                                                                                                                                    │                  
                    │ ❱  423 │   │   │   host_id = self.db.get_hosts(self.host)[0].id                                                                                          │                  
                    │    424 │   │   │   self.db.add_loggedin_relation(user_id, host_id)                                                                                       │                  
                    │    425 │   │   │                                                                                                                                         │                  
                    │    426 │   │   │   out = f"{domain}\\{self.username}:{process_secret(self.password)}                                                                     │                  
                    │        {self.mark_guest()}{self.mark_pwned()}"                                                                                                           │                  
                    ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯                  
                    IndexError: list index out of range

Expected behavior No exceptions

Screenshots

NetExec info

OS: Kali
Version of nxc: v1.3.0 Need for Speed
Installed from: pipx

termanix commented 1 month ago

How many hosts you are talking about? Cause while I testing to banks, I didn't get error like this.

joaovarelas commented 1 month ago

How many hosts you are talking about? Cause while I testing to banks, I didn't get error like this.

Im doing scan to multiple /24 networks, but I would say around 250 to 300 hosts at least with port 445 reachable. I'm not sure what triggers this error but for less hosts it does not happen...

Additionally if I scan the single host that provoked the exeception, the output is fine.

EDIT: I noticed i did not specify domain -d domain but doesn't nxc assume the machine domain by default?

termanix commented 1 month ago

If the txt file contains subnets, this could be the reason. I always find active hosts first and then start testing. I'm talking about thousands hosts btw.

joaovarelas commented 1 month ago

If the txt file contains subnets, this could be the reason. I always find active hosts first and then start testing. I'm talking about thousands hosts btw.

That make sense. I read the exception with attention and the issue shall be here, by retreiving non existent index [0]

❱  423 │   │   │   host_id = self.db.get_hosts(self.host)[0].id

Will have a look at this.

joaovarelas commented 1 month ago

So today I am trying several things:

nxc smb hosts.txt -u user -p password where hosts.txt contains lots of /24 CIDR addresses
nxc smb hosts.txt -u user -p password where hosts.txt contains lots of ipv4 addresses without network prefix
same as above but with couple of empty lines, whitespaces, bad-formatted ipv4 addresses

I am not getting that exception now... Weird

termanix commented 1 month ago

Working with subnets could lead to such situations. Subnet scanning is much larger than host scanning. I'm glad you didn't encounter it. However, the maitainers will of course do in-depth research when they are available. I will examine it too in detail when available.

NeffIsBack commented 1 month ago

Can you give me your version info?

NeffIsBack commented 1 month ago

The issue is that apparently the host that you are trying to log into does not exist in the database. Though this should not happen as it gets added in enum_host_info earlier.

My best guess would be that you ran into a race condition with SQLite where you tried to read from the db before the host was added. Perhaps because SQLite was overwhelmed by all these connections?

Btw if you want to retest the issue you have to clear the database (or remove the nxc/workspace folder entirely). Otherwise the host was added by your previous scan

NeffIsBack commented 1 month ago

My best guess would be that you ran into a race condition with SQLite where you tried to read from the db before the host was added. Perhaps because SQLite was overwhelmed by all these connections?

Though, concurrency issues in the database should have been fixed with #400

joaovarelas commented 1 month ago

The issue is that apparently the host that you are trying to log into does not exist in the database. Though this should not happen as it gets added in enum_host_info earlier.

My best guess would be that you ran into a race condition with SQLite where you tried to read from the db before the host was added. Perhaps because SQLite was overwhelmed by all these connections?

Btw if you want to retest the issue you have to clear the database (or remove the nxc/workspace folder entirely). Otherwise the host was added by your previous scan

Thanks @NeffIsBack at first I had the impression it was indeed some issue with concurrency. I will clear the database and try to reproduce again.

EDIT: 1 out of 5 runs i get the exception with the same exact command, after clearing and initiating the empty database.

Kali 2024.1
Python 3.12.6
1.3.0 - NeedForSpeed - 6d4fdfd

Command: nxc smb hosts.txt -u validuser -p validpass. hosts.txt has 160 ipv4 address, i believe all of them with port 445 reachable.

Im now trying to trigger the exception with --debug but without success.

joaovarelas commented 1 month ago

Ok so triggered the exception with --debug but don't really know if it will be helpful. I had to obfuscate some strings on the log, nxc-exception.log

NeffIsBack commented 1 month ago

Took a look at the log but it is really hard to debug without being able to test it. The question would be why this host wasn't in the database in the first place. If you still have access you could:

Delete the DB
Trigger the error
Take a look at the DB to see if the host was added at some point
Take a look at the debug log to see if the protocol maybe failed to add the host to the db

nikaiw commented 6 days ago

Hello,

I've had the same (maybe) issue recently running nxc 1.3.0 - NeedForSpeed - 72fee6a It could also be just related to the code for listing shares, I'm usure.

just running:

nxc smb <redacted> --shares
2024-11-11 02:45:12,433 - INFO - SMB         <redacted>    445    <redacted>          [*] Windows 10 / Server 2019 Build 17763 x64 (name:<redacted>) (domain:<redacted>) (signing:False) (SMBv1:False)
2024-11-11 02:45:12,436 - INFO - SMB         <redacted>    445    <redacted>         [-] IndexError: list index out of range
2024-11-11 02:45:12,437 - INFO - SMB         <redacted>    445    <redacted>          [-] Error enumerating shares: [Errno 32] Broken pipe

Unfortunately I know that it's not really enough info for helping and I don't have access to that target anymore.. However checking in my logs, I had the issue in the past with the following backtrace:

2024-09-16 17:31:50,577 - ERROR - Exception while calling proto_flow() on target <redacted>: list index out of range
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.10/site-packages/nxc/connection.py", line 166, in __init__
    self.proto_flow()
  File "/home/user/.local/lib/python3.10/site-packages/nxc/connection.py", line 226, in proto_flow
    if self.print_host_info() and (self.login() or (self.username == "" and self.password == "")):
  File "/home/user/.local/lib/python3.10/site-packages/nxc/connection.py", line 555, in login
    if self.try_credentials(domain[user_index], user, owned[user_index], secr, cred_type[secr_index], data[secr_index]):
  File "/home/user/.local/lib/python3.10/site-packages/nxc/connection.py", line 490, in try_credentials
    return self.plaintext_login(domain, username, secret)
  File "/home/user/.local/lib/python3.10/site-packages/nxc/protocols/smb.py", line 424, in plaintext_login
    host_id = self.db.get_hosts(self.host)[0].id
IndexError: list index out of range

NeffIsBack commented 5 days ago

Found the root cause for this yesterday. This is triggered by a logic flaw when running against hosts with anonymous auth (and we don't supply a user).

Will push a fix the next days.

EDIT: Nevermind, maybe this error is slightly different. Should still fix an index out of range error when using --shares

Pennyw0rth / NetExec

Exception when executing a lot of SMB targets: IndexError: list index out of range #454