"Too many open files" errors cause some targets was missed

Dliv3 commented 2 years ago

Steps to reproduce

https://github.com/byt3bl33d3r/CrackMapExec/blob/master/cme/protocols/smb.py#L411

Modify the create_smbv3_conn function to the following code to print exception message

    def create_smbv3_conn(self):
        try:
            self.conn = SMBConnection(self.host, self.host, None, self.args.port)
            self.smbv1 = False
        except socket.error as e:
            logging.debug('SMBv3 connection error on {}: {}'.format(self.host, e))
            return False
        except Exception as e:
            logging.debug('Error creating SMBv3 connection to {}: {}'.format(self.host, e))
            return False

        return True

Then, execute CME, using the default 100 threads or more

python crackmapexec.py -t 100 --verbose smb 10.0.1.1/24

CME verbose output (using the --verbose flag)

DEBUG SMBv3 connection error on 10.0.1.242: [Errno Connection error (10.0.1.242:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.113: [Errno Connection error (10.0.1.113:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.244: [Errno Connection error (10.0.1.244:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.245: [Errno Connection error (10.0.1.245:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.249: [Errno Connection error (10.0.1.249:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.248: [Errno Connection error (10.0.1.248:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.247: [Errno Connection error (10.0.1.247:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.250: [Errno Connection error (10.0.1.250:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.251: [Errno Connection error (10.0.1.251:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.252: [Errno Connection error (10.0.1.252:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.254: [Errno Connection error (10.0.1.254:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.253: [Errno Connection error (10.0.1.253:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.255: [Errno Connection error (10.0.1.255:445)] [Errno 24] Too many open files

CME Version (cme --version)

5.1.7dev

OS

macOS Big Sur 11.2.3

Detailed issue explanation

The Too many open files errors cause some targets was missed in the CME scanning process and final results

I know that I can use ulimit to increase the maximum file descriptors, but It seems better if this issue can be solved in the code level.

mpgn commented 2 years ago

thanks for the report, I will look into it :)

executionByFork commented 2 years ago

Not only a problem on macOS. This exists in Kali Linux as well (Version: 5.1.5dev)

Workarounds until a bugfix is pushed:

Increase the upper bound on file descriptors with ulimit
Chunk the scan into smaller batches

mpgn commented 2 years ago

Increase the upper bound on file descriptors with ulimit

That the solution I use. Maybe I should print the error not only in debug mode

executionByFork commented 2 years ago

Printing this error outside of debug mode would certainly help raise awareness of the issue and notify people that data is getting dropped during their scans, but I wouldn't call ulimit a solution. It's a workaround. The issue is in how CME handles connections.

From what I understand, this bug is either caused by too many concurrent connections, or the file descriptors for these sockets are not getting cleaned up properly.

If the file descriptors aren't being cleaned up programmatically and it is just left up to garbage collection, they can easily pile up, causing the upper bound to be hit and this error to occur before garbage collection attempts to close them. In this case the bug can be fixed by closing file descriptors properly.
If it is too many concurrent connections, then I imagine it should be possible to limit the number of connections CME attempts to hold open at once, waiting for sockets to close before opening new ones.

mpgn commented 2 years ago

Another idea https://twitter.com/mcohmi/status/1514420094079700994?s=20&t=3QSVZgKiT2NuRKx9hqMv2A

ZerkerEOD commented 1 year ago

Workarounds until a bugfix is pushed:

Increase the upper bound on file descriptors with ulimit

Chunk the scan into smaller batches

Where is the ulimit supposed to be set? I don't see it in the cme.conf file.

executionByFork commented 1 year ago

hrm, it does seem ulimit hasn't been explained at all in this thread. That would probably be helpful.

ulimit is short for user limit (I believe) and it is a limiter in place for how many file handles a single user on a system can have open at one time. In other words, it is not a crackmapexec implemented limit, it is a linux system enforced limit. So, you have to change this limit in the system config files, not crackmapexec files.

You can change the ulimit for a user with the command ulimit. There should be a man or help page for it, and also this stack overflow question covers a fair bit about changing the limits. https://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limit-for-a-non-root-user

byt3bl33d3r / CrackMapExec