Improve Service Scanner Signatures

[Ali-Razmjoo]

Hello,

Thanks to @pradeepjairamani for contribution #148, now we have a service scanner, which we need to improve by adding more signatures in it! there are a few documents for its usage in here and also its implemented in port_scan module!

To add signatures, you may take a look at the conditions in engine.py.

there are two types of conditions, first one is and conditions. it contains an array of conditions, conditions type could be a string or an array. if the condition type is the string, then it must be in the response data, if it's array, one of the strings in the array must be in the response data. for instance:

[True and True [True or False or True] and True]

a real example condition:

ports_services_and_condition = {
    "http": [["HTTP/0.9", "HTTP/1.0", "HTTP/1.1", "HTTP/2.0"]],
    "ftp": ["FTP", ["214", "220", "530", "230", "502", "500"]],
    "ssh": ["SSH"],
    "telnet": ["Telnet"],
    "smtp": ["SMTP", ["220", "554", "250"]],
    "imap": ["IMAP"],
    "mariadb": ["MariaDB"],
    "mysql": ["MySQL"],
}

the second type of condition is or, it contains an array of conditions, conditions type could be a string or an array. if the condition is a string, one of the strings must be in response data or if it's an array, all of the strings in the array must be in the response data. for instance:

[True or True or [True and False and True] or True]

a real example condition:

ports_services_or_condition = {
    "http": ["400 Bad Request", "401 Unauthorized", "302 Found", "Server: cloudflare", "404 Not Found", "HTML", "Content-Length:", "Content-Type:"],
    "ftp": [["Pure-FTPd", "----------\r\n"], "\r\n220-You are user number", ["orks FTP server", "VxWorks VxWorks"],
            "530 USER and PASS required", "Server ready.\r\n5", "Invalid command: try being more creative", "220 Hotspot FTP server (MikroTik 6.27) ready", "220 SHARP MX-M264N Ver 01.05.00.0n.16.U FTP server.",
            "220 Microsoft FTP Service", "220 FTP Server ready.", "220 Microsoft FTP Service", "220 Welcome to virtual FTP service.", "220 DreamHost FTP Server", "220 FRITZ!BoxFonWLAN7360SL(UI) FTP server ready."],
    "ssh": ["-OpenSSH_", "\r\nProtocol mism", "_sshlib GlobalSCAPE\r\n", "\x00\x1aversion info line too long"],
    "telnet": ["Welcome to Microsoft Telnet Service", "no decompiling or reverse-engineering shall be allowed",
               "is not a secure protocol", "recommended to use Stelnet", "Login authentication"],
    "smtp": ["Server ready", "SMTP synchronization error", "220-Greetings", "ESMTP Arnet Email Security", "SMTP 2.0",
             "Fidelix Fx2020"],
    "imap": ["BAD Error in IMAP command received by server", "IMAP4rev1 SASL-IR", "OK [CAPABILITY IMAP4rev1",
             "LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE NAMESPACE AUTH=PLAIN AUTH=LOGIN]",
             "LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN AUTH=DIGEST-MD5 AUTH=CRAM-MD5]"],
    "mariadb": ["is not allowed to connect to this MariaDB server", "5.5.52-MariaDB", "5.5.5-10.0.34-MariaDB"],
    "mysql": ["is not allowed to connect to this MySQL server"]
}

You can collect the data and find services from shodan.io, or use discovery() on your network and find out which port/service is UNKNOWN. then grab some signatures and add it to the framework.

In [1]: from lib.payload.scanner.service.engine import discovery

In [2]: discovery("127.0.0.1")
Out[2]:
{80: 'http',
 443: 'http/ssl',
 445: 'UNKNOWN',
 902: 'UNKNOWN',
 912: 'UNKNOWN',
 2179: 'UNKNOWN',
 3306: 'mariadb',
 6000: 'UNKNOWN'}

Regards.

[shaddygarg]

@Ali-Razmjoo, I was trying to perform the same operation on my localhost and results were different everytime.

In [1]: from lib.payload.scanner.service.engine import discovery

In [2]: discovery("127.0.0.1")
Out[2]: {443: 'UNKNOWN', 3306: 'UNKNOWN'}

In [3]: discovery("127.0.0.1")
Out[3]: 
{80: 'http',
 443: 'UNKNOWN',
 631: 'UNKNOWN',
 3306: 'UNKNOWN',
 5432: 'UNKNOWN',
 8002: 'http'}

In [4]: discovery("127.0.0.1")
Out[4]: 
{80: 'http',
 139: 'UNKNOWN',
 443: 'UNKNOWN',
 445: 'UNKNOWN',
 631: 'UNKNOWN',
 3306: 'UNKNOWN',
 5432: 'UNKNOWN',
 8001: 'UNKNOWN',
 8002: 'http'}

In [5]: discovery("127.0.0.1")
Out[5]: 
{80: 'http',
 139: 'UNKNOWN',
 443: 'UNKNOWN',
 445: 'UNKNOWN',
 631: 'UNKNOWN',
 3306: 'UNKNOWN',
 5432: 'UNKNOWN',
 8001: 'UNKNOWN',
 8002: 'http'}

Am I doing anything wrong or is it some problem with the module!! Performing a port scan however works fine for me.

[shaddygarg]

When trying out ports as inputs, it is giving me this output:

In [1]: from lib.payload.scanner.service.engine import discovery

In [2]: discovery("127.0.0.1", [5000,8002])
Out[2]: {}

In [3]: discovery("127.0.0.1", [5000,8002])
Out[3]: {5000: 'http'}

In [4]: discovery("127.0.0.1", [5000,8002])
Out[4]: {5000: 'http', 8002: 'http'}

In [5]: discovery("127.0.0.1", [5000,8002,8001])
Out[5]: {5000: 'http', 8002: 'http'}

In [6]: discovery("127.0.0.1", [5000,8002,8001])
Out[6]: {5000: 'http', 8002: 'http'}

In [7]: discovery("127.0.0.1", [5000,8002,8001])
Out[7]: {5000: 'http', 8002: 'http'}

In [8]: discovery("127.0.0.1", [5000,8002,8001])
Out[8]: {5000: 'http', 8001: 'UNKNOWN', 8002: 'http'}

Why do I have to run discovery for more than 2 to 3 times to get a result?

[shaddygarg]

And now I have this. I tried printing the data. The data got printed after it returned a {}. I think this is due to non closing of socket connections. The previous sockets are waiting for results and are interfering into my newer scans.

[pradeepjairamani]

I think this is due to the function returning data without waiting for the thread to complete, I will work on this issue.

[shaddygarg]

@pradeepjairamani, I have opened a new issue regarding this #171.