Add platform fingerprinting

[user021]

It would reduce scan time when the path_traversal module is used if could specify the operating system type (unix or windows) I thought at something like " --modules=path_traversal:os=unix " Implementing some thing that can automatically detect os type (some lite version of nmap or idk) would be more effective for people who prefer fire-and-forget. Either way, i thought it would be nice if we could specify os manually, would spare time and bandwidth of useless requests, looking forward to hear your opinion.

[Zapotek]

Individual modules can't receive options like plugins do, not yet (but there is planned support for it). That being said, I thing this should be a system wide option, asking the user to provide platform information would severely decrease scan time as it would cut the audit in half -- probably more.

Can't believe this didn't occur to anyone before, two thumbs up! :+1: :+1:

[user021]

Later went through my mind that that maybe a option separately from modules should be made (e.g.--os=windows/unix and maybe --technology=PHP/Ruby/Python/JSP/ASP/NET) since there are 2 modules who deppend on os type (looking through arachni files i noticed that os_cmd_injection_timing have only "sleep TIME" as payload so i don't think is included here) as for web technology would have impact over code_injection module and maybe code_injection_timing / xpath ? if i'm not mistaken.

[Zapotek]

Yeah that was my point.

[Zapotek]

Jotting some thoughts down:

• [x] Each page should be fingerprinted to decide which platforms are in play based on:
  • Headers
  • X-Powered-By
  • Server -- Be wary of reverse proxies.
  • Cookie names
  • JSESSIONID
  • PHPSESSID
  • ASPSESSIONID
  • more...
  • Parameters
  • jsessionid
  • phpsessid
  • aspsessionid
  • more...
  • URL file extension
  • .php
  • .jsp
  • .asp
  • more...
  • More...
• [x] Pages should expose a platform object like:
  • page.platform.unix?, page.platform.linux?, page.platform.windows?, page.platform.php?, page.platform.java?, etc.
  • page.platform.list -- Listing all applicable platforms, like [ :unix, :linux, :php ], [ :windows, :asp ], etc.
• [x] Modules should adapt at run-time to only send payloads for the platforms of the page they're auditing.
• [x] Users should be allowed to:
  • Disable the fingerprinter and have Arachni audit everything.
  • Specify extra global platforms.

That should do it. :)

[user021]

Let me get this straight, "Each page should be fingerprinted" means that it will analyzes each page even after os type is detected as well as technology, isn't that an increase on CPU as well memory without point ? isn't a better idea if arachni sends some requests before the scan starts in order to detect this and then the fingerpinter should be shut down

Also, what exactly you mean with "Specify extra global platforms" ? And in our case, if we have platform unix and linux, will the audit react different in each case ? if so, what exactly will be audited different

[Zapotek]

No not after, that analysis will be the one that counts. And there won't be any performance overhead since it will take place during the page parse that is being performed anyway. Plus, there's no guarantee that each page you see will be served from the same server so it makes more sense.

If the user wants to make sure that the correct platform will be globally used he can explicitly let us know instead of relying on the fingerprinter.

Lastly, Linux is Unix but Unix is not Linux -- which won't make much difference to the existing payloads but I'm a stickler for that sort of thing and could be a useful distinction in the future.

[user021]

Fair enough, as long as there will be an option to specify both os and page type, ill be pleased.

[user021]

Btw have you thought about around what successful rate we shall expect from the fingerprinter system that detects os type, i think users should know that and not rely on it if the % is too low

[Zapotek]

No idea, that's pretty much impossible to know beforehand.

[user021]

I have a question, if for example on a website that have as base technology html, there are pages which have the .html extension on end and there are pages which simply don't have any extension, if i will run the scanner with fingerprinter disabled and specify platform as html, auditing the pages which don't have .html at end will make any sense or in this situation, with the fingerprinter activated will have more benefit? also if users could only be able to disable half of fingerprinter, more exactly, specifiy page type only without os detection

[Zapotek]

Well... html is not a technology, if the only data the fingerprinter has is an .html extension then it has no data and the modules will send all payloads as they do now -- same as if there's no extension and no other data. The fingerprinter will always err on the side of caution, no data doesn't mean don't send anything, it means send everything.

Providing your own platforms with an enabled fingerprinter would cause the modules to use these for pages for which the fingerprinter failed. Providing your own platforms with a disabled fingerprinter would cause the modules to use your data for every page.

Those are just initial ideas btw, no guarantee that that will be the final behavior.

Does that answer your question?

[user021]

Yeah, and my bad, i meant php not html. But on the same website, it can't be made by using two different technologies, it have to be one and good i suppose. Sorry for my newbish questions, i don't have many knowledges relayed to that.

[Zapotek]

Well, that can very well happen but there are more reasons for the fingerprinting to be per page. It's because that will enable the fingerprinter to work even when people are using the framework as a library to perform custom audits for pretty much anything they fancy. In this case there will be no single website to have as a reference.

[user021]

One more thing, whenever the fingerprinter will fail to determine the platform type, will display this as a message on console when running it as -v or only under --debug

[Zapotek]

I'd actually like to show it as an info message, displayed by default.

[user021]

Yeah, that would be nice : )

[cclements]

might be handy to have the fingerprint warn if what it detects doesn't match the use specified flavor.

[Zapotek]

Good idea. Also, I thought of an example of where unix vs linux might matter. We may be able to narrow down the platform to unix but not be able to determine the flavor, that would at least allow us to skip windows payloads and send linux, bsd, solaris etc.

[user021]

Looks interesting : ) since the first idea i see things expanded to identify databases too. I am very curious how successful that will be, let me know when stuff settled down a bit so i can run a few tests.

[Zapotek]

Not really, the DB entries are there for the user to identify.

[user021]

That works well for me, but in case that user doesn't identify, can you go a little further with it ...and make the audit process smart, in case that sql module is on and the web throws an error with the DB type, from that point, the audit will only use payload for that specific db, just wondering

[Zapotek]

At that point it'll be a lost cause since that knowledge will have been gained after the modules have sent their HTTP requests for the audit of that element, so it'll be too late. I was planning to do this anyway for the sake of any plugins that might process the fingerprint DB later on (and issues will get a #platform attribute since that info will be available so why not include it) but it won't help the audit in the way you described.

[Zapotek]

I forgot to mention that fingerprinting happens per resource (URI). Multiple DBs per site is rare so most of the time setting some platforms as global will work but I don't won't to make that assumption due to the corner case of multiple DBs per site.

[user021]

I did run a few scans lately looking closely about how the new fingerprinting works, comparing it to Mantra browser, i have no idea how the addons from Mantra operate but for example, this link : http://www.bodaciousboutique.com/khaki-c-133_135_134_32_132_146.html Arachni detects it as apache, php while the browser can see the OS too, more exactly Debian

[Zapotek]

Arachni can see it too but it doesn't care, it doesn't need to know the exact Linux distro, just identifying it as Linux is enough.

[user021]

Well then, why it doesn't list that? "Identified as: Linux"

[Zapotek]

Ah sorry I misunderstood, you're right. I've added a few more keywords for linux distros and I'll push in a bit -- having line troubles.

[user021]

Doin more harmless tests on a few sites, more exactly, on http://www.lipsy.co.uk/store/just-in and comparing results over several pages i noticed :

Mantra : ASP.NET, IIS - web server, Windows Arachni : windows, asp, aspx

[Zapotek]

Yeah I was trying to find the PassiveRecon addon's source code in order to get more signature but couldn't find it. I think that PassiveRecon assumes IIS because the language is ASPX, however I see no explicit indication that the server is indeed IIS.

[user021]

Idk how the addon finds it but an nmap scan confirms, is IIS (80/tcp open http Microsoft HTTPAPI httpd 2.0 (SSDP/UPnP))

[Zapotek]

That may be so, but it doesn't change the fact that PassiveRecon is probably guessing as it does passive recon while nmap actively probes the remote service to gage its behavior and gain a more accurate picture.

On 06/27/2013 09:24 PM, user021 wrote:

Idk how the addon finds it but an nmap scan confirms, is IIS (80/tcp open http Microsoft HTTPAPI httpd 2.0 (SSDP/UPnP))

— Reply to this email directly or view it on GitHub https://github.com/Arachni/arachni/issues/331#issuecomment-20144891.

[user021]

Used the --lsplat command a while ago with the same build and it worked, now however i get this error, i did not try to restart OS in order to reproduce it again since have a scan goin on, same thing with --lsplug but beside that, i can run scans, any idea what's goin on ? update: weird, if i run scan with trainer module, works, if i use sqli module, i get similar error.

[user021]

1. for some reason, it fails to detect PHP language for this website http://wasmitbuechern.de/
2. same as 1 just another site http://adevarul.ro/continut/stiri/mit
3. another website, not sure here if the addon from Mantra is guessing or not, it reports Java as programming language http://videolectures.net/

4. again, not sure if guessing or not but just to let you know,

   Mantra: ASP.NET, IIS, Windows Arachni: JSP

http://www.henkel.ro/dezvoltare-durabila/mit-initiative-6174.htm

[user021]

Btw, the addon who gathers this information in browser is Wappalyzer http://wappalyzer.com/download

https://addons.mozilla.org/en-us/firefox/files/browse/207426/file/content/apps.json#top

[Zapotek]

Cool! I though it was PassiveRecon. This is has the code on GitHub and here are the signatures: https://github.com/ElbertF/Wappalyzer/blob/master/share/apps.json

I'll updated Arachni's signatures with that data and we should be good to go! :)

[user021]

Nice, what about that error i reported up there, any idea what's goin on? ill restart my OS soon but still is weird

[Zapotek]

Oh yeah sorry about that, I saw it as soon as I woke up and then forgot about it. Did you mess with that file? Looks like it may have been corrupted.

Could you put that file (sqli.rb) in a gist so I can have a look?

[user021]

OSHIT, actually, i did not mess with it but some1 else did lol, i did open that file today in order to make sure that the fingerprinting have no effect over sqli module. i can see only one way, when the file was open, my bird was on my keyboard and pressed the 0 key without me noticing and that was enough to mess it up, sorry xD

[Zapotek]

ROFL! Fair enough.

[user021]

Downloaded latest build but i see no changes over my previous post, more exactly on points 1, 2, 3 and 4 So, is the addon guessing or how menages to find it out while Arachni can't, on the same page

[Zapotek]

That's because the builds haven't been updated since the 19th. I'm having some issues with my line/ISP. But even so, 1 shows no indication that it is PHP. Same as for 2, and although it does set a PHPSESSID cookie it's for a different subdomain. Same goes for 3. Finally, 4 is indeed JSP.

[user021]

Im trying to get in touch with dev from the addon because id like to see in real time or logs debug info about how exactly it identify them, if is just because the subdomain thing then it makes perfect sense why Arachni sees something else

[user021]

Alright, so is the subdomain thing on 1 and 2 so the addon's fault, but on 4, here's what he told me :

"OpenText CMS is detected which implies ASP.NET (and Java) and thus IIS and Windows Server."

Could you implement that on Arachni too ? or that kind of guessing is not 100% accurate

[Zapotek]

Well, I figured that some day I may add application specific fingerprints but I'd like to think about it a bit more.

[user021]

[*] Spider: [HTTP: 200] http://www.burlingtoncoatfactory.com/burlingtoncoatfactory/Women/Pants-Leggings-53141.aspx [~] Identified as: windows, iis, asp, aspx

how could a single page be asp and aspx at the same time ? addon from browser sees it as aspx only

[Zapotek]

Doesn't matter, the audit payloads work on both aspx and asp so a generic check is enough. This is not about accuracy or recon, this is about having enough info to not send non-applicable payloads during the audit.

[beunwa]

I agree with user021 the ability to fingerprint applications would be really really apreciate.

Maybe you could rely on whatweb have done who use a plugin system https://github.com/urbanadventurer/WhatWeb/tree/master/plugins

[Zapotek]

That'd be a huge amount of effort for something that doesn't really fit the nature of the project. It's supposed to be a dynamic scanner, the fingerprinting parts are here purely as an optimization.

I may get around to implement this in the future but at the time there are many more and much more important things that need attention.

[beunwa]

Maybe the good solution is a simple tutorial on how to write an arachni fingerprinter plugin ? So everyone could be able to write is own and share it

[Zapotek]

You won't be able to affect the fingerprinter components, because they only care about platforms that have auditable payloads. But if you want to simply log that piece of information you can write a normal plugin and then a plugin formatter for the reports you want to use.

See: http://support.arachni-scanner.com/kb/writing-plugins