USArmyResearchLab / Dshell

Dshell is a network forensic analysis framework.
Other
5.44k stars 1.14k forks source link

Web decoder overhaul #115

Closed amm3 closed 4 years ago

amm3 commented 4 years ago

I've been meaning to look at this for a while. General use of the web decoder has been throwing UnicodeEncodeError on certain types of traffic. The cause seemed to be in the line if response: (line 88 currently), which is just checking to see if we have a response object available. (Sometimes the response isn't captured and we obviously want Dshell to handle those conditions gracefully.) The existing expression seems to cause an invocation of len() on the dpkt.http.Message object (when it exists), which causes the UnicodeEncodeError exception on evaluating len(str(self)).

The problem is probably in dpkt, but rather than dig into that, I changed our statement such that it checks for a response by comparison with None: if response!=None:. This accomplishes the decoder's purpose and is probably slightly more efficient as well.

While working on the module, I did some other code cleanup and reworked the keyword-value motif so that all HTTP headers (client and server) are now passed as keys to alert(). I am a big user of jsonout, so this has proven very useful in my analysis.

Side note: I thought tweaking the key-names so that request and response headers were more distinguished, but weighed this against the backward compatibility advantage of keeping key names consistent with the current decoder.

dev195 commented 4 years ago

Looks good. I'll go ahead and accept it.