Transport refactor

[OJ]

This PR contains a stack of changes that are based around refactoring transport to allow for transport hot-swapping. The goal is to allow the user to change transports on the fly. In order to support this, a number of changes had to happen:

• The notion of a transport had to be changed from something fixed to something that could move.
• Transport functions needed to be a tied to a changable context.
• Different transport mechanisms needed to conform to a "standard" set of functions that could all be used in a generic fashion.

During the course of making these changes, other things were done including:

• Reformatting of code and tidying up.
• Splitting of Win32 and Posix more.
• Addition of "machine ID" code which generates an identifier for the machine based on a combination of HDD serial and hostname. This functions differently on Windows and POSIX, but should be enough to allow for unique identification of machines across sessions.

  Overview of changes

• The Remote type has been changed so that it includes a Transport instance which is a set of function pointers to various functions within the code base.
• The Transport instance contains a pointer to a transport-specific context. This context type will vary depending on the transport in question. For example, tcp transports will contain a file descriptor for the socket that communications is being done on, and will have a reference to SSL-related contexts. HTTP(S) payloads will instead of URI/URL, proxy information, etc.
• Layout of the setup, dispatch, and teardown has been normalised across the transports (at least soemwhat).
• A new core command called has been added which allows for switching of communications channels. This allows for the modification of transports on the fly. The mechanism makes use of the stageless meterpreter features to deal with re-establishment of comms.
• POSIX meterp has been updated to include stageless-like functionality on the server with regards to comms, but the MSF side needs to play catch up and we need to have stageless payload support as well. That's in the pipeline, for now the code changes here for transport switching have been commented out, but the refactoring of the transport has been done.

  Verification

  Staged Linux x86

• [x] x86 bind_tcp works as before.
• [x] x86 reverse_tcp works as before.

  Staged Windows x86

• [x] x86 bind_tcp works as before.
• [x] x86 reverse_tcp works as before.
• [x] x86 reverse_ipv6_tcp works as before.
• [x] x86 reverse_http works as before.
• [x] x86 reverse_https works as before.

  Staged Windows x64

• [x] x64 bind_tcp works as before.
• [x] x64 reverse_tcp works as before.
• [x] x64 reverse_ipv6_tcp works as before.
• [x] x64 reverse_https works as before.

  Stageless Windows x86

• [x] x86 bind_tcp works as before.
• [x] x86 reverse_tcp works as before.
• [x] x86 reverse_ipv6_tcp works as before.
• [x] x86 reverse_http works as before.
• [x] x86 reverse_https works as before.

  Stageless Windows x64

• [x] x64 bind_tcp works as before.
• [x] x64 reverse_tcp works as before.
• [x] x64 reverse_ipv6_tcp works as before.
• [x] x64 reverse_https works as before.

  Machind ID Command

• [x] the machine_id command correctly functions on Linux x86.
• [x] the machine_id command correctly functions on Win x86.
• [x] the machine_id command correctly functions on Win x64.

  Transport Command

There is still some work to be done here with regards to making the transport command support setting of proxy and cert hash verification when setting transport to http(s).

• [x] The transport command correctly functions on Win x86.
• [x] The transport command correctly functions on Win x64.
• [x] The transport command is not functional on Linux x86.

The transport command requires a PR on the MSF side of the fence: https://github.com/rapid7/metasploit-framework/pull/5046

There is still work to do, but this at least gets things moving so that people can review things.

Sample runs

I'll add details of the runs to the MSF side so you can see how it works. There'll be overlap in testing with that module and this one.

[metasploit-public-bot]

Test PASSED. Refer to this link for build results (access rights to CI server needed): https://ci.metasploit.com//job/GPR-MeterpreterWin/168/ Test PASSED.

[metasploit-public-bot]

Test PASSED. Refer to this link for build results (access rights to CI server needed): https://ci.metasploit.com//job/GPR-MeterpreterWin/169/ Test PASSED.

[metasploit-public-bot]

Test PASSED. Refer to this link for build results (access rights to CI server needed): https://ci.metasploit.com//job/GPR-MeterpreterWin/170/ Test PASSED.

[jlee-r7]

Should the tcp transport functions be moved into a new file? Seems odd that it is in the same place as the generic stuff for creating transports, etc, while wininet and winhttp are separate.

[OJ]

Yes, they probably should. Also, the wininet stuff should now be deleted given that we don't (and won't) use it any more.

[metasploit-public-bot]

Test PASSED. Refer to this link for build results (access rights to CI server needed): https://ci.metasploit.com//job/GPR-MeterpreterWin/171/ Test PASSED.

[OJ]

Had some conflicts with master once the WinHTTP proxy fixes made it in. Merged the changes and did some more testing.

[metasploit-public-bot]

Test PASSED. Refer to this link for build results (access rights to CI server needed): https://ci.metasploit.com//job/GPR-MeterpreterWin/172/ Test PASSED.

[metasploit-public-bot]

Test PASSED. Refer to this link for build results (access rights to CI server needed): https://ci.metasploit.com//job/GPR-MeterpreterWin/173/ Test PASSED.

[bcook-r7]

Testing now to unblock #5046

[bcook-r7]

posix seems to have broken with the last 3 commits. server_setup exits without a word. at 5f263db things worked fine.

[OJ]

Thanks Brent. Looking at it now!

[bcook-r7]

no problem - will take a look again later this evening

[metasploit-public-bot]

Test PASSED. Refer to this link for build results (access rights to CI server needed): https://ci.metasploit.com//job/GPR-MeterpreterWin/174/ Test PASSED.

[bcook-r7]

death to wchar - looks good otherwise!

[bcook-r7]

I wasn't able to get transport/x64 stageless to work yet, but I'll merge this so far so we have some bins and can work from that.

[bcook-r7]

Cool beans, I'll have a PR for #5046 shorly. I'm still waiting on GES to add jenkins jobs for the new merged repo, so I'll be giving you manual Linux bins directly again.

[bcook-r7]

Jenkins issues - will have bins in the morning.