This PR resolves #21 by adding the new function Processor.CheckRequest(*tls.ClientHelloInfo,*http.Request) as a wrapper around the functionality of Processor.Check() along with updates to documentation, tests, and a demo to show webserver developers how they could use mitmengine to get reports on incoming traffic. It addresses the use-case mentioned by Matt Holt of Caddy in this issue.
The largest unexpected design challenge faced was the fact that the crypto/tls.ClientHelloInfo structure does not expose which TLS extensions were in use in the actual client hello. These extension sets are part of normal signatures parsed under the original design. The presence of five extensions could be inferred from the availability of fields in crypto/tls.ClientHelloInfo, but dozens more are not visible. This causes matching failures.
In consultation with Luke, I decided to modify the signature matching logic to ignore TLS extensions when using CheckReport(). To keep the original API intact, I made both Check() and CheckReport() into wrappers around the original check() functionality with an added flag to allow CheckReport() to have TLS extension mismatches ignored. This allowed for minimal changes to existing functionality. This also meant that when testing that Check() and CheckReport() generate consistent responses for all sample traffic I needed to write extra logic to account for the difference of available inputs to each.
The result is that we can use the extensive existing reference set of signatures while ignoring TLS extensions that are not visible to CheckRequest(). This is a trade-off to make an easier-to-use API for webserver developers at the cost of some accuracy. This trade-off is documented so developers can make an informed selection.
Other Design Choices
crypto/tls.ClientHelloInfo structure infers multiple supported TLS versions—even for pre TLS 1.3 connections. Chose to use the highest supported version because we don't have access to the actual version in use by the connection.
Used uasurfer to parse UA strings rather than other parsers since that was already a project dependency for simplicity.
Documented behavior differences CheckReport() vs. Check() for clarity
Squashed commits to minimize others' potential rebase conflicts.
Where to find changes:
Changed Check() in processor.go to check() and added two wrapper functions: CheckRequest and Check. This allows each to specify to check() whether to check TLS extensions against fingerprints. The implementation of this skip was a simple change to the switch statement generating the report to require the useExtensions flag to be set in order to flag any difference in TLS extensions.
Added a new function to fputil/ua.go that can fingerprint a user-agent string with the signature: func UAFingerprintFromUserAgentString(s string) UAFingerprint. It uses uasurfer to parse the User-Agent similar to the existing code in cmd/demo/main.go.
Added a new function to fputil/request.go that can fingerprint a request using a *tls.ClientHelloInfo and a *http.Request with the signature: func FingerprintClientHello(chi *tls.ClientHelloInfo, r *http.Request) (RequestFingerprint, error)
Added a new demo webserver integration at cmd/webserver_integration_demo/main.go
Added documentation of new functionality in README.md.
Tests
I added tests to ensure new features are working as intended and to alert in the future if they stop working.
Added a test in fputil/ua_test.go to validate new User-Agent string to UAFingerprint given various known and unknown User-Agents as well as an empty string. The actual parsing is done by uasurfer, so this test ought to be a canary to let us know if there are breaking changes made there.
Added a test in request_test.go to validate creating a client fingerprint from a tls.ClientHelloInfo structure and a *http.Request structure—including when inputs are missing or unset. Also tests error return values for various error conditions.
Added tests to processor_test.go to test overall integration. Every time Check() was tested, we now double check that CheckRequest would have given an equivalent answer, considering its more limited view. This required creating a function to go backwards and build a http.Request and tls.ClientHelloInfo from existing test fingerprints. This allowed me to leverage the large number of already existing test cases for Check().
Demo
I created a demo at cmd/webserver_integration_demo/main.go. Instructions to generate a cert and run the HTTPS server demo are in README.md. Then you can fetch https://localhost:8443 with any client (optionally with MITM) and see the mitmengine report returned rendered semi-nicely in HTML.
Sources Used
Luke asked me to cite my sources. This is not every page I looked at, but these were the most helpful:
This PR resolves #21 by adding the new function
Processor.CheckRequest(*tls.ClientHelloInfo,*http.Request)as a wrapper around the functionality ofProcessor.Check()along with updates to documentation, tests, and a demo to show webserver developers how they could use mitmengine to get reports on incoming traffic. It addresses the use-case mentioned by Matt Holt of Caddy in this issue.The largest unexpected design challenge faced was the fact that the
crypto/tls.ClientHelloInfostructure does not expose which TLS extensions were in use in the actual client hello. These extension sets are part of normal signatures parsed under the original design. The presence of five extensions could be inferred from the availability of fields incrypto/tls.ClientHelloInfo, but dozens more are not visible. This causes matching failures.In consultation with Luke, I decided to modify the signature matching logic to ignore TLS extensions when using CheckReport(). To keep the original API intact, I made both Check() and CheckReport() into wrappers around the original check() functionality with an added flag to allow CheckReport() to have TLS extension mismatches ignored. This allowed for minimal changes to existing functionality. This also meant that when testing that Check() and CheckReport() generate consistent responses for all sample traffic I needed to write extra logic to account for the difference of available inputs to each.
The result is that we can use the extensive existing reference set of signatures while ignoring TLS extensions that are not visible to CheckRequest(). This is a trade-off to make an easier-to-use API for webserver developers at the cost of some accuracy. This trade-off is documented so developers can make an informed selection.
Other Design Choices
crypto/tls.ClientHelloInfostructure infers multiple supported TLS versions—even for pre TLS 1.3 connections. Chose to use the highest supported version because we don't have access to the actual version in use by the connection.uasurferto parse UA strings rather than other parsers since that was already a project dependency for simplicity.Where to find changes:
Changed
Check()inprocessor.gotocheck()and added two wrapper functions:CheckRequestandCheck. This allows each to specify tocheck()whether to check TLS extensions against fingerprints. The implementation of this skip was a simple change to the switch statement generating the report to require theuseExtensionsflag to be set in order to flag any difference in TLS extensions.Added a new function to
fputil/ua.gothat can fingerprint a user-agent string with the signature:func UAFingerprintFromUserAgentString(s string) UAFingerprint. It usesuasurferto parse the User-Agent similar to the existing code incmd/demo/main.go.Added a new function to
fputil/request.gothat can fingerprint a request using a*tls.ClientHelloInfoand a*http.Requestwith the signature:func FingerprintClientHello(chi *tls.ClientHelloInfo, r *http.Request) (RequestFingerprint, error)Added a new demo webserver integration at
cmd/webserver_integration_demo/main.goAdded documentation of new functionality in
README.md.Tests
I added tests to ensure new features are working as intended and to alert in the future if they stop working.
Added a test in
fputil/ua_test.goto validate new User-Agent string to UAFingerprint given various known and unknown User-Agents as well as an empty string. The actual parsing is done by uasurfer, so this test ought to be a canary to let us know if there are breaking changes made there.Added a test in
request_test.goto validate creating a client fingerprint from atls.ClientHelloInfostructure and a*http.Requeststructure—including when inputs are missing or unset. Also tests error return values for various error conditions.Added tests to
processor_test.goto test overall integration. Every time Check() was tested, we now double check that CheckRequest would have given an equivalent answer, considering its more limited view. This required creating a function to go backwards and build a http.Request and tls.ClientHelloInfo from existing test fingerprints. This allowed me to leverage the large number of already existing test cases for Check().Demo
I created a demo at
cmd/webserver_integration_demo/main.go. Instructions to generate a cert and run the HTTPS server demo are inREADME.md. Then you can fetchhttps://localhost:8443with any client (optionally with MITM) and see themitmenginereport returned rendered semi-nicely in HTML.Sources Used
Luke asked me to cite my sources. This is not every page I looked at, but these were the most helpful:
About TLS
Preserving ClientHelloInfo after handshake in a HTTPS Server:
Messaging in golang tests
Etiquette in open-source PRs:
User-Agents and headers
Misc