mirror http only (non-tls) if terminating tls on haproxy possible?

[amsnek]

Is it somehow possible to directly mirror traffic unencrypted if terminating on haproxy? (without construcing a loop that sends to a backend (that is a fronted) or similiar complex?

[zaga00]

Hello @amsnek, if haproxy terminates ssl, unencrypted content is sent to the mirror.

[amsnek]

then I must have some error -> will retry. For me the Mirroring worked only for requests on the haproxy Frontend on port 80/unencrypted -> with TLS + Termination, nothing was send to the port 80 (localhost) mirror ( --mirror-url http://localhost) -> I somehow assumed it was maybe because of encryption

[amsnek]

Edit: the cause is the haproxy frontend running with H2

config is essentially per examples

# haproxy.cfg:
frontend MIRROR
        bind 10.10.10:80
        bind 10.0.10.10:443  ssl crt-list /data/crt_list-TEST.txt alpn h2,http/1.1
        mode http
        option forwardfor
        tcp-request inspect-delay 3s

        option http-buffer-request
        filter spoe  engine mirror  config /data/mirror.conf
    default_backend mirrortest

backend mirroragents
    mode tcp
    balance roundrobin
    timeout connect 5s
    timeout server 5s
    server agent1 localhost:12345

backend mirrortest
     mode http
     http-response set-header Strict-Transport-Security max-age=31536000
     server mirrortest 127.0.0.1:443 ssl verify none 

# /data/mirror.conf
[mirror]
spoe-agent mirror
    log global
    messages mirror
    use-backend mirroragents
    timeout hello 500ms
    timeout idle 5s
    timeout processing 5s

spoe-message mirror
    args arg_method=method arg_path=url arg_ver=req.ver arg_hdrs=req.hdrs_bin arg_body=req.body
    event on-frontend-http-request

starting mirror agent with

spoa-mirror -r0 -u"http://localhost:8100/" --logfile /var/log/haproxy-mirror.log

running a python simple http server:

python3 -m http.server --bind localhost 8100
Serving HTTP on 127.0.0.1 port 8100 (http://127.0.0.1:8100/) ...

if a http request is done with http: curl "http://domain.tld/test.txt" -> a request appears on the python simplehttp

python3 -m http.server --bind localhost 8100
Serving HTTP on 127.0.0.1 port 8100 (http://127.0.0.1:8100/) ...
127.0.0.1 - - [31/Aug/2023 10:19:32] code 404, message File not found
127.0.0.1 - - [31/Aug/2023 10:19:32] "GET /test.txt HTTP/1.1" 404 -

if a http request is done with https: curl "https://domain.tld/test.txt" -> nothing appears

in the spoa-mirror log the following error catched my eye:

[ 1][    5.971942]       "GET http://localhost:8100https://domain.tld/test.txt HTTP/???" 0 0/0 0.000 Port number ended with 'h'

if i restart the agent with the following, notice the space after :8100/ -> a request appears on the python simple http server, even though its a bad request (400)

spoa-mirror -r0 -u"http://localhost:8100/ " --logfile /var/log/haproxy-mirror.log

127.0.0.1 - - [31/Aug/2023 10:33:32] code 400, message Bad request syntax ('GET / /test.txt HTTP/1.1')
127.0.0.1 - - [31/Aug/2023 10:33:32] "GET / /test.txt HTTP/1.1" 400 -

-> this error is suspicious -> after further digging this seems related to how H2 behaves -> disabling H2 ( bind 10.0.10.10:443 ssl crt-list /data/crt_list-TEST.txt http/1.1) resolves this

Should I open a seperate issue for this (HTTP2 issue with mirror agent)?

[zaga00]

@amsnek, thanks for the debug output.

I think the problem is that in that case, instead of args arg_method=method arg_path=url arg_ver=req.ver arg_hdrs=req.hdrs_bin arg_body=req.body , args arg_method=method arg_path=path arg_ver=req.ver arg_hdrs=req.hdrs_bin arg_body=req.body should be written in the file spoe.cfg

[amsnek]

thanks @zaga00 !

can confirm, this works for http/1.1 unencrypted + TLS, as well as with HTTP2 and HTTP3. This would be the better default configuration 👍

[amsnek]

Hello @zaga00 I just realised that while "arg_path=path" works, it does not contain the full request (query etc). is there a way to provide the full path+arguments? (URI)? Should I open a feature request instead?

[zaga00]

Hello @amsnek,

can you give an example with a log for what you're writing about? I have tried using haproxy 2.8 and I think it works ok (haproxy is up on port 10080, the backend web server is on port 8000, while the mirror is sent to port 8100).

% curl -4v "http://localhost:10080/index.html?name=ferret&color=purple"

% tail /tmp/.thttp*
==> /tmp/.thttpd-8000.log <==
127.0.0.1 - - [26/Feb/2024:20:23:15 +0100] "GET /index.html?name=ferret&color=purple HTTP/1.1" 200 44 "" "curl/7.74.0"
==> /tmp/.thttpd-8100.log <==
::1 - - [26/Feb/2024:20:23:15 +0100] "GET /index.html?name=ferret&color=purple HTTP/1.1" 200 44 "" "curl/7.74.0"

[amsnek]

Hello @zaga00

Thanks for looking into this, can you provide your mirror conf? For me the following setup only mirrors the path and not arguments (?ferret=true is missing on the mirror destination). Maybe I am doing something wrong though

my mirror.conf:

spoe-agent mirror log global messages mirror use-backend mirroragents timeout hello 500ms timeout idle 5s timeout processing 5s

spoe-message mirror args arg_method=method arg_path=path arg_ver=req.ver arg_hdrs=req.hdrs_bin arg_body=req.body event on-frontend-http-request

haproxy.cfg uses mirror.conf on the frontend that receives the curl request

filter spoe engine mirror config /data/haproxy/mirror.conf

curl request (domain.tld redacted/points to haproxy frontend which has the SPOE)

curl -v -k https://domain.tld/?ferret=true

spoa-mirror mirrors to 127.0.0.1:4080

/data/spoa-mirror -r0 -u"http://127.0.0.1:4080" [ 1][ 16.794383] "GET http://127.0.0.1:4080/ HTTP/1.1" 200 0/33175 458.132 ok

-> the argument ?ferret=true is not passed I use this to mirror to modsecurity, i receive this:

on the receiving ,mirror end

{ "transaction": { "client_ip": "127.0.0.1", "time_stamp": "Tue Feb 27 13:04:37 2024", "server_id": "5da4e01a242b60f9db80fa02d71d870ee1571c98", "client_port": 13289, "host_ip": "127.0.0.1", "host_port": 4080, "unique_id": "17090354770.214570", "request": { "method": "GET", "http_version": 1.1, "uri": "/", "headers": { "Host": "domain.tld", "user-agent": "curl/7.61.1", "accept": "/" } ...

spoa-mirror version

spoa-mirror v1.2.18 [build 2519] by Miroslav Zagorac mzagorac@haproxy.com, Feb 19 2024

[zaga00]

Hello @amsnek,

I used test/haproxy-mw.cfg and test/spoe.cfg configurations for testing.

[amsnek]

Hello @zaga00 Hmm, yes with "arg_path=url" it seems to work as expected ... I could have sworn that with "arg_path=url" it caused issues with the http scheme -> thanks. Will change arg_path=path" -> arg_path=url" Closing the Issue, thanks :)

[zaga00]

Hello @amsnek,

yes, I think that in some cases arg_path=url should be used and in others arg_path=path - I don't remember exactly when but I think it depends on the protocol being used

[amsnek]

Hello @zaga00

Yeah, you are right, the issue happens with H2 + HTTPS.

GET http://127.0.0.1:4080https://domain.tld/?ferret=true HTTP/???" 0 0/0 0.000 Port number ended with 'h'

Would it be possible to get "arg_path=uri" instead path/url? uri being path+parameters 😄

[zaga00]

Hello @amsnek,

I think you should contact the haproxy mailing list (haproxy@formilux.org) for this problem because it is a problem in the implementation of SPOP - a change in the behavior of arguments related to a change in protocol.

I want to say - it's not a bug in spoa-mirror program.

[amsnek]

hmm ah I see. I thought it was mostly implementing "URI vs URL or PATH" but didnt consider that is something that has to be aviable in the SPOP (assuming that isnt currently). https://docs.haproxy.org/2.6/configuration.html#1.2.1

[amsnek]

but Thanks a lot! This has given me some insights, also reminded me that H2 causes issues with my intentions to use the mirror module.

[amsnek]

I wonder if this works in the haproxy enterprise module https://www.haproxy.com/documentation/haproxy-enterprise/enterprise-modules/traffic-mirroring/

-> should face the same issue I assume

[zaga00]

For questions related to HAProxy Enterprise, please apply here: https://www.haproxy.com/contact-us, or send an email to contact@haproxy.com .

[amsnek]

yeah thanks, I will take a look into that