cloudflare / pingora

A library for building fast, reliable and evolvable network services.
Apache License 2.0
20.21k stars 1.1k forks source link

how to avoid the error and why the error: Crit: FD mismatch: fd: 23, addr: SockaddrStorage #296

Open futurist opened 1 week ago

futurist commented 1 week ago

Describe the bug

Just run the example: https://github.com/cloudflare/pingora/blob/main/pingora-proxy/examples/ctx.rs

Got follow error:

02:12:28 [ERROR] Crit: FD mismatch: fd: 23, addr: SockaddrStorage { ss: sockaddr_storage { ss_family: 2, __ss_align: 0 } }, peer: SockaddrStorage { ss: sockaddr_storage { ss_family: 2, __ss_align: 0 } }

image

Pingora info

Please include the following information about your environment:

Pingora version: 0.2.0 Rust version: 1.78.0 Operating system version: Debian 11 (Linux 695db9f754-2flsb 5.4.210.bsk.8-amd64 #5.4.210.bsk.8 SMP Debian 5.4.210.bsk.8 Mon Jun 19 12:27:45 UTC x86_64 GNU/Linux)

Expected results

No error shown, or figure out how to avoid the error

Observed results

Error shown and don't know why.

futurist commented 1 week ago

I found the error message is in below line: https://github.com/cloudflare/pingora/blob/a432c2da9b71a2cf1e5169cabc1205cc272c4c9c/pingora-core/src/protocols/mod.rs#L237

But I don't think it's critical errors, maybe the log level shouldn't be ERROR?

eaufavor commented 1 week ago

The error is critical in the way that the connection looked like that it was not pointing to the destination we want to go to, which was bad in many ways. But this could be a false positive case on Mac. So let me check

futurist commented 1 week ago

sorry the OS reported is wrong, I've used a Cloud IDE from macOS, the cargo was run on Linux Debian 11 (updated).

JosiahParry commented 3 days ago

I am running into this issue as well. I reload a page which works from my browser. But the logging is indicating an error:

[2024-06-26T17:09:16Z ERROR pingora_core::protocols] Crit: FD mismatch: fd: 25, addr: SockaddrStorage { ss: sockaddr_storage { ss_len: 16, ss_family: 2, __ss_pad1: [182, 62, 0, 0, 0, 0], __ss_align: 0 } }, peer: SockaddrStorage { ss: sockaddr_storage { ss_len: 16, ss_family: 2, __ss_pad1: [182, 62, 127, 0, 0, 1], __ss_align: 0 } }

I am using an M1 macbook pro with sonoma if that matters

eaufavor commented 1 day ago

addr: SockaddrStorage { ss: sockaddr_storage { ss_len: 16, ss_family: 2, ss_pad1: [182, 62, 0, 0, 0, 0], ss_align: 0 } }, peer: SockaddrStorage { ss: sockaddr_storage { ss_len: 16, ss_family: 2, ss_pad1: [182, 62, 127, 0, 0, 1], ss_align: 0 } }

I think what happened is that you use 0.0.0.0 as the address of the upstream. Is that the case here @JosiahParry ?

0.0.0.0 maps to 127.0.0.1 in many OSes. Hence the mismatch in our check.

I still need to figure out what happened to the original bug report since the debug message from SockaddrStorage is not as helpful.

JosiahParry commented 1 day ago

@eaufavor that is exactly what it was! Great catch thank you so much!

To confirm: my local apps are spawning at 127.0.0.1 and i was creating the backend with Backend::new(&format!("0.0.0.0:{port}"))

switching to 127.0.0.1:{port} got rid of the messages

futurist commented 17 hours ago

PS: the reported issue case also use 0.0.0.0 as backend host ip.