aviramha
commented
3 months ago Hi,
Thanks for reporting this.
192.168.65.254 is not the docker container IP right?
Closed serbanghita closed 2 months ago
aviramha
commented
3 months ago Hi,
Thanks for reporting this.
192.168.65.254 is not the docker container IP right?
serbanghita
commented
3 months ago from outside:
~ docker inspect --format '{{ .NetworkSettings.IPAddress }}' 187
10.130.0.2from inside
root@1874070f34e2:/var/www# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 65535
inet 10.130.0.2 netmask 255.255.255.0 broadcast 10.130.0.255
ether 02:42:0a:82:00:02 txqueuelen 0 (Ethernet)
RX packets 24896 bytes 6741137 (6.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 21875 bytes 15023467 (14.3 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 57144 bytes 34004876 (32.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 57144 bytes 34004876 (32.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0root@1874070f34e2:/var/www# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.130.0.1 0.0.0.0 UG 0 0 0 eth0
10.130.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0root@1874070f34e2:/var/www# ping host.docker.internal
PING host.docker.internal (192.168.65.254): 56 data bytes
64 bytes from 192.168.65.254: icmp_seq=0 ttl=63 time=0.610 ms
64 bytes from 192.168.65.254: icmp_seq=1 ttl=63 time=0.907 ms
aghajani
commented
3 months ago Hi, Thanks for reporting this.
192.168.65.254is not the docker container IP right?
No, it's the host-gateway IP (aka host.docker.internal) I am wondering why mirrord tries to re-resolve this IP address :thinking: (and the port is among filters of outgoing) although this is already an IP not a hostname
aghajani
commented
3 months ago my guess is this is happening because of mirrord trying to reverse lookup the IP first, then if it succeeds, tries to make the connection through the local container which the DNS lookup (the host-address found through reverse lookup) cannot be resolved on local container. (code section below - sockets.rs)
fn get_local_address_to_connect(address: SocketAddr) -> HookResult<SocketAddr> {
// Aviram: I think this whole function and logic is weird but I really need to get
// https://github.com/metalbear-co/mirrord/issues/2389 fixed and I don't have time to
// fully understand or refactor, and the logic is sound (if it's loopback, just connect to
// it)
if address.ip().is_loopback() {
return Ok(address);
}
let cached = REMOTE_DNS_REVERSE_MAPPING
.get(&address.ip())
.map(|entry| entry.value().clone());
let Some(hostname) = cached else {
return Ok(address);
};
let _guard = DetourGuard::new();
(hostname, address.port())
.to_socket_addrs()?
.next()
.ok_or(HookError::DNSNoName)
}If this is the case, one solution could be to skip the reverse lookup all and all if the address is matched in the outgoing->filter list (since in the end this is supposed to be local)
aviramha
commented
3 months ago This is a behavior we've seen before - we have an open issue to support bypassing DNS requests but haven't implemented it yet. This should've been resolved by https://github.com/metalbear-co/mirrord/issues/702 but seems we haven't created an open issue or decided not to (and can't find where)
aviramha
commented
3 months ago tbh an easy solution would be to resolve any IP locally, or atleast as a fallback. @Razz4780 wdyt?
Razz4780
commented
3 months ago I looked into implementation and I think the problem occurs earlier than get_local_address_to_connect. telnet first tries getaddrinfo("192.168.65.254", "9003", ...). If dns is enabled in the mirrord config, we attempt to resolve this in the cluster (which fails). We should probably pick up calls for IP addresses and apply the outgoing filter there
Razz4780
commented
3 months ago @serbanghita I prepared a custom version of mirrord CLI that hopefully resolves your problem. You can download it from the artifacts here. I'd appreciate if you could try it out and share feedback ^^ For the fix to work, your mirrord config must contain an outgoing local filter matching the address with no protocol specified (the config linked in issue description works)
serbanghita
commented
3 months ago artefact used https://github.com/metalbear-co/mirrord/actions/runs/9856256997/artifacts/1681826148
mirrord 3.108.0Tried debugging, throws a lot of errors in the console:
2024-07-09T13:23:00.021147Z ERROR ThreadId(01) mirrord_layer::error: Error occured in Layer >> IO(Custom { kind: Uncategorized, error: "failed to lookup address information: Name or service not known" })
2024-07-09T13:23:00.044214Z ERROR ThreadId(01) mirrord_layer::error: Error occured in Layer >> IO(Custom { kind: Uncategorized, error: "failed to lookup address information: Name or service not known" })Also tried this:
root@f9abd949be40:/var/www# cat /mirrord.json
{
"agent":
{
"ephemeral": true
},
"feature":
{
"network":
{
"incoming":
{
"ignore_localhost": true
},
"outgoing":
{
"ignore_localhost": true,
"filter":
{
"local":
[
":9000",
":9003"
]
}
}
},
"env":
{
"override":
{
"DEV_TOOLS_SWAP": "1"
}
}
}
}
root@f9abd949be40:/var/www# mirrord exec -t deployment/core-api -n serbanghita-core --context virta-dev --no-telemetry --steal -f /mirrord.json -- env COMPOSER_ALLOW_SUPERUSER=1 telnet 192.168.65.254 9003
! Warning: field OutgoingConfig.filter is marked as unstable. Please note API may change
When targeting multi-pod deployments, mirrord impersonates the first pod in the deployment.
Support for multi-pod impersonation requires the mirrord operator, which is part of mirrord for Teams.
You can get started with mirrord for Teams at this link: https://mirrord.dev/docs/overview/teams/
* Running binary "env" with arguments: ["COMPOSER_ALLOW_SUPERUSER=1", "telnet", "192.168.65.254", "9003"].
* mirrord will target: deployment/core-api, a configuration file was loaded from: /mirrord.json
* operator: the operator will be used if possible
* env: all environment variables will be fetched
* fs: file operations will default to read only from the remote
* incoming: incoming traffic will be stolen
* outgoing: forwarding is enabled on TCP and UDP
* dns: DNS will be resolved remotely
⠐ ! Warning: field OutgoingConfig.filter is marked as unstable. Please note API may change
✓ Running on latest (3.108.0)!
✓ ready to launch process
✓ layer extracted
✓ operator not found
✓ container created
✓ container is ready
✓ config summary Trying 192.168.65.254...
2024-07-09T13:18:59.198888Z ERROR ThreadId(01) mirrord_layer::error: Error occured in Layer >> IO(Custom { kind: Uncategorized, error: "failed to lookup address information: Name or service not known" })
telnet: Unable to connect to remote host: Input/output error
root@f9abd949be40:/var/www# then I've explicitly put the IP 192.200.65.254 in mirrord.json
"filter":
{
"local":
[
"192.200.65.254:9000",
"192.200.65.254:9003"
]
}and the result is
root@f9abd949be40:/var/www# mirrord -V
mirrord 3.108.0
root@f9abd949be40:/var/www# mirrord exec -t deployment/core-api -n serbanghita-core --context virta-dev --no-telemetry --steal -f /mirrord.json -- env COMPOSER_ALLOW_SUPERUSER=1 telnet 192.168.65.254 9003
! Warning: field OutgoingConfig.filter is marked as unstable. Please note API may change
When targeting multi-pod deployments, mirrord impersonates the first pod in the deployment.
Support for multi-pod impersonation requires the mirrord operator, which is part of mirrord for Teams.
You can get started with mirrord for Teams at this link: https://mirrord.dev/docs/overview/teams/
* Running binary "env" with arguments: ["COMPOSER_ALLOW_SUPERUSER=1", "telnet", "192.168.65.254", "9003"].
* mirrord will target: deployment/core-api, a configuration file was loaded from: /mirrord.json
* operator: the operator will be used if possible
* env: all environment variables will be fetched
* fs: file operations will default to read only from the remote
* incoming: incoming traffic will be stolen
* outgoing: forwarding is enabled on TCP and UDP
* dns: DNS will be resolved remotely
⠐ ! Warning: field OutgoingConfig.filter is marked as unstable. Please note API may change
✓ Running on latest (3.108.0)!
✓ ready to launch process
✓ layer extracted
✓ operator not found
✓ container created
✓ container is ready
✓ config summary Trying 192.168.65.254...
telnet: Unable to connect to remote host: Network is unreachable
root@f9abd949be40:/var/www# Could you run again with RUST_LOG=mirrord=trace env set and share logs?
aghajani
commented
3 months ago @Razz4780 Oh great! It works! I guess @serbanghita made a mistake and replaced his local machine mirrord with your version not the one inside the running docker container. I can verify the issue has resolved with the custom build you provided, when can we have this in a release :heart:
Razz4780
commented
3 months ago @aghajani Cool, happy it works for you! The solution I implemented is a bit hacky though and has possible bad interactions with other mirrord features. We plan to solve your problem in a bit different way, one that is more configurable - by adding a local/remote filter to the feature.network.dns config. I opened an issue to track it here. I suspect we'll be able to release it in a couple days ^^
With the new extended config, you'll be able to achieve the same result by changing your mirrord config like this:
{
"agent":
{
"ephemeral": true
},
"feature":
{
"network":
{
"incoming":
{
"ignore_localhost": true
},
"outgoing":
{
"ignore_localhost": true,
"filter":
{
"local":
[
":9000",
":9003"
]
}
},
"dns":
{
"filter":
{
"local":
[
":9000",
":9003"
]
}
}
}
}
}You're right, thanks for pointing this out, I'll retry tomorrow morning
Serban Ghita http://ro.linkedin.com/in/serbanghita http://ghita.org | http://mobiledetect.net
On Tue, 9 Jul 2024 at 21:07, Mostafa Aghajani @.***> wrote:
@Razz4780 https://github.com/Razz4780 Oh great! It works! I guess @serbanghita https://github.com/serbanghita made a mistake and replaced his local machine mirrord with your version not the one inside the running docker container. I can verify the issue has resolved with the custom build you provided, when can we have this in a release ❤️
— Reply to this email directly, view it on GitHub https://github.com/metalbear-co/mirrord/issues/2579#issuecomment-2218343147, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIOHIPRMLENNBUYFDCWX3TZLQRHLAVCNFSM6AAAAABKQOPQFCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJYGM2DGMJUG4 . You are receiving this because you were mentioned.Message ID: @.***>
serbanghita
commented
2 months ago I can confirm this works 🙇 🍻
network filter:
"network":
{
"incoming":
{
"ignore_localhost": true
},
"outgoing":
{
"ignore_localhost": true,
"filter":
{
"local":
[
":9000",
":9003"
]
}
}
},I used the binary from aarch64-unknown-linux-gnu inside the Docker instance
Razz4780
commented
2 months ago Hey @serbanghita @aghajani We've just release version 3.111.0, now you can use the new DNS filter configuration ^^
serbanghita
commented
2 months ago 🥇 Thank you, the winning config was:
{
"agent": {
"ephemeral": true
},
"feature": {
"network": {
"incoming": {
"ignore_localhost": true
},
"outgoing": {
"ignore_localhost": true,
"filter": {
"local": [
":9000",
":9003"
]
}
},
"dns": {
"enabled": true,
"filter": {
"local": [
":9000",
":9003"
]
}
}
}
}
}If feature.network.dns filter is not included I get
2024-07-22T08:48:01.095221Z ERROR ThreadId(01) mirrord_layer::error: Error occured in Layer >> IO(Custom { kind: Uncategorized, error: "failed to lookup address information: Name or service not known" })
2024-07-22T08:48:01.172902Z ERROR ThreadId(01) mirrord_layer::error: Error occured in Layer >> IO(Custom { kind: Uncategorized, error: "failed to lookup address information: Name or service not known" })`I included this for all the PHP (+xdebug) devs out there searching for this
Bug Description
Mac OS + running Docker image with mirrord inside + PHP + xdebug
this is from the Docker image cli:
Steps to Reproduce
Dockerfile- this is a PHP app, image also installs mirrordmirrord.jsonthe log contains these errors
When I request an URL and inject
XDEBUG_SESSION_START, I can see traffic but the debugger doesn't work.Enabling logs:
Backtrace
No response
Relevant Logs
No response
Your operating system and version
Mac OS
Local process
docker
Local process version
No response
Additional Info
Once the container is up, I also tried from within the Docker container:
throws
but when I changed
dns: falseinmirrord.jsonxdebug info