Closed fliespl closed 3 years ago
Okay, I figured the problem is systemd-resolved.
Curious now, if I should mask it as I did with other services or there is better / proper way?
Hm. What's systemd-resolved doing to break it, do you know? (Is your /etc/resolv.conf by chance going missing?)
Once systemd starts after running genie -s /etc/resolv.conf is being replaced by systemd-rssolved with broken dns info: nameserver 127.0.0.53
Ah, right. Yeah, that's normal for systemd-resolved; it's designed to take over DNS resolution from local clients, hence the rewriting of /etc/resolv.conf.
You can just disable it, which should restore the default without-systemd behavior. Myself, though, I prefer the systemd-resolved resolver over the default one, so prefer to resolve this by configuring it appropriately for my network. You can get fuller details on how this is supposed to work with man 8 systemd-resolved
and man 5 resolved.conf
, but the short version is that you'll need to put the DNS resolvers you want in /etc/systemd/resolved.conf.
This is what mine looks like after doing this:
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See resolved.conf(5) for details
[Resolve]
DNS=172.16.0.128 172.16.0.130
#FallbackDNS=
Domains=arkane-systems.lan
LLMNR=yes
MulticastDNS=yes
DNSSEC=allow-downgrade
DNSOverTLS=opportunistic
#Cache=yes
DNSStubListener=yes
#ReadEtcHosts=yes
But the important bits you have to set are the DNS= and Domains= lines.
Yeah, I know about that. What I meant was: how should it work under wsl2.
The problem is - it's unreliable at start, cause WSL generates it's own resolvconf by default.
So:
generateResolvConf = true
is true by default,I am just not sure if it should take over if generatedResolvConf is true.
Ah, right, gotcha.
For myself, I just turn off generateResolvConf and generateHosts, since I essentially never use WSL without genie/systemd, so it doesn't matter to me; a systemd-resolved instance is always running. I can see why you might want to disable systemd-resolved and stick with the autogenerated resolv.conf if you do sometimes use WSL in the raw, though.
...I should probably put some "things you might want to consider" on the wiki one of these days, but time continues to be unfortunately scarce.
Sorry to add to your plate :)
TBH I don't use generateResolvConf - I disable it and just create my own resolve.conf file.
I wanted to create proper playbook for both debian and ubuntu. I will just make a seperete rules for debian and ubuntu then. On debian - custom resolv.conf file and on ubuntu - suggested changes to systemd-resolved.
Thanks! Appreciate all your work here :)
That's how it should work, though. resolv.conf contains that to point DNS lookups at systemd-resolved (listening on 127.0.0.53), and then systemd-resolved does the lookup based on the nameservers you specify in resolved.conf.
My resolv.conf looks just the same.
Thank you for all your help @cerebrate. Really appreciate it!
Strange thing is that newest build + systemd-genie made DNS resolution fail when: generateResolvConf = false
and systemd-resolver is set up.
flies@flies-pc-wsl:/mnt/c/Users/flies$ ping google.com ping: google.com: Name or service not known
/etc/systemd/resolved.conf [Resolve] DNS=8.8.8.8
I have to generate: /etc/resolv.conf with
nameserver 127.0.0.53
on each boot.
@fliespl @celebrate It is expected. You should configure DNS in another file - see [Resolved service configuration] /etc/systemd/resolved.conf (https://www.freedesktop.org/software/systemd/man/systemd-resolved.service.html). Put your DNS address into DNS properties following the instruction and it will work amazing. You can test your configuration using resolvectl query. Use either your DNS proxy, DNS box or service running on the WiFi router as a 1st primary DNS and the best available DynDNS Cloud DNS service IP as the second.
@PavelSosin-320 my configuration is correct. I think you missed the point or I didn't understand you properly.
/etc/systemd/resolved.conf [Resolve] DNS=8.8.8.8
$ resolvectl query wp.pl wp.pl: 212.77.98.9 -- link: eth0
$ ping wp.pl ping: wp.pl: Name or service not known
It was working this way up till newest WSL + Genie update with such configuration generateResolvConf = false
But it started failing after updates.
After I add following - it starts resolving correctly.
nameserver 127.0.0.53 to /etc/resolv.conf
$ ping wp.pl PING wp.pl (212.77.98.9) 56(84) bytes of data. 64 bytes from www.wp.pl (212.77.98.9): icmp_seq=1 ttl=53 time=10.0 ms
P.S. Don't miss to enable and start systemd-networkd service to ensure DNS reachability. Check it using networlctl: eth0 ether routable unmanaged
@fliespl I see wp.pl via Quad9 DNS perfectly: ping wp.pl PING wp.pl (212.77.98.9) 56(84) bytes of data. 64 bytes from www.wp.pl (212.77.98.9): icmp_seq=1 ttl=53 time=85.2 ms
I use DynDNS service running in my WiFi router backed by 9.9.9.9 DNS that has an instance running on AWS infra close to me . So, I have a double DNS cache: router's OpenWRT plus systemd-resolved cache.
@PavelSosin-320 what is your windows build + genie version + contents of /etc/resolv.conf?
@fliespl I'm Windows insider, so Windows build causes me more problems than solutions. Currently, I use genie 1.39 but it works stable since 1.37. My wsl.conf contains [network] generateHosts = true <- keep it true or create blank /etc/hosts to avoid problems with NetworkManager. generateResolvConf = false
P.S. As I saw for your geolocation the best choice is either OpenDNS 1.1.1.1 (Frankfurt) or 9.9.9.9 (Warsaw, Frankfurt). GoogleDNS is too far.
@PavelSosin-320 it doesn't matter which DNS I choose ;) It's not the root cause.
Gonna dig into it.
Okay, just did a clean install.
It seems that:
resolvectl query
works correctly
but direct system calls end up trying "127.0.0.1" for dns (instead of expected 127.0.0.53 - which DNS resolver uses).
Strace: connect(7, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
Ideas?
For systemd-resolved to be configured correctly, now I engage in the absolute last resort of reading the documentation, what it really wants is for /etc/resolv.conf to be a symbolic link to /run/systemd/resolve/stub-resolv.conf . It uses the existence of this link to figure out what it ought to be doing, and if it's a file rather than a symlink to the right place, it will do things rather wrong.
See systemd-resolved(8) for more details, but that's probably the first thing to do.
@cerebrate yeap, that's what I do now after boot (Create this symlink manually).
But the problem is that it doesn't stick between boots, so a little confused, cause I didn't have to do it with previous versions (not sure if it's wsl change or systemd-genie?).
@cerebrate did you have a chance to look at this issue?
Or did I misunderstood your point and there is nothing to fix and it should be done manually on each boot (symlinking /etc/resolv.conf).
It's fine by me, cause I already have a custom service for that, but might be tricky for others.
Happy to help - I can provide any info that you might require.
Where that one is concerned...
It turns out I've actually been suffering from that one for myself for a while, hacked a quick script together to fix it until I got around to solving the actual problem, then promptly forgot about it.
Unfortunately, I have now looked at it and concluded that I have have Not A Bloody Clue as to what could be deleting the resolv.conf symlink; only the more so after finding reports of people using Debian not-under-WSL who are having the exact same problem. So, short version, I have and probably will again, but I'm not expecting success any time soon, I'm afraid. đ˘
Do you think that setting sysdig process or auditctl with highest priority (start as fast as possible) could help find what is deleting that symlink?
@cerebrate just letting you know, that none of my ideas worked :D both required custom additional modules, which (obviously) didn't work right away with WSL.
Gonna try again if I have some other idea to pin-point it.
@cerebrate did you have a chance to look at this issue?
Or did I misunderstood your point and there is nothing to fix and it should be done manually on each boot (symlinking /etc/resolv.conf).
It's fine by me, cause I already have a custom service for that, but might be tricky for others.
Happy to help - I can provide any info that you might require.
I have just solved the problem in a same way. According to systemd documentation, systemd auto-detects whether /etc/resolv.conf is a symlink to /run/systemd/resolve/stub-resolv.conf or not and proceeds accordingly, in particular it does not touch the symlink if it's valid and it seems to remove it is symlink to anything else. I have noticed that the same stub file is located in /usr/lib/systemd/resolv.conf - please make sure you aren't linking to this one - even of contents are the same; your /etc/resolv.conf symlink should point explicitly to /run/systemd/resolve/stub-resolv.conf. Then, it should not disappear between "reboots".
Well, it seems that /etc/resolv.conf symlink is disappearing that way or another by default no matter what the target is if generateResolvConf is disabled. And it actually seems that we have to recreate it on our own on each launch.
I feel that wslâs config option to disable generating resolv.conf automatically was designed to prevent wsl from touching the userâs manual dns config if he decides to use such, so it stays persistent between shutdowns. If the resolv.conf is still being deleted and needs to be reverted, the aim isnât achieved.
Anyway. I prefer to avoid using hacky custom fixes if possible and trying to stick with the features offered by systemd. Symlink to systemd stub config can be therefore easily recreated without custom scripts, but by utilizing systemd-tmpfiles.d. To achieve this, one must create a config e.g. /etc/tmpfiles.d/00-stub-resolv.conf with contents:
L /etc/resolv.conf - - - - /run/systemd/resolve/stub-resolv.conf
Then, resolv.conf symlink should be recreated by systemd itself on genie init ;) This result in dns resolving fully working via systemd-resolved as designed (let me remind that one should also put his preferred DNS servers config into /etc/systemd/resolved.conf or - if one like to keep system config files provided by default untouched - /etc/systemd/resolved.conf.d/override.conf).
Please note that if using tmpfiles.d, when genie is initialized, /etc/resolv.conf will be replaced by the symlink even if it was created withgenerateResolvConf set to true (To avoid this, change first part of that tmpfiles.d entry above from L
to L+
). However, i prefer the default behavior combined with auto generation enabled. Auto generated resolv.conf means dns resolving will be working normally when running without genie / systemd. When enabling systemd, resolv.conf will be overwritten, and resolving will be handled to systemd.
@esgie Thank you kindly for your investigation of this issue. Your solution to it will be included in genie 1.43.
@esgie
Please note that if using tmpfiles.d, when genie is initialized, /etc/resolv.conf will be replaced by the symlink even if it was created withgenerateResolvConf set to true (To avoid this, change first part of that tmpfiles.d entry above from L to L+). However, i prefer the default behavior combined with auto generation enabled. Auto generated resolv.conf means dns resolving will be working normally when running without genie / systemd. When enabling systemd, resolv.conf will be overwritten, and resolving will be handled to systemd.
Can I check which build you're on? On 21390, this doesn't appear to be the behavior: auto-generated hosts and/or resolv.conf will overwrite the genie/systemd-resolved versions even when inside the bottle/systemd has started successfully.
Well, i was sure I have double-checked it yesterday, but today I have started to have troubles with dns again and... I canât explain why, but my conclusions were exactly opposite to the reality ;) Yes, you are right. WSL resolv.conf, on auto-generation enabled, takes prime. But thatâs because AFAICS i have in addition totally %#* tmpfiles.d commands. And again itâs the opposite to what I concluded. So according to: https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html L - will NOT overwrite the target if it already exist L+ - WILL overwrite the target if it exist
So if you used my tmpfiles conf in its original form, it simply didnât work as the target existed. Changing L to L+ should work as expected.
Pobierz aplikacjÄ Outlook dla systemu iOShttps://aka.ms/o0ukef
Od: Alistair Young @.> WysĹane: Wednesday, June 9, 2021 7:50:59 PM Do: arkane-systems/genie @.> DW: Sambor Gogacz @.>; Mention @.> Temat: Re: [arkane-systems/genie] No network after installing systemd-genie until restart (#130)
Please note that if using tmpfiles.d, when genie is initialized, /etc/resolv.conf will be replaced by the symlink even if it was created withgenerateResolvConf set to true (To avoid this, change first part of that tmpfiles.d entry above from L to L+). However, i prefer the default behavior combined with auto generation enabled. Auto generated resolv.conf means dns resolving will be working normally when running without genie / systemd. When enabling systemd, resolv.conf will be overwritten, and resolving will be handled to systemd.
Can I check which build you're on? On 21390, this doesn't appear to be the behavior: auto-generated hosts and/or resolv.conf will overwrite the genie/systemd-resolved versions even when inside the bottle/systemd has started successfully.
â You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Farkane-systems%2Fgenie%2Fissues%2F130%23issuecomment-857910155&data=04%7C01%7C%7C9fc117a206de44e0e2f408d92b6f242d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637588578608715312%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=0e6GbD28Zfrz1cMC%2Fc3XzMcfETyjrL7Sf0s2TfAEZnQ%3D&reserved=0, or unsubscribehttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABUKZKE5JJT34LHR5O7CTI3TR6SYHANCNFSM4ZC4D6FA&data=04%7C01%7C%7C9fc117a206de44e0e2f408d92b6f242d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637588578608715312%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=VESE0mogGC3aQIxcfJBHKsLU2ZsLqEDhimuYIGKOpFw%3D&reserved=0.
Perhaps not everyone wants to use systemd-resolved
, though. What would I do if I want to use something else, like unbound?
After installing genie, delete the /usr/lib/genie-stub-resolv.conf file to stop it from replacing resolv.conf with this link.
(I'd have included an option if genie did the job itself, but it doesn't, so. This is just optimizing for the default case, at the moment, figuring that people interested in replacing their resolver with a non-default cause can figure out what to do most easily.)
According to https://www.freedesktop.org/software/systemd/man/systemd-resolved.service.html there are in fact four modes of how systemd-resolved can handle /etc/resolv.conf. You can read how they work in the linked documentation, althought in practice they differ only with the target of /etc/resolv.conf link (mode names arenât defined throughout the document so they are my own proposals). 1) âstubâ - /run/systemd/resolve/stub-resolv.conf 2) âstub-staticâ - /usr/lib/systemd/resolv.conf 3) âlegacyâ - /run/systemd/resolve/resolv.conf 4) âunmanagedâ - in this mode /etc/resolv.conf is handled by âother packagesâ so /etc/resolv.conf isnât touched in any way.
I wonder if it is possible to implement such a config option in genie.ini so genie will handle symlinking of /etc/resolv.conf according to user needs.
I would say âunmanagedâ should be the default mode after all. The problem of current default behavior (âstubâ) is that it requires to configure DNS server in /etc/systemd/resolved.conf first, otherwise networking will not work and this is not obvious and may be discouraging for users.
Pobierz aplikacjÄ Outlook dla systemu iOShttps://aka.ms/o0ukef
Od: Alistair Young @.> WysĹane: Wednesday, August 4, 2021 1:24:23 AM Do: arkane-systems/genie @.> DW: Sambor Gogacz @.>; Mention @.> Temat: Re: [arkane-systems/genie] No network after installing systemd-genie until restart (#130)
After installing genie, delete the /usr/lib/genie-stub-resolv.conf file to stop it from replacing resolv.conf with this link.
(I'd have included an option if genie did the job itself, but it doesn't, so. This is just optimizing for the default case, at the moment, figuring that people interested in replacing their resolver with a non-default cause can figure out what to do most easily.)
â You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Farkane-systems%2Fgenie%2Fissues%2F130%23issuecomment-892230891&data=04%7C01%7C%7Ccda9f54cabb24f7cd1c108d956d5d407%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637636298644935799%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=DrKQzvIEUTt%2BZAwVLyykmXPpNna3pwXe5Waf%2FcQZ0z8%3D&reserved=0, or unsubscribehttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABUKZKCDJC3QZALHAKZG4IDT3B3CPANCNFSM4ZC4D6FA&data=04%7C01%7C%7Ccda9f54cabb24f7cd1c108d956d5d407%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637636298644935799%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=kfAPV4VP8IUEh%2FH6h1QdyyTJG4YYTsW0H2d8GJqWcTM%3D&reserved=0.
@xuanruiqi Working with systemd-based distro people have no choice to use systemd-resolved because the entire networking stack is based on the pair of networked and resolved services started automatically as the network.target and serve everybody via D-Bus socket interface. If something is started via systemd unit and needs networking it would be better to add Requires=network.target and After=network-ready target to the unit to avoid false start. This is the "systemd way" to avoid described problem. Resolved itself works 10 times better than WSL's "generate resolved.conf".
Well, I use Arch Linux, and it's not really necessary to use systemd-resolved. For instance I prefer to use unbound, and it's fully supported by Arch out of the box.
@xuanruiqi According to ArchLinux networking Arch Linux has crossed the front lines and plays now for systemd team. I expect that all utilities and libraries for Arch Linux use systemd networking today, including ping. P.S. The simple way to restore networking is systemctl daemon-reexec - it will restart systemd networking. Then check systemctl status to see if the system is running.
Using systemd (and systemd networking stack) doesn't mean all functionality have to be used. systemd-resolved isn't enabled by default on Arch, and also isn't really what most Arch users use. Many Arch users prefer unbound, for example. Arch provides out-of-box support for many DNS resolvers with systemd integration, including of course systemd-resolved but also many other resolvers.
That said, I agree with @esgie - "stub" mode shouldn't be the default as many users would rather not use systemd-resolved. It might be a good idea to make this configurable in genie.ini.
@xuanruiqi The straightforward way to disable services is systemctl [--user]disable. "Jus-do-It" It's the persistent configuration and will be preserved for subsequent runs. I don't see any scenario when genie -i is explicitly or implicitly executed at least once and resolved can be disabled. I hope that systemd is able to process dependencies in this case correctly. The proposed mechanism may contradict the way how systemd works. If some unit requires/waits for networking target and this target will be reached partially due to invisible to systemd reasons it can break the entire unit chain. The resolve. conf already allows to choose how /etc/resolve.conf is used. So, it will be not only redundant functionality but also confusing documentation. Shortly, I don't think that genie can alter the normal systemd functionality.
You seem to be missing my point. Many DNS resolvers have full systemd integration and can drop-in replace resolved; that's why Arch (and perhaps other dists) provide them as replacement for the default resolver. It will handle the networking target correctly, so there's no need to "alter the normal systemd functionality", because it is supposed to integrate with normal systemd functionality.
That said, the only difference from resolved is that it doesn't use the stub in /run/systemd/resolve/stub-resolv.conf, and apparently resolved is the only resolver that uses it. So I argue that a symlink shouldn't be the default, or at least configurable.
Does DnsStubListener=no in the resolved.conf solves it? If in /etc/wsl.conf "generate resolve.conf" is false you can use any resolver. Provided by MS resolve.conf generation mechanism conflicts with all resolvers because it ignores all DNS features and gets ISP-provided DNS with no features. It was OK until Win 11 because Windows DNS client was very primitive. After upgrade to Win 11 WSL built-in name resolution stopped working. Without Protocols: LLMNR=resolve -mDNS +DNSOverTLS DNSSEC=no/unsupported Linux resolvers can't communicate with DNS.
On Thu, Aug 5, 2021 at 12:21 PM Xuanrui Qi @.***> wrote:
You seem to be missing my point. Many DNS resolvers have full systemd integration and can drop-in replace resolved; that's why Arch (and perhaps other dists) provide them as replacement for the default resolver. It will handle the networking target correctly, so there's no need to "alter the normal systemd functionality", because it is supposed to integrate with normal systemd functionality.
That said, the only difference from resolved is that it doesn't use the stub in /run/systemd/resolve/stub-resolv.conf, and apparently resolved is the only resolver that uses it. So I argue that a symlink shouldn't be the default, or at least configurable.
â You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/arkane-systems/genie/issues/130#issuecomment-893304996, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM7YVKARSM4NQWXGX7UZLXDT3JJ2PANCNFSM4ZC4D6FA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .
Okay, let's kick this discussion over to a new issue instead of repurposing this one. Now at #187.
The resolver package comes with Distro installation directly from the distro repository. It is installed with its units spread over all directories where systemd searches unit files. I suppose that in 99% of cases it is initially enabled but can be disabled using systemctl disable unit-name. Certain unit attributes can be changed using systemctl edit unit-name. It doesn't change the original unit but creates the new drop-in file. It content is merged with the original unit next time when systemd reexec the daemon when the machine is rebooted, systemd daemon is reexecuted or genie -i runs. These are the precise boundaries that systemd draws for administrators. Since systemd package is installed with the distro it can't be changed using the outer config file - it is not read by systemd. Once systemd --system or systemd --user is fired it works according to its own highly standardized rules developed by Redhat. Although systemd.io is an independent company today the developers in this company are RedHat, i.e. IBM employers. The endorsement of genie by systemd.io last year is an enormous achievement because it opens the doors for genie into the enterprise software world. RedHat is not only the main Linux provider but also the main security and IT automation provider in this world. The genie only fires systemd. The Feature request for management systemd using outer file files is de-facto systemd feature request. If something can be done using systemctl, systemd-run, etc., and unit files it shall be done using existing tools, otherwise the doors can close.
The endorsement of genie by systemd.io last year
Out of curiosity, is this online somewhere? I managed to miss it entirely...
Using ubuntu 20.04 WSL2.
After initial install my networking goes down. I need to wsl --shutdown and then it works correctly. Any idea what can I do to fix it during deploying systemd-genie?