Blockers for systemd?

[fpqc]

For various reasons, it isn't obvious to me exactly how to determine what features are missing from WSL's implementation in order to get something like systemd up and running. From the documentation, we know for sure that WSL does not yet implement any of the syscalls related to cgroups, and doesn't include any of the kernel namespaces, which is probably a blocker at the moment. Are there any other missing syscalls or socket options that would block systemd from running?

I also am attaching the output of strace systemd --test

systemd-trace.txt

(using the docker image of Ubuntu yakkety installed with RoliSoft's python script)

Also, I know that WSL currently uses a kind of hardcoded init process (which sets up the network interfaces, remakes the /etc/hosts file, does some other stuff, then (I think) forks /bin/bash by default). Would the whole initialization process need to be rewritten from scratch to use systemd as the init, assuming all of the other dependencies were eventually implemented (cgroups, aforementioned missing syscalls and socket options)?

[iz0eyj]

Something in this old thread ( @jackchammons ), but it seems that WSL (and all the Win10 kernel design) is a Top Secret. :) https://blogs.msdn.microsoft.com/wsl/2016/06/08/wsl-system-calls/

[rodrymbo]

Considering how open the Devs have been hereabouts, I'd say the issue isn't so much "keeping it secret" as "it's a moving target".

I don't think it will take too long for the Devs to figure out that what they are presenting to us is "single-user, console". To a Linux person it is like Safe Mode is to a Windows person. Yes you can do stuff in Safe Mode, and sometimes you are amazed that it even works at all, but it's not the same as running Windows, just as running WSL without cron or syslog or sshd or a terminal emulator (what we have got is sort of a console emulator) and other servers/daemons/background tasks, is not the same as running an Ubuntu developer machine.

I can be happy without upstart/init/systemd running everything that comes with a fresh Ubuntu install. But the things I work on generally do logging to syslog, and don't have an option for logging to a file; so trying to work on them without rsyslogd is quite difficult. It's nice to be able to run a bash script from a shell prompt, but what would be nicer is being able to run a few from cron, to automate things. And so forth.

So let's see if we can figure out what is blocking systemd, but I'd guess that when the Devs decide it's something they need to implement, they'll figure out a way.

[fpqc]

@rodrymbo I'm just interested for my own curiosity. Like, is the main blocker just cgroups, or is it something more difficult? Like, is WSL architected in such a way that its init process needs to be a special one generated by MS, or did they just make a stripped-down init process because they haven't implemented cgroups and didn't want to bother supporting upstart when they knew it was already obsolete?

Basically, I want to compare against this: Gentoo Kernel Options for Systemd

Aside from cgroups, the requirements don't seem that crazy to emulate. I don't see the specific syscalls mentioned in that guide listed in the list of supported syscalls for WSL, but on the face of it, you need cgroups, "open by fhandle syscalls", eventpoll syscall, signalfd syscall, and timerfd syscall. The rest of the requirements appear to be things that are at various stages of implementation.

P.S. Mercifully, it looks like the Redhat devs have given up for now on pushing things like dbus into the kernel for the forseeable future (there was a big blowout on the kernel.org mailing list over kdbus, and kdbus was dropped from Fedora Rawhide a bit later).

[iz0eyj]

@fpqc init, upstart etc etc requires a continuous operation; in this moment WSL exists only when calling bash ... perhaps in the future will be different ... (?) I think now you could only simulate an init or an upstart, but does not implement them really (emulate them) for system calls I think that certain things cannot be implemented (emulated) because WSL is not a standalone operating system, but I see no reason why they cannot be simulated at least providing a dummy response. I think the reason is only the young age of WSL, just a few months, but at the end we will have a very close to the "true" Linux.

Sorry for my bad English.

[fpqc]

@iz0eyj That's not true, do $ ps -A

You will see that the init daemon is running at pid 1.

[rodrymbo]

@fpqc - The Microsoft init (/init, rather than /sbin/init or whatever) is definitely proprietary. It and any other parts of the WSL environment are probably written to meet the bare minimum for whatever requirements they thought would be needed. If the Devs decided they needed cgroups, it will be there, and if they didn't, it probably won't. :P

Not saying you shouldn't be curious, or that trying to figure it out based on clues might not provide an interesting way to pass time.

It looks to me as though they could implement most of a whole linux userland system if they wanted, and much more of the rest of the system, by which I mean to a user everything would work as expected including services. So the issue will likely be how much of that can users like us convince them (in Uservoice or whatever) is worthwhile.

@iz0eyj - right now, most of what you describe is happening: when Bash.exe runs, init is running (apparently launched or managed from svchost.exe or lxssmanager or something). No, it doesn't start when Windows starts (yet) but neither does each Bash.exe console start another init. And yes, it does stop (and tear down the environment) when the last Bash.exe is closed; but until that last Bash.exe closes, init is there, running along in the background. From my perspective, the tearing down of the linux environment is just a matter of the Devs requiring that the system "shutdown -h now" when the last Bash.exe closes. So in a sense, init is running continuously ... when the environment is running.

I agree that for many things it would be good enough for init (or whatever accepts system calls) to return dummy responses that satisfy whatever we are trying to run. For example, ifconfig could return what appear to be two devices, a loopback device and an ethernet device -- even though they don't really exist. (Or maybe there is a pseudo-device that would work without pretending to be a hardware adapter.) As long as networking works, I won't be too picky. I suppose someone will need a VPN, or more control over a tun/tap, and maybe that can come in the future.

@fpqc - For some things it is helpful to look in Process Explorer to see that init is running under a svchost.exe. What you see from ps -A (or ps -Af) could just be a different init in each different bash.exe console, but checking Process Explorer shows that it's the same one (or at least it appears to be) and it is running independently of Bash.exe (aside from being shutdown when the last Bash.exe exits).

[rodrymbo]

@fpqc - is there a reason for implementing systemd in WSL other than bragging rights? It looks to me like that would bring along a lot of baggage we might not need. Could we go to something closer to sysvinit or whatever for starting a few services, or would having systemd implemented give us a lot of benefits I'm not seeing?

[cerebrate]

For myself, I would like to be able to run systemd not to replace the WSL init, but because being able to start a pseudo-init with systemd --system --unit=foo.target would be a convenient way to start and subsequently manage a set of services/daemons required for a particular development, etc., configuration. By having multiple .target configuration files, that way systemd could be used to set up whatever service-set happened to be needed for a given session in a suitably Linuxy way.

(And, I suppose, those who want services running all the time could put bash.exe -c /bin/systemd into a startup shortcut, but that's just gravy compared to the convenient start-and-manage service functionality that'd come out of it.)

((I added a uservoice request for this here, BTW: https://wpdev.uservoice.com/forums/266908-command-prompt-console-bash-on-ubuntu-on-windo/suggestions/16571479-support-for-running-systemd .))

[schmitch]

Well this future would be great to use PostgreSQL or MySQL in Xenial instead of the Windows Installer. Which would be really really huge, especially if systemd would be invoked on Windows startup and not on bash Startup.

The creators update did many things right, but this would be extremly huge as well. The best thing would be that Windows would be clean and the Development environment can be trashed and cleaned really really easy.

[carwyn]

If the desire is to manage a set of services under WSL in a more "linuxy" way it may be worth generalizing this discussion to look at other light weight approaches of doing this. SysV init is one option but there are others that have been developed for more constrained environments that may be of interest for example:

• https://wiki.gentoo.org/wiki/OpenRC - also used in Alpine Linux (as used by Docker for Windows and OS X)
• https://wiki.openwrt.org/doc/techref/procd - as used in OpenWRT
• All the other options discussed here https://devuan.org/os/init-freedom/

I'd hazard a guess that systemd may be one of the trickier ones to implement let alone track.

I suspect the main motivation for WSL is twofold: for developer tools and for running containers. In both cases they tend to be single thing or chain/group of things per invocation.

[fpqc]

@carwyn All of the distro devs with whom MS has announced support (Ubuntu, Redhat, and OpenSuse) are now firmly in the Systemd camp, and all of their package managers and packages assume systemd is running. MS does not want to maintain its own distribution of Linux userland for WSL. Presumably if they do make a switch over to systemd from WSL-init, there should be nothing stopping you from writing shellscripts to wrap the interop daemon to work with sysVinit scripts.

[carwyn]

@fpqc granted, although there is also a MS relationship with Docker although that seems more about container management rather than what's in them. Having said that Docker seem to be sniffing around Unikernels a lot.

[bugparty]

one of Ubuntu official develops says M$s intent is not a complete linux, its just something like chroot.

[seadonfrank]

You may want to try using Upstart to run and manage third-party services Also, Follow me on linkedIn , I am going to publish an article very soon on WSL and running opensource project with WSL. Hopefully, it should give you a comprehensive guide while you start working with WSL

[therealkenc]

@kenshen112 writes in another issue:

...all .service files in wsl are by default located in the /lib folder mentioned above whilst they should be in the /etc equivalent. ... ...If the wsl team could move the file locations and fix permissions we might see systemd start.

The locations of those files are controlled by Canonical via the Debian systemd Maintainers via Freedesktop.org, not WSL. If you believe moving the file locations will help you could create a launchpad PPA for systemd.

My /etc/systemd/system looks alright enough on WSL. Which it does because it is just the stock Ubuntu image and systemd package. Could be missing something, but even if I am, that isn't the end of your voyage. Getting closer though (for some relative value of 'close`).

[weirdbeardgame]

@therealkenc as far as I understand all .service files should be in /etc/systemed which most aren't and that's what causes the error as far as I can see if i execute the command in my post copying the files from there previous location to /etc/ it works aside from the permission's issue. if that's how conical has it by default though then I wonder if there's a symlink missing somewhere???

[therealkenc]

I haven't looked into systemd for a few months, and I am not sure why the .service entries you think might be missing aren't there. This is what mine looks like (unmodified) for what it is worth. But missing entries shouldn't cause systemd to fail (as far as it goes); it would just mean less services start. What methodology are you using to start systemd, if any? You are running systemctl status in #2787, but that is putting the horse before the cart because systemd isn't running per #2209 #1579.

[weirdbeardgame]

so first off https://askubuntu.com/questions/894419/systemd-on-ubuntu-16-04-gives-no-such-interface-error one small example of the same problem / fix occurring in ubuntu 16.04 after I make the change my etc folder looks like https://pastebin.com/u3UkEeeH but now there's permission issues as previously stated.

sysctrl is a tool that allows one to create and manage service files used by systemd - the reason I said it could help us https://www.digitalocean.com/community/tutorials/how-to-use-systemctl-to-manage-systemd-services-and-units it manages the stopping and starting of systemd services a key piece of what systemd does as a whole so whilst it might not fully fix systemd it'll give us some of the functionality of it at least for sysadmin's that's incredibly useful.

[fpqc]

@kenshen112 no. Only changes or custom systemd unitfiles go into /etc/systemd.

default unitfiles belong in /usr/lib/systemd/system

Systemd provides tooling to build and store diffs of default unitfiles in your /etc/systemd/system as well as completely custom unitfiles.

[akostadinov]

It makes no sense to go back to the old *sh init files. SystemD IMO provides a superios way to manage services and all modern services in a linux distro have a service file ready. Why would one do the effort to write init files?

[fpqc]

@akostadinov because right now there is no WSL systemd support.

[weirdbeardgame]

And the systemd directories aren't read / respected correctly in WSL currently.

[fpqc]

@kenshen112 yes, because systemd is not launched. Right now WSL uses MS's custom init daemon, which also sets up the whole interop and tty piping infrastructure

[weirdbeardgame]

@fpqc I'm aware that systemd isn't running but that's not the issue. The issue is that the folder's / files exist but arent read as I mentioned above which prevents systemd from starting to some extent.

[fpqc]

@kenshen112 Systemd doesn't start because it needs to run as pid1 and requires some kernel surface that isn't yet implemented. MS would need to make some significant changes to the WSL environment initialization to support systemd at pid1 even when they do support all required kernel features. MS's PID1 opens a special bus and socket that connects the Windows console instance to a tty device.

Here is an in-depth explanation of the current architecture.

https://blogs.msdn.microsoft.com/wsl/2016/10/19/windows-and-ubuntu-interoperability/

I suggest also watching the accompanying video

[akostadinov]

This is a valid issue. The proposed workaround - to use init shell scripts is inadequate. Regardless of how hard it is to enable this. Otherwise for many use cases current implementation is unsuitable. If this is the intention of MS - to allow only some particular use cases - that's also fine. There is an alternative to run real linux which makes more sense for most production use cases anyway.

[cerebrate]

It occurs to me that running systemd as the default option has a conflict with the desire to run existing distributions unchanged, inasmuch as the systemd configuration that comes with them will attempt to start a bunch of services that either can't run under WSL (spewing errors) or which generate undesired side-effects.

Since Microsoft don't want to maintain their own userland distribution and nobody wants the first post-install WSL task to be cleaning up your systemd config, this is going to be an issue unless it's optional or manual.

[akostadinov]

Since distros are providing their own build for WSL afaik the above argument sounds moot to me. It is like running systemd as pid 1 in docker/kubernetes. And it is not necessary to be by default. In containers one chooses which process to launch as pid 1.

[cerebrate]

I was under the impression that that wasn't the case; certainly in current WSL distros systemd isn't configured in a WSL-specific/friendly way.

And sure, it doesn't have to be the default, in which case this isn't a problem. I'm just addressing the proposal of replacing the current WSL init with systemd.

[akostadinov]

I assume current WSL distros do not configure systemd because it can't yet work anyway.

[sourcejedi]

@fpqc

Basically, I want to compare against this: Gentoo Kernel Options for Systemd. Aside from cgroups, the requirements don't seem that crazy to emulate.

Yeah, that's my wild speculation too. Note you can basically ignore devtmpfs as well (on the flipside, you are definitely relying on PTY support). And you don't need to support any writes to sysfs: systemd already supports running in a container, where sysfs is read-only.

writev(2, [{"Trying to run as user instance, "..., 64}, {"\n", 1}], 2) = 65

If you're trying to make any sense out of this, you probably want to look at what the full error message says! Fortunately I can guess.

$ git grep 'Trying to run as user instance, '
src/core/main.c:                log_error("Trying to run as user instance, but the system has not been booted with systemd.");
src/core/main.c:                log_error("Trying to run as user instance, but $XDG_RUNTIME_DIR is not set.");

I'm not clear on the exact requirements, but you can't run subsidiary systemd instances - like the per-user managers for user services - unless the system itself is managed by a systemd manager. The per-user managers work within the cgroup hierachy which is set up by the system manager. The system manager expects to have overall control of the cgroup hierarchy. You probably want a system manager anyway. https://www.freedesktop.org/wiki/Software/systemd/ControlGroupInterface/

IMO, if you can't manage to run systemd as PID 1, then you're struggling.

It's absolutely allowed to have processes that run before the systemd system manager, to do the initialization tasks that have been mentioned here so far. We call them initrds or container managers :-P. Specifically, your proprietary init can set everything up as PID 1, and then transmogrify into /lib/systemd/systemd using the magic of exec().

My feeling is the above would be very easy for the distro partners to see if they look at it.

However the architectural diagram is a bit misleading in this sense IMO. I think you need to refer to https://blogs.msdn.microsoft.com/commandline/2017/12/04/background-task-support-in-wsl/ as well. Since you can open multiple bash windows on the same underlying system, WSL needs to launch processes inside the Linux system. This is what makes it somewhat less clear:

Although systemd already supports containers with dynamic consoles (machinectl login), this presents a few challenges. You have to talk DBus to systemd.

Instead, I would try injecting the process complete with PTY, as WSL does currently, but have the process first talk DBus to systemd to place it in a proper scope unit before it exec()s bash. This might be as simple as just launching systemd-run --quiet --scope /bin/bash in place of bash! The point of this is to avoid the bash shells ending up in root.slice, which doesn't look good.

[sourcejedi]

I imagine MS might want to hack in a few /sys features, whereas systemd in container has so far been written to expect a read-only /sys. If the /sys mount exists and is not marked as a read-only mount, we will try to run udev, and that probably won't do anything sensible :). If nothing else it might have to get ConditionVirtualization=!wsl.

https://github.com/systemd/systemd/issues/8036#issuecomment-361223115

edit: this example seems to have been good derail-bait. speculative and not sure it's a useful approach.

[WSLUser]

Found an interesting discussion that could be relevant here: https://unix.stackexchange.com/questions/34462/why-does-linux-allow-init-bin-bash

Relevance being we eventually need to switch PID 1 from current init to systemd and change init to /bin/init or sbin/init. But interestingly enough, we actually don't have to use /(s)bin/init per article above.

Here's another article discussing PID 1: https://unix.stackexchange.com/questions/369835/replacing-pid-1-in-linux

Someone willing to nuke their WSL instance could make a switch to enable PID 1 on systemd and determine everything causing to fail. Not sure if executing via CMD will allow an decent strace of those failings that could be posted here but would be a good start.

[therealkenc]

But interestingly enough, we actually don't have to use /(s)bin/init per article above.

Per the article above, there is no GRUB in WSL because WSL has no Linux kernel.

Someone willing to nuke their WSL instance could make a switch to enable PID 1

Running systemd (or anything else) on PID 1 is entirely doable, mind. No WSL instances were harmed in the making of this motion picture.

trace of those failings that could be posted here but would be a good start.

It would not. [Digs up reason....] "If any of those ... were even marginally inside WSL's scope ... I would have opened an actionable and reproducible test case". Assuming they weren't dupes. Which, they mostly are.

[therealkenc]

Just 'sayin.

[WSLUser]

Now I have the beg the question, what in the world is wsltub?! I'm not seeing any mention of this anywhere in the Release Notes or blogs. Repro steps? or was this just a hidden cookie you discovered? What issues are left with it, if any?

[therealkenc]

what in the world is wsltub

A tub in which wsl can splash around. I'll post code if I take it to the point of being consumer friendly. This is just first light.

[fpqc]

hahaha, that's great! When I saw someone got docker running I figured this was a matter of time.

[therealkenc]

When I saw someone got docker running

I hadn't seen that. Link?

[cerebrate]

@therealkenc See the reddit post here:

https://www.reddit.com/r/bashonubuntuonwindows/comments/8cvr27/docker_is_running_natively_on_wsl/

[fpqc]

It was also posted on this board; I got a notification about it.

[gdwnldsKSC]

I'm not sure if i'm seeing it yet, but there are non-systemd userlands out there that are in active use. I hand-roll/sideload a few myself, and enjoy the flexibility that WSL has that enables these all to work. I do a lot of embedded device work so systemd would, well, just be in my way. I suppose if the WSL "systemd" could have a mode that acted like traditional init I would have no complaints, however. I don't want to include any files in my install tgz's or layouts that i'll have to remove later just to support running under systemd.

Would such a change to enable systemd as WSL's init be entirely distro-specific, or global?

Could WSL be instructed just to use the distro provided init if we're deviating from the 'provided' init ? Maybe by the distro's launcher.exe's install process when registering with WSL?

Forgive me if some of these were already addressed in this issue, but i've read through the entire thing and only saw brief mentions of leaving capability for non-systemd distros there.

[fpqc]

@garydwnldsKSC Whatever they do in the future to support systemd probably would also support a different init when you compile the launcher for sideloading. I doubt that if they enable init-replacement it will be hardcoded for systemd.

[therealkenc]

I doubt that if they enable init-replacement it will be hardcoded for systemd

Technically WSL /init isn't hardcoded right now (and never has been). WSL respects the init path specified in HKCU\Microsoft\Windows\CurrentVersion\lxss\[gid]\KernelCommandLine along the lines of GRUB. The catch is we don't know right now the minimal requirements of init to make lxssmanager (and by extension wsl.exe) happy. I was going to look into it but never really got around before they made /init static.

In any case, there is no need to worry about WSL making some kind of unilateral move to systemd. If or when systemd is supported by WSL, every other init under the sun be it OpenRC or MSFT's home grown /init will be supported too. Because systemd depends on all the things.

[weirdbeardgame]

If you guys are willing to consider an alternative option, https://www.reddit.com/r/bashonubuntuonwindows/comments/8xkko7/openrc_systemd_init_alternitve_starts_works/ OpenRC an alternative option works far better then systemd or init in the WSL it goes so far as to "boot" the kernel (or lack thereof in the WSL case) and start several key processes not the least of which is cron which after this functions absolutely perfectly as it should. udev and process requiring such however due to missing features in the WSL will not start so this alternative isn't perfect by any means and so far it seems only Ubuntu / Debian have packages which may or may not be an issue depending on how you'd implement this.

This also can start a slightly modified lxdm session that loads a desktop as it should. One final caveat is that this won't auto start by default but a clever usage of the windows task scheduler could alleviate that to some degree.

[therealkenc]

Yep the OpenRC scripts have basically worked since ever. The only thing that has changed in WSL recently is that once the daemons are started (via an OpenRC script or otherwise) they no longer die a horrible death when the last terminal is closed (since 17074).

isn't perfect by any means and so far it seems only Ubuntu / Debian have packages which may or may not be an issue

You can get a Debian-like (read: Ubuntu-like) experience with Devuan. Their whole schtick is removing the systemd goblins from Debian packages. On WSL the udev gaps can be worked-around by just configuring away those /etc/init.d/ tasks.

loads a desktop as it should

No one should load a desktop because that makes no kind of sense. But I digress.

In any case, the lxdm daemon is not specific/related to OpenRC anymore than say rsyslogd or apache2. That they run (or not as the case may be) isn't related to systemd or WSL's init. Doing a sudo openrc default does no more (and no less) than a shell script that calls a list of sudo service [some service] start from your stock WSL Ubuntu and (what you are calling) systemd.

[weirdbeardgame]

With lxdm I understand that the point I was trying to make was the "shell script" that runs the service start works well enough that it can start a display manager as well as cron (which to my understanding though I could be totally misinformed has been a tough sucker in the past with WSL) Though I am curious what bars you guys from adding something like Udev support and what bars you guys from making this the defacto standard since it appears to be far more functional then systemd - if you would even call this a replacement for that even or is that up to the operating system manufacturer? I apologize for my ignorance and am simply curious.

[therealkenc]

Though I am curious what bars

Time, money.

you guys from adding something like Udev support

n.b. I am not those guys. Actually there hasn't been a single post in this thread from those guys because this is a navel-gazing discussion thread not a bug or feature request.

from making this the defacto standard

Because in this path of the space time continuum systemd ended up being the de-facto init system on the numerically most popular Desktop Linux distributions. MSFT has intended WSL to be distribution agnostic. You could always ask Canonical to ditch systemd for OpenRC. Or as I said you could run Devuan. But people (numerically) don't want to run Devuan. They expect the systemctl blah blah blah HOWTO they blindly copied off the Interwebs to work.

appears to be far more functional then systemd

It isn't. As I said, what you are doing is no more and no less functional than calling sudo /etc/init.d/something. Whether the daemons the scripts launch work or not isn't related MSFT's init or systemd or upstart or OpenRC's init. They're scripts. Daemons where the necessary syscall surface has been implemented run with varying levels of success. Those that don't, well, they don't.

[aleks-mariusz]

A tub in which wsl can splash around. I'll post code if I take it to the point of being consumer friendly. This is just first light.

@therealkenc been a while since you posted this; any chance you have (or could make) a quick write up for what you have already that might be the less friendly version, you know, for those who like to get their hands dirty with linux and don't need things that friendly? :-)

[therealkenc]

Yes it has been a while; I got sidetracked on a "what would it take" time-sink on nmap. I'll try to get back to systemd and will follow up if there is anything worth posting (friendly or otherwise).