pop-os / pop

A project for managing all Pop!_OS sources
https://system76.com/pop
2.47k stars 87 forks source link

Boot freezes on: A start job is running for dev-mapper-cryptswap #316

Closed spysiuk closed 4 years ago

spysiuk commented 6 years ago

Distribution: NAME="Pop!_OS" VERSION="18.04 LTS" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Pop!_OS 18.04 LTS" VERSION_ID="18.04" HOME_URL="https://system76.com/pop" SUPPORT_URL="http://support.system76.com" BUG_REPORT_URL="https://github.com/pop-os/pop/issues" PRIVACY_POLICY_URL="https://system76.com/privacy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic

Issue/Bug Description: After recent update boot freezes on A start job is running for dev-mapper-cryptswap ... / no limit. Until I press any key timer is increasing.

Expected behavior: Boot without user intervention.

Other Notes: I tried to comment swap in /etc/fstab and /etc/crypttab but after that I didn't get login screen - had to go to other terminal, remove comments in both files and reboot machine.

lars-str commented 5 years ago

Same bug for me. ASUS TP200S. Fresh install of version 19.04 and during first boot after installation the bug comes up... After several tries booting up the machine, the computer booted. But it seems that the bug is coming back randomly.. I've added now the nofail option, so there is now a limit of 1m30s..

TeamLinux01 commented 5 years ago

I also ran into this problem on my Galago Pro (galp2). I did a fresh install of Pop!_OS 19.04 intel/AMD v5 from a Samsung USB 3.0 flash drive, choosing to encrypt my drive. Then I ran some custom setup scripts from my github repo, https://github.com/TeamLinux01/scripts pop_os_setup.sh I also use an external WD 4TB USB 3.0 HDD, which is formatted Ext4.

Update: Reinstalled again and it came up with "A start job is running for /dev/mapper/cryptswap (** / no limit) on the first reboot.

tobico commented 5 years ago

I too am having this issue with a fresh install of Pop!_OS 19.04 running under Parallels on Mac. I did not enable encryption during install.

konadave commented 5 years ago

I've been experiencing this intermittently today. Fresh install of Pop!_OS 19.04 on a System76 Gazelle (gaze-12) on Monday. First saw the problem today, and have no idea what the issue is.

What I did find though that some might find helpful, is hitting Ctrl+Alt+Del caused it to bypass/skip the start job and it seemed to boot normally.

sunnyyssc commented 5 years ago

I'm having the same issue on VM. I can boot into 18.10 and 18.04 just fine expect of 19.04. It will boot sometimes after restarting the VM several times. I also didn't enable encryption when installing. The suggestion from @konadave does not work for me, maybe it is because I'm running on VM...

ghost commented 5 years ago

@TrendMend We haven't able to replicate the issue (despite doing dozens of installs per day in the lab), so there's no way to determine the cause.

I can add "mee too" to this thread. Acer Nitro 5 with M.2 SSD Intel Fresh Pop!_os 19.04 updated or not. I see a message "a start job is running for cryptswap" every time I turn on and it will be off each time I press ctrl+alt+del.

TopSwagCode commented 5 years ago

I'll add a "mee too" :) Fresh install on virtual box.

wshaddix commented 5 years ago

Same issue for me as well on a fresh bare metal laptop install. I hit "CTRL+ALT+DEL" as many times as it takes (usually no more than 3) to get it to boot. The message that it's hung up on is "A start job is running for /dev/mapper/cryptswap"

dubl3a commented 5 years ago

I was forced to boot into recovery mode and comment out the cryptswap entry in both fstab and crypttab respectively. This allows me to boot without issue.

As I have 32GB of ram, do not utilize sleep functions, do I need cryptswap at all? Is this the proper way to disable it or should I do more?

This is on a bare metal install with 4 disks using LUKS.

mmstick commented 5 years ago

You're free to remove swap if you want, or disable encryption. However, note that disabling swap encryption would allow a malicious actor to read any sensitive content that was written to the swap partition.

autonomousit commented 5 years ago

@mmstick - Just had this happen while setting up a brand new Darter Pro with Pop OS 19.04 when it reboot after the initial encryption.

CTRL+ALT+DEL allowed it to boot but before that it was hung indefinitely. The setup process continued and reboots after completing the user creation process where fine.

A few minutes later after installing some updates to Pop OS I rebooted (as requested by the update) and it occurred again. Since then both reboots and cold boots after a shutdown have been fine.

Let me know what info you want as there seems to be multiple attempts/ideas in this thread and it's unclear which are/aren't helpful

jazari-akuna commented 5 years ago

Fresh install and upgrade gets me the same issue on first reboot... Pop os 19.04 kernel 5.0.0-15 4.3 GB swap, 16GB ram, Intel 7300u, nvme ssd Lenovo x270 This is really bad, I would like a stable OS... I hoped this OS could be used on a large computer park... If you need any info to fix this do not hesitate to contact me.

jpaveg commented 5 years ago

Same here, PopOS 19.04, fresh install on a laptop.

Distribution: Pop!OS 19.04 Hardware: Lenovo Thinkpad T540p

Disk Encryption: Disabled

Issue/Bug Description: Frequent issues with shutting down & booting, hangs until I force the shutdown, or CTRL-ALT-DELETE and bypass the boot hang issue.

This is on a fresh install.

Expected behavior: Boot & shutdown without user intervention.

Other Notes: Upon hitting ESC while waiting to boot, I noticed the line the OP reported: "A start job is running for /dev/mapper/cryptswap (time here / no limit)

Please let me know what files / output I can attach to my post to help out.

Lunarequest commented 5 years ago

hey i am facing similar problem on a system install. it is a minor in convince Distributor ID: Ubuntu Description: Pop!_OS 19.04 Release: 19.04 Code name: disco attached is my log file system.log

sedmicha commented 5 years ago

Same issue with a fresh Install of Pop OS 19.04 (Intel) on my Acer Aspire 5, UEFI was used and no disk encryption. Reboots seem fine. It fixes itself after a few reboots after getting stuck but is very annoying. If there's any information I could provide or help in any way I'll be more than happy to.

Lunarequest commented 5 years ago

same here UEFI mode, maybe it has something to with the use of UEFI? here is the output of systemd-analyze critical-chain The time after the unit is active or started is printed after the "@" character. The time the unit takes to start is printed after the "+" character.

graphical.target @1min 56.115s └─multi-user.target @1min 56.115s └─hddtemp.service @1min 41.273s +125ms └─network-online.target @1min 41.272s └─NetworkManager-wait-online.service @1min 37.213s +4.058s └─NetworkManager.service @1min 33.085s +4.126s └─dbus.service @1min 32.778s └─basic.target @1min 32.775s └─sockets.target @1min 32.775s └─snapd.socket @1min 32.562s +212ms └─sysinit.target @1min 32.473s └─systemd-timesyncd.service @1min 32.130s +343ms └─systemd-tmpfiles-setup.service @1min 31.449s +615ms └─local-fs.target @1min 31.444s └─run-user-120.mount @1min 39.474s └─local-fs-pre.target @1min 31.320s └─lvm2-monitor.service @1min 31.309s +10ms └─lvm2-lvmetad.socket @1.296s └─system.slice @1.261s └─-.slice @1.261s

4SeasonsFlow commented 5 years ago

I have just experienced this problem twice on fresh install of PopOS (boot remained at /dev/mapper/cryptswap for over 30 min first time before I gave up and rebooted). On a Asus Vivobook e203na with celeron n3350, 4gb ram, 64gb emmc storage. After 1 or 2 reboots with ctrl alt del the system boots up and works fine. If I try another fresh install will it go away or do I just have to deal with it. BTW complete noob to Linux.

Lunarequest commented 5 years ago

4SeasonsFlow are you running pop os in UEFI mode?

4SeasonsFlow commented 5 years ago

4SeasonsFlow are you running pop os in UEFI mode?

I think so but am not sure so I'll show you how I tried to check. I followed the 2nd Method of this link and below is what the terminal said. The Kingston drive is what i used to try and then install pop os while the sandisk was just there for files. Also, the problem has not reappeared since the situation I mentioned in my previous post.

~$ sudo efibootmgr BootCurrent: 0004 Timeout: 1 seconds BootOrder: 0004,0005,0003,0006,0007,0001,0002,0008 Boot0001 UEFI:CD/DVD Drive Boot0002 UEFI:Removable Device Boot0003 UEFI: SanDisk, Partition 1 Boot0004 Pop!_OS 19.04 Boot0005 UEFI OS Boot0006 UEFI: KingstonDataTraveler 3.0PMAP, Partition 1 Boot0007 UEFI: KingstonDataTraveler 3.0PMAP, Partition 1 Boot0008 UEFI:Network Device

kenklak commented 5 years ago

I can add to the chorus. This happens creating VMs using the Intel/AMD 19.04 ISO in both Windows and Linux, using VMWare Workstation 15 and VirtualBox, and whether you use encryption or not. And the message doesn't timeout. It simple doesn't allow you to go any farther.

I do have Pop on bare metal with no issue. Great OS, especially for us nVidia card folks.

Note: To get around the issue on boot, enter the recovery kernel and choose the option to get a root terminal. Edit the /etc/fstab and not only add the nofail parameter, but also a timeout parameter ( I chose 60 seconds, but that is up to you) as nofail never comes into play as the command doesn't timeout ( I waited a full 10 minutes and counting ):

/dev/mapper/cryptswap none swap nofail,x-systemd.device-timeout=60 0 0

4SeasonsFlow commented 5 years ago

Can someone help? Having trouble using nofail method. I booted in recovery and entered

/dev/mapper/cryptswap none swap nofail,x-systemd.device-timeout=60 0 0

into the terminal. I just get told that no such file or directory exists.

kenklak commented 5 years ago

Can someone help? Having trouble using nofail method. I booted in recovery and entered

/dev/mapper/cryptswap none swap nofail,x-systemd.device-timeout=60 0 0

into the terminal. I just get told that no such file or directory exists.

Sorry, this isn't a terminal command. If you have entered the options to get to the root terminal prompt, you need to edit the /etc/fstab file (e.g. vi /etc/fstab) and look for a line like the one above. Replace the 'default' parameter with the 'nofail,x-systemd.device-timeout=60' (no quotes - they are just there to help frame the text). Then, save the changes.

4SeasonsFlow commented 5 years ago

If you have entered the options to get to the root terminal prompt

How do I access these options? Is it settings, nautilus or somewhere else? Sorry for the trouble.

zarino commented 5 years ago

I had this same issue too, starting a few days ago.

Pop!_OS 19.04 i5-8400 RX Vega 56 1 TB M.3 SSD (unencrypted) Gnome 3.32.2

Machine would just hang on a black screen at startup maybe 3 out of 4 times. Eventually I found out I could display the console during that blank startup screen, and saw the dreaded:

[ * * *    ] A start job is running for /dev/mapper/cryptswap (51s / no limit)

I left it to run for 10 minutes, then found this github ticket, forced a restart, was lucky enough to get straight in, and updated my /etc/fstab as @kenklak suggested:

/dev/mapper/cryptswap none swap nofail,x-systemd.device-timeout=60 0 0

I also took the opportunity to allow Esc to display the console during startup, by editing my /etc/default/grub file and then running update-grub:

GRUB_CMDLINE_LINUX_DEFAULT="splash"

I’ve restarted about a dozen times since then, and I’ve only seen the cryptswap message once, for about 20 seconds. Not sure whether the nofail,x-systemd.device-timeout=60 change is to thank for that, or whether it’s just luck.

Lunarequest commented 5 years ago

go to boot use boot override and select your hard drive to boot into options

GiorgioAresu commented 5 years ago

I'm still having this issue on my Lenovo T430, UEFI install with encryption. There is really no definitive solution?

kenklak commented 5 years ago

I don't know of any. Strange thing is that I was only having this problem on Pop_OS VMs and yesterday, I had it once on my Pop_OS host. Rebooted, and it didn't happen again (and I made the /etc/fstab change just in case). Pop_OS comments that they haven't seen this, but clearly many people are. Very strange error given the randomness of its occurance.

mmstick commented 5 years ago

@kenklak We've seen it happen once or twice in the last year on a laptop, but restarting fixed it and it could not be replicated again, so it is in the rare "Heisenbug" category. It does appear to be non-QEMU VMs that experience the issue the most. System76 hardware appears to be largely unaffected.

kenklak commented 5 years ago

I see. Good to know and thanks for introducing me to QEMU lol. I didn't know about it until now.

ryan-fox commented 5 years ago

This issue happens to me frequently. I'm running Pop_OS on bare metal on System76 hardware.

I originally installed 18.10 (cosmic) before upgrading to 19.04 (disco) via the software update (not a fresh install). I can't remember if I experienced this issue before the upgrade or not. To be honest, I've been working around the issue for a long time.

I will see the following if I hit a key after successfully putting in my passphrase to unlock the drive:

[ * * *    ] A start job is running for /dev/mapper/cryptswap (51s / no limit)

To get past the issue I have to reboot and get lucky. Today it's taken a lot of retries to finally get past the issue.

In my BIOS security tab I could see that the secure boot was enabled:

system76-bios-security

Once I was able to access my machine, I was able to confirm that I'm booting with UEFI enabled:

$ sudo efibootmgr
[sudo] password for kiba: 
BootCurrent: 0005
Timeout: 1 seconds
BootOrder: 0005,0002,0001,0006,0000
Boot0000* ubuntu
Boot0001* UEFI: IP4 Realtek PCIe GBE Family Controller
Boot0002* UEFI: IP6 Realtek PCIe GBE Family Controller
Boot0005* Pop!_OS 18.10
Boot0006* UEFI OS
$ df -h --local | grep /boot
/dev/nvme0n1p1         498M  212M  286M  43% /boot/efi
pkulak commented 5 years ago

I'm getting this on a brand new Meerkat. Haven't done anything but update and run the browser a couple times.

Hit it once, cut the power to shut down and then it booted.

jmfranks commented 5 years ago

We've seen it happen once or twice in the last year on a laptop, but restarting fixed it and it could not be replicated again, so it is in the rare "Heisenbug" category. System76 hardware appears to be largely unaffected.

I have two System 76 meerkats which exhibit this -- one rarely and a new one consistently. Usually booting from the boot menu (F10) works. Also commenting out the swap line in /etc/fstab works reliably. But I assume I have no swap. How can I mount an unencrypted swap partition? Can I use the partition from /etc/crypttab? Would it have to be reformatted?

TrendMend commented 5 years ago

I am constantly getting this error on my acer aspire laptop, I would be willing to ship it out to you guys/gals as long as I get it back sometime this year... It has an ssd so shipping (shouldn't) kill the install.

horrokotoj commented 5 years ago

Hi guys! Thanks for some great posts! The nofail comment in fstab seems to have resolved the issue for me. At least partly. Now, when I try to boot I seem to be able to start and load the OS but no UI emerges. Lines stating processes that start during boot stops at either [ OK ] Started LSB: automatic crash report generation, or at [ OK] started GNOME Display Manager. Does any one have a clue what might have happened here?

TomFaulkner commented 5 years ago

I have rebooted a dozen times so far today and not been able to get passed this issue. Another user is in the same situation in the Pop subreddit.

I am on a Galago Pro on Pop 19.04. I have had this issue off and on for months but it has never taken this many attempts.

See case 38271 for my support ticket.

rstrube commented 5 years ago

This happens occasionally on my Darter Pro 2019. I would estimate 1 time in every 20 boots. It also sometimes happens when shutting the system down.

jmfranks commented 5 years ago

I can tell you what seems to have worked for me. YMMV. I have two meerkats exhibiting this issue. On setup I chose unencrypted file systems. I have commented out the lines involving cryptswap in both /etc/fstab and /etc/crypttab and made no other changes. After that I have not seen any boot problems. Of course you have to boot once at least in order to make these changes. What has surprised me is that there is still a mounted swap partition (as shown in /proc/swaps) despite no swap partition being listed in /etc/fstab. I assume this swap is not encrypted, but that is what I expected when I chose an unencrypted file system.

jackpot51 commented 5 years ago

@mmstick this looks like a systemd bug - https://github.com/systemd/systemd/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+cryptography

rstrube commented 5 years ago

@jackpot51 @mmstick because this issue is affecting a large number of users, perhaps System76 could provide some official instructions on how to disable cryptgraphic swap? Simply commenting out the line in the fstab would completely disable swap no?

rstrube commented 5 years ago

@Nodocify Would this work?

sudo swapoff -a
sudo mkswap /dev/nvme0n1p4
sudo swapon /dev/nvme0n1p4
sudo vim /etc/fstab

Where /dev/nvme0n1p4 is the correct block device for your swap partition.

Then comment out current /dev/mapper/cryptswap line and replace with:

PARTUUID=f0c3ae89-9ef2-4c19-8a90-bf6439f5869f none swap defaults  0 0

Where UUID is the UUID of the partition

Also comment out the cryptswap line in /etc/crypttab

I noticed that you used the device label (e.g. /dev/nvme0n1p4) in the /etc/fstab file, but the rest of the file uses partition UUIDs.

mmstick commented 5 years ago

Of course, it needs to be the PARTUUID, which is different from the file system-specified UUID, fetched from lsblk -o NAME,FSTYPE,PARTUUID. Only MBR and GPT tables have PartUUIDs, and MBR emulates the support at best.

mmstick commented 5 years ago

I believe I've found a solution. I will need to figure out how it can be applied in an automatic fashion, however.

mmstick commented 5 years ago

Try this:

sudo systemctl edit systemd-cryptsetup@cryptswap.service

Then paste and save

[Service]
ExecStartPost=/sbin/udevadm trigger /dev/mapper/%i

It should be fixed. It's a bug in cryptsetup's systemd integration. So I will need to patch systemd to apply this automatically by default.

jmfranks commented 5 years ago

@rstrube @mmstick rstrube says: Simply commenting out the line in the fstab would completely disable swap no?

I think not necessarily. From https://wiki.archlinux.org/index.php/Swap#Activation_by_systemd

systemd activates swap partitions based on two different mechanisms. Both are executables in /usr/lib/systemd/system-generators. The generators are run on start-up and create native systemd units for mounts. The first, systemd-fstab-generator, reads the fstab to generate units, including a unit for swap. The second, systemd-gpt-auto-generator inspects the root disk to generate units. It operates on GPT disks only, and can identify swap partitions by their type GUID, see systemd#GPT partition automounting for more information.

I am running two systems with no swap entry in /etc/fstab and nothing in /etc/crypttab. Both seem to have swap activated on the swap partition.

jmfranks commented 5 years ago

Also from https://wiki.archlinux.org/index.php/Systemd#GPT_partition_automounting

GPT partition automounting

On a GPT partitioned disk systemd-gpt-auto-generator(8) will mount partitions following the Discoverable Partitions Specification, thus they can be omitted from fstab.

mmstick commented 5 years ago

I've created a patch for systemd that seems to work. I will do some further testing before it's made available as an update.

rstrube commented 5 years ago

I just updated the systemd packages from the System76 proposed PPA, and now on a Darter Pro 2019, I was successfully able to boot 10/10 times. Ordinarily I run into issues once every 10 times or so with the cryptswap problem.

The package versions that I updated to was systemd 240-6ubuntu5.2pop1

rstrube commented 5 years ago

I've performed another 15 boot sequences and have not run into the cryptswap issues. This fix has been working very well for me.

siistibuddy commented 5 years ago

How are you guys able to implement these fixes when POP won't even boot passed the cryptswap step? I have tried every keystroke imaginable. Also @mmstick, VMWare has a ton more adoption than QEMU, would you be willing to add player to your QA testing?

MarkJaroski commented 5 years ago

Hi mmstick, are you going to do a pull request upstream for this bug?

I have this problem on Ubuntu 19.04 on a Dell Sputnik, so I think it's a little more widespread than Pop.