Boot freezes on: A start job is running for dev-mapper-cryptswap

spysiuk commented 6 years ago

Distribution: NAME="Pop!_OS" VERSION="18.04 LTS" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Pop!_OS 18.04 LTS" VERSION_ID="18.04" HOME_URL="https://system76.com/pop" SUPPORT_URL="http://support.system76.com" BUG_REPORT_URL="https://github.com/pop-os/pop/issues" PRIVACY_POLICY_URL="https://system76.com/privacy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic

Issue/Bug Description: After recent update boot freezes on A start job is running for dev-mapper-cryptswap ... / no limit. Until I press any key timer is increasing.

Expected behavior: Boot without user intervention.

Other Notes: I tried to comment swap in /etc/fstab and /etc/crypttab but after that I didn't get login screen - had to go to other terminal, remove comments in both files and reboot machine.

mmstick commented 6 years ago

You'd only need to comment the crypttab, and you would need to always do an update-initramfs -u after that.

spysiuk commented 6 years ago

Thank you mmstick. After commenting entry in crypttab I still have 'A start job is running for dev-mapper-cryptswap' during boot but with 1:30 limit (last time there was no limit). After that time booting continues but in journalctl I have: lip 07 20:21:01 vaio systemd[1]: dev-mapper-cryptswap_qSCcI.device: Job dev-mapper-cryptswap_qSCcI.device/start timed out. lip 07 20:21:01 vaio systemd[1]: Timed out waiting for device dev-mapper-cryptswap_qSCcI.device. lip 07 20:21:01 vaio systemd[1]: Dependency failed for /dev/mapper/cryptswap_qSCcI. lip 07 20:21:01 vaio systemd[1]: dev-mapper-cryptswap_qSCcI.swap: Job dev-mapper-cryptswap_qSCcI.swap/start failed with result 'dependency'. lip 07 20:21:01 vaio systemd[1]: dev-mapper-cryptswap_qSCcI.device: Job dev-mapper-cryptswap_qSCcI.device/start failed with result 'timeout'.

mmstick commented 6 years ago

What's the contents of your fstab / crypttab?

spysiuk commented 6 years ago

fstab: UUID=2A29-825C /boot/efi vfat umask=0077 0 0 UUID=24dddfd8-0034-4eb8-ba71-4c3aa3ff83fb / ext4 noatime,errors=remount-ro 0 0 UUID=d7636828-6d7f-4088-a1f9-93fa43c85052 /home ext4 noatime,errors=remount-ro 0 0 /dev/mapper/cryptswap_qSCcI none swap defaults 0 0

crypttab (currently commented): cryptswap_qSCcI UUID=ecb21db5-ce2a-4668-864c-ca43b75422f8 /dev/urandom swap,offset=1024,cipher=aes-xts-plain64,size=512

mmstick commented 6 years ago

Should remove the /dev/mapper part from the fstab, too.

zvirdaniel commented 6 years ago

I have exactly the same issue as well. systemd-analyze blame told me that systemd-cryptsetup@cryptswap.service took over 50 seconds to load. I guess that is when I pressed some buttons.

zvirdaniel commented 6 years ago

After commenting out:

/etc/crypttab: cryptswap UUID=ca2f2392-903f-4731-b24f-0e5aaf30803a /dev/urandom swap,offset=1024,cipher=aes-xts-plain64,size=512

/etc/fstab: /dev/mapper/cryptswap none swap defaults 0 0

I was left without an active swap partition and the system booted only after several reboots. systemd-analyze blame told me that plymouth-quit-wait.service took 2min 11.275s to boot. Possible solutions?

rkennesson commented 5 years ago

ran into this when installing popOS 19.04 with VMware and VirtualBox even when choosing not to encrypt the drive (encrypting the drive also hung but I have no idea why). Is there a fix available and referably one I can use during install? I tried some of the others maybe I typed them in wrong or didn't follow them properly but I couldn't get them to work.

specifically I get this error

A start job is running for /dev/mapper/cryptswap (_min _s / no limit)

mmstick commented 5 years ago

Sounds like you've deleted the swap partition, or reformatted it after the installer created it. The swap partition created by the installer will always be encrypted, but it expects the swap partition to have the same UUID between boots so that it may create an encrypted swap partition inside of the swap partition.

rkennesson commented 5 years ago

I haven't done anything other than a fresh install in a VM using the popOS installer and reboot. Whatever the result is it was done by the installer.

rkennesson commented 5 years ago

Here is the full install log

https://pastebin.com/hgFPbaVs

mmstick commented 5 years ago

cryptswap UUID=7e82b0cf-15e0-4241-9d0b-08364d846701

If you wait long enough, or you boot in single user mode with 1 as a kernel option, systemd will give up trying to load the cryptswap entry. Could you confirm that the UUID of the swap partition is the same as in the crypttab file?

rkennesson commented 5 years ago

crypttab:

cryptswap UUID=7e82b0cf-15e0-4241-9d0b-08364d846701 /dev/urandom swap,offset=1024,cipher=aes-xts-plain64,size=512

cryptswap UUID=7e82b0cf-15e0-4241-9d0b-08364d846701 It looks the same to me.

mmstick commented 5 years ago

Strange. A copy of your system logs would be helpful. This command would dump the logs collected since the last system boot, to the system.log file in your home folder (if you aren't logged in as root).

journalctl -b0 > ~/system.log

rkennesson commented 5 years ago

I haven't been able to boot into the system and can only access the install ISO.

To switch to single user mode i've added a 1 and single (as some sites suggest) to the grub menu item and the system still hangs with ( _ / no limit).

mmstick commented 5 years ago

What VM are you using?

rkennesson commented 5 years ago

I'm using VMware. My first troubleshooting step was to try it in VirtualBox but I had the same issue. I'm checking it in VirtualBox again right now just to make sure.

edit: same behaviour in VirtualBox

mmstick commented 5 years ago

I've only been testing installs in KVM/QEMU VMs from a Pop!_OS host. I'll have to try this out in those two.

ShawnMahon commented 5 years ago

I am hitting the same issue. VMplayer trying to install 19.04 using the amd/intel image.

rkennesson commented 5 years ago

Are you on the latest VMplayer and/or on Windows 10?

My issue was with the latest VMWare Workstation or latest VirtualBox and Windows 10.

ShawnMahon commented 5 years ago

I'm on VMWare 14, I havent updated to 15 yet. I am on the latest Win10.

Spoil001 commented 5 years ago

I'm hitting the same issue without using a VM. The system is working fine on some boots and taking forever on others.

UIDs match

/etc/fstab 
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system>  <mount point>  <type>  <options>  <dump>  <pass>
PARTUUID=e4086903-5e38-4f57-beca-e06a05223493  /boot/efi  vfat  umask=0077  0  0
/dev/mapper/cryptswap  none  swap  defaults  0  0
UUID=27108123-06bd-4347-a0c5-9ea2ba62a034  /  ext4  noatime,errors=remount-ro  0  0

/etc/crypttab 
cryptswap UUID=f8e00ba8-e6b6-446a-a37a-1acfcfafc679 /dev/urandom swap,offset=1024,cipher=aes-xts-plain64,size=512

ls -Al /dev/disk/by-uuid 
insgesamt 0
lrwxrwxrwx 1 root root 10 Apr 24 18:46 26E000B4E0008BED -> ../../sda1
lrwxrwxrwx 1 root root 10 Apr 24 18:46 27108123-06bd-4347-a0c5-9ea2ba62a034 -> ../../sdb3
lrwxrwxrwx 1 root root 10 Apr 24 18:46 35E7-8D4F -> ../../sdb1
lrwxrwxrwx 1 root root 10 Apr 24 18:46 4378585b-b624-4132-a9da-0d7898ddbb9a -> ../../dm-0
lrwxrwxrwx 1 root root 10 Apr 24 18:46 C00A06010A05F566 -> ../../sda4
lrwxrwxrwx 1 root root 10 Apr 24 18:46 CE01-1FAA -> ../../sda2
lrwxrwxrwx 1 root root 10 Apr 24 18:46 f8e00ba8-e6b6-446a-a37a-1acfcfafc679 -> ../../sdb2

Gabriel-p commented 5 years ago

Same issue here. Freshly installed Pop-OS 19.04 on an empty SSD disk. 4 min and counting waiting for cryptswap (first boot after installing the OS with no encryption)

mmstick commented 5 years ago

There seems to be a correlation with vmware and virtualbox. I do multiple installs a day with KVM/QEMU without issue. I wonder, are these EFI or legacy BIOS installs? I'm always using the OVMF firmware.

Gabriel-p commented 5 years ago

In my case I'm installing on an old Dell laptop through a burned USB. Not sure if this is a "legacy BIOS install"?

Also I checked installing with the encrypted disk, and I get same issue: waiting for cryptswap, no limit.

rkennesson commented 5 years ago

On VirtualBox I switched to EFI mode (using the standard install and ejecting the virtual disk) and I get this error at startup

Boot Failed. EFI DVD/CDROM

after a string of reboots where sometimes it boots and sometimes it doesn't. It now seems to boot reliably but shows that same error everytime.

TylerSchmidtke commented 5 years ago

I just experienced this as well on an 19.04 installation on a SSD, no virtualization. After multiple restarts, the system finally booted.

rkennesson commented 5 years ago

Thank you all for posting.

If you know of anyone else having issues ask them to post their experience and if they have any solutions.

Spoil001 commented 5 years ago

My short term solution was to remove the cryptswap. Losely based on this article: https://www.logilab.org/blogentry/29155 First remove the cryptswap, then edit /etc/crypttab (comment the cryptswap line) and etc/fstab change the cryptswap line to normal swap. Reboot

nekdolan commented 5 years ago

I have the same issue with a clean 19.04 install (+some cloud services) on a lenovo l460. I had no issue with 18.04. If I restart after the system hangs things seem to work fine. I also don't seem to be able to shut down the system.

EDIT: the issue seems to have fixed itself. I am guessing that I should have waited until all of my cloud was encoded.

pierrep commented 5 years ago

I just did a fresh install of pop-os 19.04 on a Dell XPS 13 9360, got the latest updates, then shutdown. On restart just now, I'm getting this error. I rebooted and everything worked as normal.

flucoe commented 5 years ago

I just did a fresh install too (pop-os 19.04), got the latest updates, and have the same problem. Rebooting didn't change things.

mmstick commented 5 years ago

If you add nofail as a mount option in /etc/fstab for the cryptswap entry, does this allow you to boot? That would at least allow you to continue booting even if the swap partition cannot be found. Though it would be nice to know what's causing your swap partition to disappear.

flucoe commented 5 years ago

Sorry, I just saw this. Today, without having made any changes, I didn't have any issues. I can try your suggestion but I wouldn't know how to identify whether adding nofail makes a difference.

mmstick commented 5 years ago

If you're able to boot, it's working.

pierrep commented 5 years ago

I'm getting this most of the time now, whenever I do a cold boot. My fstab looks like this:

# <file system>  <mount point>  <type>  <options>  <dump>  <pass>
PARTUUID=e80261e0-2ee4-4c33-b93c-0f8dcd263b7c  /boot/efi  vfat  umask=0077  0  0
PARTUUID=e6d29290-76bb-43f2-931d-f5bc87307013  /recovery  vfat  umask=0077  0  0
UUID=09c20a5d-6415-48a5-9a9c-f05323cba3e4  /  ext4  noatime,errors=remount-ro  0  0
/dev/mapper/cryptswap  none  swap  defaults  0  0

Any thing I can do to stop this? I did a vanilla install with no custom partitioning.

Nodocify commented 5 years ago

I also experienced this same issue with a fresh install of 19.04 on my galago pro, default options through the installer. The problem occured at first reboot after install. Then I reinstalled and this time it was after a couple reboots that I encountered the A start job is running for /dev/mapper/cryptswap. I also opened a support case with System76, but they had no idea how to assist. They weren't even aware this github issue existed.

So this is my solution as I feel that a cryptswap is overkill and I understand the security concerns with NOT having swap encrypted. I took the idea from @Spoil001 where he mentioned to disable the cryptswap.

Do this at your own risk and please understand what you are doing before doing it. (Obviously replace the device names with the ones from your setup).

Boot any linux live cd

Unlock luks sudo cryptsetup luksOpen /dev/nvme0n1p3 cryptroot enter password

Get the logical volume to mount. sudo lvscan

Mount the logical volume. sudo mount /dev/data/root /mnt cd /mnt/etc

sudo vim fstab comment out the swap line

sudo vim crypttab remove the cryptswap line

reboot This allowed Pop_OS to boot without a swap.

Pop_OS

Next disable the cryptswap and create a standard swap.

Turn off swap. sudo swapoff -a

Format a normal swap over the cryptswap. sudo mkswap /dev/nvme0n1p4

Turn on swap. sudo swapon /dev/nvme0n1p4

Add entry to fstab for future reboots. sudo vim /etc/fstab Uncomment and replace /dev/mapper/cryptswap with /dev/nvme0n1p4 Reboot and make sure that you now have a standard swap mounted.

I needed to get this resolved quickly and get my computer up and going for work. Hopefully this can help someone as a quick and dirty solution.

@mmstick I tried your nofail solution of adding that to the swap options in /etc/fstab. It didn't prevent the error message and I let it go for 10 minutes in which it didn't timeout. I then shutdown and did the above. Please let me know if there is any assistance I can offer to better assist in an official resolution.

mmstick commented 5 years ago

@Nodocify You can try adding nofail to the crypttab entry's options, too.

pierrep commented 5 years ago

Just wondering @mmstick - should the swap be encrypted even if encryption was turned off in the installer, as it was in my install (I chose no encryption)?

mmstick commented 5 years ago

Swap is always encrypted

shaunhey commented 5 years ago

Just wanted to leave a comment that I’m also seeing this occasionally on 19.04 installed directly on a Dell Latitude 7370 (XPS’ cousin). I’ve since disabled cryptswap as others recommended and so far I have not encountered the issue again.

earlev4 commented 5 years ago

Hello. I am experiencing the same issue on VirtualBox (Version 6.0.6 r130049 (Qt5.9.5)) and VMware Workstation 15 Pro (Version 15.0.4 build-12990004) when installing pop-os_19.04_amd64_intel_4. Boot up just sits at: A start job is running for /dev/mapper/cryptswap I just installed RHEL 8 in VirtualBox and did not experience any issues. Also tried booting up in GRUB with "Ubuntu, with Linux 5.0.0-13-generic (recovery mode)" and experienced the same issue. Thanks!

Draconicrose commented 5 years ago

Also encountering this issue on Pop!_OS 19.04, installed on bare metal.

pt8o commented 5 years ago

Same here across two devices. My work device was on 18.10 and I upgraded to 19.04. My home computer was a wipe and clean install. It happens about 50% of the time.

I did the nofail thing, so they will log in now if cryptswap is hanging, but after a 90 second timeout. Would love to get this resolved! Let me know if there's a log I can post or anything like that.

luispjoaquim commented 5 years ago

I can confirm this issue in two different laptops (bare metal install).

YggrYergen commented 5 years ago

I also confirm this issue on a lenovo. (model name: 80UL; Pop!_os 19.04)

It gets "solved" as follows:

Method 1: When initializing computer hit "fn+f2", System will prompt to choose beetwen options to boot-in. Choose "pop!_os! oldkernel".

This method, for my machine, hasn't had any unconsistent behavior yet. It should get you running straight to a normal start.

If that didn't work i recommend you: Method 2: do the same but instead of "pop!_os olkernel" Choose "pop!_os recovery".

System will load as a for-install version, do nothing but set the system to restart normally. This is, from the top-right menu.

Please comment if this method works for you. I'll be coming back with consistency reports.

reidstidolph commented 5 years ago

Nothing substantial for me to add, other than "me too".

Pop 19.04, Freshly installed and updated
Lenovo X1 Carbon 5th gen
Disk encryption enabled
System had hung on shutdown, so I forced it off by holding down the power button.
Hung on boot up exactly as described above.
After 4 boot attempts, it booted fine.
Subsequent reboots have gone normally so far (knock on wood).

TrendMend commented 5 years ago

Still no fix... I am stuck with 1min 30second wait time after adding nofail... but it would be nice to not have this issue at all, I really hope this gets elevated.

mmstick commented 5 years ago

@TrendMend We haven't able to replicate the issue (despite doing dozens of installs per day in the lab), so there's no way to determine the cause.

mambojuice commented 5 years ago

Same issue here. Just did a fresh install, HP EliteBook 1040. Installed updates after first boot, now second boot hangs at "A start job is waiting for /dev/mapper/cryptswap" for 5+ minute before I hard reboot. After enough restarts it eventually boots.

pop-os / pop