search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.55k stars 13.71k forks source link

btrfs: Transaction aborted (error -28) and fs forced readonly #160398

Closed aij closed 2 years ago

aij commented 2 years ago

Describe the bug

After accidentally filling up a btrfs filesystem it was forced readonly with a warning in dmesg:

[2146501.030442] ------------[ cut here ]------------
[2146501.030449] BTRFS: Transaction aborted (error -28)
[2146501.030618] WARNING: CPU: 4 PID: 3540247 at fs/btrfs/inode.c:2736 btrfs_finish_ordered_io.isra.0+0x4d1/0x8b0 [btrfs]
[2146501.030628] Modules linked in: nfsv3 rpcrdma rdma_cm iw_cm ib_cm rpcsec_gss_krb5 nfsv4 dns_resolver nfs nfs_ssc nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype br_netfilter mlx4_ib mlx4_en ib_uverbs ext4 crc16 sb_edac mbcache intel_rapl_msr jbd2 edac_core iTCO_wdt intel_pmc_bxt evdev mac_hid intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp crc32_pclmul ghash_clmulni_intel aesni_intel libaes crypto_simd cryptd glue_helper rapl intel_cstate mgag200 intel_uncore psmouse serio_raw drm_kms_helper igb fb_sys_fops hpwdt syscopyarea sysfillrect watchdog ptp sysimgblt pps_core i2c_algo_bit hpilo xt_comment ses ipmi_ssif ip6_tables enclosure xt_conntrack nf_conntrack mlx4_core nf_defrag_ipv6 lpc_ich nf_defrag_ipv4 ioatdma dca acpi_ipmi thermal ip6t_rpfilter ipmi_si ipt_rpfilter ipmi_devintf ipmi_msghandler xt_pkttype nf_log_ipv6 nf_log_ipv4 nf_log_common xt_LOG acpi_power_meter xt_tcpudp tiny_power_button nft_compat button nft_counter nf_tables
[2146501.030734]  nfnetlink sch_fq_codel loop tun tap macvlan veth bridge stp nfsd llc auth_rpcgss ceph nfs_acl lockd libceph fscache ib_ipoib grace sunrpc ib_umad ib_core kvm_intel drm kvm agpgart irqbypass backlight i2c_core fuse configfs pstore ip_tables x_tables autofs4 sd_mod t10_pi crc_t10dif crct10dif_generic ahci atkbd libahci libps2 mpt3sas uhci_hcd libata ehci_pci ehci_hcd crct10dif_pclmul crct10dif_common usbcore raid_class scsi_transport_sas scsi_mod usb_common i8042 rtc_cmos serio af_packet dm_mod btrfs blake2b_generic libcrc32c crc32c_generic crc32c_intel xor raid6_pq
[2146501.030812] CPU: 4 PID: 3540247 Comm: kworker/u130:2 Tainted: G          I       5.10.93 #1-NixOS
[2146501.030814] Hardware name: HP ProLiant DL380e Gen8, BIOS P73 05/21/2018
[2146501.030848] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[2146501.030878] RIP: 0010:btrfs_finish_ordered_io.isra.0+0x4d1/0x8b0 [btrfs]
[2146501.030882] Code: 0f ba a8 40 0a 00 00 02 72 22 83 fb fb 0f 84 d0 03 00 00 83 fb e2 0f 84 c7 03 00 00 89 de 48 c7 c7 18 ef 19 c0 e8 f2 3b 71 c8 <0f> 0b 89 d9 ba b0 0a 00 00 48 c7 c6 c0 27 19 c0 4c 89 e7 41 bf 01
[2146501.030884] RSP: 0018:ffffb205ce1d3d80 EFLAGS: 00010282
[2146501.030887] RAX: 0000000000000000 RBX: 00000000ffffffe4 RCX: ffff9b43ef61ca88
[2146501.030889] RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff9b43ef61ca80
[2146501.030891] RBP: ffff9b306a8db498 R08: ffff9b443ff67c28 R09: 0000000000013ffb
[2146501.030893] R10: 00000000ffffc000 R11: 3fffffffffffffff R12: ffff9b43392c0c98
[2146501.030894] R13: 000000007fcc4000 R14: 000000007fcc4fff R15: ffff9b3686f02fc0
[2146501.030897] FS:  0000000000000000(0000) GS:ffff9b43ef600000(0000) knlGS:0000000000000000
[2146501.030899] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2146501.030901] CR2: 00007f1cd4109000 CR3: 000000203c410004 CR4: 00000000001706e0
[2146501.030903] Call Trace:
[2146501.030945]  btrfs_work_helper+0xdb/0x2f0 [btrfs]
[2146501.030955]  process_one_work+0x1df/0x370
[2146501.030960]  worker_thread+0x50/0x400
[2146501.030964]  ? process_one_work+0x370/0x370
[2146501.030967]  kthread+0x11b/0x140
[2146501.030971]  ? __kthread_bind_mask+0x60/0x60
[2146501.030976]  ret_from_fork+0x22/0x30
[2146501.030981] ---[ end trace eff61eb78bf9a432 ]---
[2146501.030987] BTRFS: error (device sda1) in btrfs_finish_ordered_io:2736: errno=-28 No space left
[2146501.031090] BTRFS info (device sda1): forced readonly
[2146501.031381] BTRFS: error (device sda1) in __btrfs_free_extent:3099: errno=-28 No space left
[2146501.031488] BTRFS: error (device sda1) in btrfs_run_delayed_refs:2148: errno=-28 No space left

Steps To Reproduce

I'm not yet sure whether this is reproducible, but given that it's a filesystem error I figured it may be worth trying to debug the relevant code regardless.

Steps to reproduce the behavior:

  1. Created filesystem with mkfs.btrfs -L scratch /dev/sd[a-l]1
  2. Lots of time passed and upgrades happened. The system is being deployed via nixops.
  3. Exported via NFS
  4. Moved filesystem from using 12 smallish (200GB) partitions to 6 full disks via btrfs device remove and btrfs replace
  5. Filled up the FS by writing large files over NFS
  6. Noticed I couldn't delete any files because it was mounted readonly
  7. Noticed error in dmesg

NFS export was via

    services.nfs.server = {
      enable = true;
      hostName = "10.0.9.13";
      extraNfsdConfig="rdma=yes";
      exports = ''
        /scratch 10.0.9.0/24(mp,rw,insecure,async,no_subtree_check,all_squash)
      '';
    };

Expected behavior

Should be able to delete files to free up space.

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

This seems likely an upstream kernel bug, but https://bugzilla.kernel.org/ specifically says "Please use your distribution's bug tracking tools"

The name "scratch" is accurate... I do not mind losing data on this FS, but would like to help improve btrfs stability for other uses if I can. :)

A bit more data:

root@m3:~# uname -a
Linux m3 5.10.93 #1-NixOS SMP Thu Jan 20 08:17:52 UTC 2022 x86_64 GNU/Linux
root@m3:~# btrfs --version
btrfs-progs v5.14.1 
root@m3:~# btrfs fi show
Label: 'root'  uuid: 6aba8afd-2d8f-4576-9a2b-3a553b5ab8d9
        Total devices 2 FS bytes used 150.77GiB
        devid    1 size 185.33GiB used 152.06GiB path /dev/sdm3
        devid    2 size 185.33GiB used 152.06GiB path /dev/sdk1

Label: 'scratch'  uuid: 42a2b918-ec2f-4a16-baef-14f62d5900f9
        Total devices 6 FS bytes used 16.35TiB
        devid    1 size 2.73TiB used 2.73TiB path /dev/sdb
        devid    3 size 2.73TiB used 2.73TiB path /dev/sde
        devid    4 size 2.73TiB used 2.73TiB path /dev/sda
        devid    6 size 2.73TiB used 2.73TiB path /dev/sdd
        devid    7 size 2.73TiB used 2.73TiB path /dev/sdc
        devid    8 size 2.73TiB used 2.73TiB path /dev/sdf

Label: none  uuid: fc52ea72-d7a9-41ef-93d6-78ea5f4eff3c
        Total devices 1 FS bytes used 2.48TiB
        devid    1 size 2.55TiB used 2.52TiB path /dev/sdn4

Label: 'scratch2'  uuid: f39e0289-650a-4960-96b4-71500338c52d
        Total devices 5 FS bytes used 2.08TiB
        devid    1 size 2.73TiB used 433.01GiB path /dev/sdg
        devid    2 size 2.73TiB used 433.00GiB path /dev/sdh
        devid    3 size 2.73TiB used 433.01GiB path /dev/sdi
        devid    4 size 2.73TiB used 433.01GiB path /dev/sdj
        devid    5 size 2.73TiB used 433.00GiB path /dev/sdl

Label: none  uuid: b7cd5149-8d9d-440e-8e8e-2054ceac3b42
        Total devices 1 FS bytes used 896.00KiB
        devid    1 size 2.55TiB used 2.17TiB path /dev/sdk2

root@m3:~# btrfs fi df /scratch
Data, single: total=16.34TiB, used=16.33TiB
System, RAID1: total=32.00MiB, used=1.77MiB
Metadata, RAID1: total=19.00GiB, used=18.84GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
root@m3:~#

Notify maintainers

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

(Probably less useful because this is a nixops target)

aij@m3:~$ nix-shell -p nix-info --run "nix-info -m"
warning: unknown setting 'experimental-features'
these paths will be fetched (0.00 MiB download, 0.00 MiB unpacked):
  /nix/store/q124axggd6wl6fh9508sxgkknkifrl0z-nix-info
copying path '/nix/store/q124axggd6wl6fh9508sxgkknkifrl0z-nix-info' from 'https://cache.nixos.org'...
warning: unknown setting 'experimental-features'
 - system: `"x86_64-linux"`
 - host os: `Linux 5.10.93, NixOS, 21.11 (Porcupine)`
 - multi-user?: `yes`
 - sandbox: `yes`
warning: unknown setting 'experimental-features'
 - version: `nix-env (Nix) 2.3.16`
warning: unknown setting 'experimental-features'
 - channels(root): `"nixos-21.05.1353.c06613c25df"`
warning: unknown setting 'experimental-features'
 - channels(aij): `"nixos-22.05pre353535.48d63e924a2, nixpkgs-22.05pre353535.48d63e924a2"`
warning: unknown setting 'experimental-features'
 - nixpkgs: `/home/aij/.nix-defexpr/channels/nixpkgs`

aij@m3:~$ nixos-version 
21.11pre-git (Porcupine)
aij@m3:~$
aij commented 2 years ago

Full dmesg output is here: https://gist.github.com/aij/2ca8ae4ba2c4a490330a04542cdfdd9a

symphorien commented 2 years ago 
[2146501.030987] BTRFS: error (device sda1) in btrfs_finish_ordered_io:2736: errno=-28 No space left
[2146501.031090] BTRFS info (device sda1): forced readonly

btrfs can become read only if you have no unallocated space available and you need to write metadata. This is an upstream "feature" and not really nixos specific. See https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_I_ran_out_of_disk_space.21 for advice. The fine people on #btrfs on libera.chat may also help you.

I can try to give a few hints, but no guarantees, your computer may take fire :)

  1. your fs is read only, so time to make backups
  2. btrfs fi us -T /mnt tells you how much of used and allocated data and metadata you have. You are in the situation where there is <1G of unallocated, and about 0 allocated but unused metadata. The goal is to arrive in a situation where there is >1G of unallocated space.
  3. (this is where I'm fuzzy about the details) To modify the fs again, either just reboot, or you need to add (temporary) space. To add temporary space, btrfs device add /dev/foo. To remove it when you know that even removing the device there would still be unallocated space, btrfs device remove /dev/foo. If your disk is full, do that with a usb drive. (it's a bit risky though).
  4. If data is also in the situation where there is 0 unused but allocated space, remove large files. The wiki recommends truncating instead of removing as it does not need extra metadata. If you do metadata intensive operations like btrfs sub remove the fs may go ro again.
  5. When there is several GB of unused but allocated metadata run btrfs balance -dusage=XXX start /mnt with XXX=5 then increasing this value (it's a percentage) until there are several GB of unallocated space.

Feel free to continue posting on this thread but I'm closing the issue as it is not nixos specific.