search

kubernetes-sigs / kubespray

Deploy a Production Ready Kubernetes Cluster
Apache License 2.0
15.94k stars 6.42k forks source link

Mask swap.target doesn't take effect after reboot #10741

Closed tu1h closed 8 months ago

tu1h commented 9 months ago

I found that swap partition would still be actived after masking swap.target and rebooting.

Related to https://github.com/kubernetes-sigs/kubespray/pull/10587

image

Environment:

CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"



- **Version of Ansible** (`ansible --version`):

- **Version of Python** (`python --version`):

**Kubespray version (commit) (`git rev-parse --short HEAD`):**

**Network plugin used**:

**Full inventory with variables (`ansible -i inventory/sample/inventory.ini all -m debug -a "var=hostvars[inventory_hostname]"`):**
<!-- We recommend using snippets services like https://gist.github.com/ etc. -->

**Command used to invoke ansible**:

**Output of ansible run**:
<!-- We recommend using snippets services like https://gist.github.com/ etc. -->

**Anything else do we need to know**:
<!-- By running scripts/collect-info.yaml you can get a lot of useful informations.
Script can be started by:
ansible-playbook -i <inventory_file_path> -u <ssh_user> -e ansible_ssh_user=<ssh_user> -b --become-user=root -e dir=`pwd` scripts/collect-info.yaml
(If you using CoreOS remember to add '-e ansible_python_interpreter=/opt/bin/python').
After running this command you can find logs in `pwd`/logs.tar.gz. You can even upload somewhere entire file and paste link here.-->
VannTen commented 9 months ago

Odd. What's in /proc/swaps ?

tu1h commented 9 months ago
image
VannTen commented 9 months ago

So I checked that swap.target could indeed be masked to disable swap devices of any kind, (just to see if I was not crazy), it does work on some systems :tm: . So, next steps:

Also, please post text and not images, it's harder to read.

tu1h commented 9 months ago
  1. systemctl --version 
    systemd 219
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN
  2. systemctl --type swap 
    UNIT                           LOAD   ACTIVE SUB    DESCRIPTION
dev-mapper-centos\x2dswap.swap loaded active active /dev/mapper/centos-swap
  3. systemctl cat dev-mapper-centos\x2dswap.swap No files found for dev-mapper-centosx2dswap.swap.
  4. systemctl show dev-mapper-centos\x2dswap.swap 
    What=/dev/mapper/centosx2dswap
Priority=-1
TimeoutUSec=1min 30s
ControlPID=0
Result=success
Slice=system.slice
MemoryCurrent=18446744073709551615
TasksCurrent=18446744073709551615
Delegate=no
CPUAccounting=no
CPUShares=18446744073709551615
StartupCPUShares=18446744073709551615
CPUQuotaPerSecUSec=infinity
BlockIOAccounting=no
BlockIOWeight=18446744073709551615
StartupBlockIOWeight=18446744073709551615
MemoryAccounting=no
MemoryLimit=18446744073709551615
DevicePolicy=auto
TasksAccounting=no
TasksMax=18446744073709551615
UMask=0022
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=4096
LimitAS=18446744073709551615
LimitNPROC=127948
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=127948
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615
OOMScoreAdjust=0
Nice=0
IOScheduling=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SecureBits=0
CapabilityBoundingSet=18446744073709551615
AmbientCapabilities=0
MountFlags=0
PrivateTmp=no
PrivateNetwork=no
PrivateDevices=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=no
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
RuntimeDirectoryMode=0755
KillMode=control-group
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=dev-mapper-centosx2dswap.swap
Names=dev-mapper-centosx2dswap.swap
Requires=-.mount system.slice
Conflicts=umount.target
Before=umount.target swap.target
After=-.mount system.slice systemd-journald.socket
RequiresMountsFor=/dev/mapper/centosx2dswap
Description=/dev/mapper/centosx2dswap
LoadState=loaded
ActiveState=inactive
SubState=dead
InactiveExitTimestampMonotonic=0
ActiveEnterTimestampMonotonic=0
ActiveExitTimestampMonotonic=0
InactiveEnterTimestampMonotonic=0
CanStart=yes
CanStop=yes
CanReload=no
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=yes
IgnoreOnSnapshot=no
NeedDaemonReload=no
JobTimeoutUSec=0
JobTimeoutAction=none
ConditionResult=no
AssertResult=no
ConditionTimestampMonotonic=0
AssertTimestampMonotonic=0
Transient=no
CollectMode=inactive
piwinkler commented 9 months ago

can reproduce it OS Linux 3.10.0-1160.105.1.el7.x86_64 x86_64 NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"

possible solution is to mask every swap unit file

[root@foo ~]# systemctl --type swap
UNIT                                                                      LOAD   ACTIVE SUB    DESCRIPTION
dev-disk-by\x2duuid-4470b768\x2def1c\x2d40f8\x2d8751\x2d104e00bfd3c6.swap loaded active active /dev/disk/by-uuid/4470b768-ef1c-40f8-8751-104e00bfd3c6

[root@foo ~]# systemctl mask "dev-disk-by\x2duuid-4470b768\x2def1c\x2d40f8\x2d8751\x2d104e00bfd3c6.swap"

on a more recent system there are no unit files generated after the swap target got masked

VannTen commented 8 months ago

systemctl cat dev-mapper-centos\x2dswap.swap

Can you quote the unit name / escape the \ ? I think the shell is swallowing it.

(And I forgot last time: could you also post the result of systemctl status on the unit ? And any relevants logs if you can look at them before log rotation)

on a more recent system there are no unit files generated after the swap target got masked

No unit files or not started ? AFAIK they should be generated by fstab-generator, just not started.

tu1h commented 8 months ago

@VannTen I post some details.

systemctl cat dev-mapper-centos\x2dswap.swap

# /run/systemd/generator/dev-mapper-centos\x2dswap.swap
# Automatically generated by systemd-fstab-generator

[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)

[Swap]
What=/dev/mapper/centos-swap

systemctl show "dev-mapper-centos\x2dswap.swap"

What=/dev/mapper/centos-swap
Priority=-1
TimeoutUSec=1min 30s
ControlPID=0
Result=success
Slice=system.slice
MemoryCurrent=18446744073709551615
TasksCurrent=18446744073709551615
Delegate=no
CPUAccounting=no
CPUShares=18446744073709551615
StartupCPUShares=18446744073709551615
CPUQuotaPerSecUSec=infinity
BlockIOAccounting=no
BlockIOWeight=18446744073709551615
StartupBlockIOWeight=18446744073709551615
MemoryAccounting=no
MemoryLimit=18446744073709551615
DevicePolicy=auto
TasksAccounting=no
TasksMax=18446744073709551615
UMask=0022
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=4096
LimitAS=18446744073709551615
LimitNPROC=127948
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=127948
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615
OOMScoreAdjust=0
Nice=0
IOScheduling=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SecureBits=0
CapabilityBoundingSet=18446744073709551615
AmbientCapabilities=0
MountFlags=0
PrivateTmp=no
PrivateNetwork=no
PrivateDevices=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=no
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
RuntimeDirectoryMode=0755
KillMode=control-group
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=dev-mapper-centos\x2dswap.swap
Names=dev-mapper-centos\x2dswap.swap
Requires=system.slice -.mount
BindsTo=dev-mapper-centos\x2dswap.device
RequiredBy=swap.target
WantedBy=dev-mapper-centos\x2dswap.device
Conflicts=umount.target
Before=umount.target swap.target
After=dev-mapper-centos\x2dswap.device system.slice systemd-journald.socket -.mount
RequiresMountsFor=/dev/mapper/centos-swap
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Description=/dev/mapper/centos-swap
LoadState=loaded
ActiveState=inactive
SubState=dead
FragmentPath=/run/systemd/generator/dev-mapper-centos\x2dswap.swap
SourcePath=/etc/fstab
UnitFileState=bad
UnitFilePreset=disabled
InactiveExitTimestamp=Wed 2023-06-21 05:03:14 EDT
InactiveExitTimestampMonotonic=32663728
ActiveEnterTimestamp=Wed 2023-06-21 05:03:14 EDT
ActiveEnterTimestampMonotonic=32677758
ActiveExitTimestamp=Tue 2024-01-02 02:15:31 EST
ActiveExitTimestampMonotonic=252979130
InactiveEnterTimestamp=Tue 2024-01-02 02:15:31 EST
InactiveEnterTimestampMonotonic=252979130
CanStart=yes
CanStop=yes
CanReload=no
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=yes
IgnoreOnSnapshot=no
NeedDaemonReload=no
JobTimeoutUSec=0
JobTimeoutAction=none
ConditionResult=yes
AssertResult=yes
ConditionTimestamp=Wed 2023-06-21 05:03:14 EDT
ConditionTimestampMonotonic=32661410
AssertTimestamp=Wed 2023-06-21 05:03:14 EDT
AssertTimestampMonotonic=32661410
Transient=no
CollectMode=inactive

systemctl status "dev-mapper-centos\x2dswap.swap"

● dev-mapper-centos\x2dswap.swap - /dev/mapper/centos-swap
   Loaded: loaded (/etc/fstab; bad; vendor preset: disabled)
   Active: inactive (dead) since Tue 2024-01-02 02:15:31 EST; 2h 6min ago
     What: /dev/mapper/centos-swap
     Docs: man:fstab(5)
           man:systemd-fstab-generator(8)

Jun 21 05:03:14 localhost.localdomain systemd[1]: Activating swap /dev/mapper/centos-swap...
Jun 21 05:03:14 localhost.localdomain systemd[1]: Activated swap /dev/mapper/centos-swap.

journalctl -xu swap.target

-- Logs begin at Wed 2023-06-21 05:02:42 EDT, end at Tue 2024-01-02 04:22:30 EST. --
Jun 21 05:02:48 localhost.localdomain systemd[1]: Stopped target Swap.
-- Subject: Unit swap.target has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit swap.target has finished shutting down.
Jun 21 05:03:14 localhost.localdomain systemd[1]: Reached target Swap.
-- Subject: Unit swap.target has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit swap.target has finished starting up.
--
-- The start-up result is done.
Jan 02 02:16:31 node1 systemd[1]: Cannot add dependency job for unit swap.target, ignoring: Unit is masked.
Jan 02 02:16:31 node1 systemd[1]: Cannot add dependency job for unit swap.target, ignoring: Unit is masked.
Jan 02 02:16:31 node1 systemd[1]: Cannot add dependency job for unit swap.target, ignoring: Unit is masked.
Jan 02 02:19:30 node1 systemd[1]: Cannot add dependency job for unit swap.target, ignoring: Unit is masked.
Jan 02 02:23:36 node1 systemd[1]: Cannot add dependency job for unit swap.target, ignoring: Unit is masked.