Closed xtrcode closed 11 months ago
can you show the numbers reported by sysctl on your end? Usually it casts automatically in these cases, so I expect one of these sysctls is empty or contains something else than a number.
sysctl vm.stats.vm
sysctl hw.physmem
Sure, the output lgtm though
TLDR:
# sysctl hw.physmem
hw.physmem: 8333033472
# sysctl vm.stats.vm.v_page_count
vm.stats.vm.v_page_count: 1967584
# sysctl vm.stats.vm.v_inactive_count
vm.stats.vm.v_inactive_count: 191927
# sysctl vm.stats.vm.v_cache_count
vm.stats.vm.v_cache_count: 0
# sysctl vm.stats.vm.v_free_count
vm.stats.vm.v_free_count: 1409719
Long version:
# sysctl vm.stats.vm
vm.stats.vm.v_pdpages: 464961
vm.stats.vm.v_tcached: 0
vm.stats.vm.v_cache_count: 0
vm.stats.vm.v_user_wire_count: 0
vm.stats.vm.v_free_severe: 7545
vm.stats.vm.v_interrupt_free_min: 2
vm.stats.vm.v_pageout_free_min: 34
vm.stats.vm.v_laundry_count: 0
vm.stats.vm.v_inactive_count: 191706
vm.stats.vm.v_inactive_target: 62962
vm.stats.vm.v_active_count: 72561
vm.stats.vm.v_wire_count: 294747
vm.stats.vm.v_free_count: 1407134
vm.stats.vm.v_free_min: 12464
vm.stats.vm.v_free_target: 41975
vm.stats.vm.v_free_reserved: 2627
vm.stats.vm.v_page_count: 1967584
vm.stats.vm.v_page_size: 4096
vm.stats.vm.v_kthreadpages: 0
vm.stats.vm.v_rforkpages: 51232
vm.stats.vm.v_vforkpages: 2324351
vm.stats.vm.v_forkpages: 3503315
vm.stats.vm.v_kthreads: 18
vm.stats.vm.v_rforks: 1551
vm.stats.vm.v_vforks: 43084
vm.stats.vm.v_forks: 33536
vm.stats.vm.v_tfree: 44445045
vm.stats.vm.v_pfree: 33191975
vm.stats.vm.v_dfree: 0
vm.stats.vm.v_pdshortfalls: 0
vm.stats.vm.v_pdwakeups: 0
vm.stats.vm.v_reactivated: 0
vm.stats.vm.v_intrans: 23172
vm.stats.vm.v_vnodepgsout: 7574
vm.stats.vm.v_vnodepgsin: 46907
vm.stats.vm.v_vnodeout: 4743
vm.stats.vm.v_vnodein: 6160
vm.stats.vm.v_swappgsout: 0
vm.stats.vm.v_swappgsin: 0
vm.stats.vm.v_swapout: 0
vm.stats.vm.v_swapin: 0
vm.stats.vm.v_ozfod: 0
vm.stats.vm.v_zfod: 30362148
vm.stats.vm.v_cow_optim: 5399
vm.stats.vm.v_cow_faults: 6766790
vm.stats.vm.v_io_faults: 5817
vm.stats.vm.v_vm_faults: 44553668
# sysctl hw.physmem
hw.physmem: 8333033472
edit: typo
Usually it casts automatically in these cases, so I expect one of these sysctls is empty or contains something else than a number.
Considering the code at https://github.com/opnsense/core/blob/11e76b63739078705b0e214e0c20b27277e0c325/src/etc/inc/util.inc##L1145-L1155 an empty string is returned if sysctl returns nothing.
A fix might be an is_numeric
check before calculating the values at system.inc:109. What do you think?
I doubt empty($value)
is a good choice here as it will transform 0
to ''
. The caller knows best what value is expected. Certainly for arithmetics the caller should make sure it is dealing with integer (or possibly float) values.
But I also think the kernel or harware is glitching here if it returns 0 on memory. ;)
I'm just a bit lost why it doesn't complain on my end to be honest vm.stats.vm.v_cache_count
equals 0 on my end too, but without complains. returning ''
when 0
was returned is certainly not a great idea.
https://github.com/opnsense/core/commit/8c2aa5d6bd7f0b718f3940f7b3d8da9b1828861b might make a difference, although I can't reproduce this on my end.
thanks! when will the fix be available in stable?
When someone with the problem can confirm the fix ideally. You can install the patch easily with this shell command:
# opnsense-patch 8c2aa5d6
running the command again will remove it if it’s not making it better.
I've applied the patch and will report if the problem persists!
# opnsense-patch 8c2aa5d6
Fetched 8c2aa5d6 via https://github.com/opnsense/core
Hmm... Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|From 8c2aa5d6bd7f0b718f3940f7b3d8da9b1828861b Mon Sep 17 00:00:00 2001
|From: Ad Schellevis <ad@opnsense.org>
|Date: Tue, 31 Oct 2023 14:25:11 +0100
|Subject: [PATCH] util.inc - get_single_sysctl : only return '' when value not
| found. for https://github.com/opnsense/core/issues/6969
|
|---
| src/etc/inc/util.inc | 2 +-
| 1 file changed, 1 insertion(+), 1 deletion(-)
|
|diff --git a/src/etc/inc/util.inc b/src/etc/inc/util.inc
|index f5520e38cf..ea1eee7802 100644
|--- a/src/etc/inc/util.inc
|+++ b/src/etc/inc/util.inc
--------------------------
Patching file etc/inc/util.inc using Plan A...
Hunk #1 succeeded at 1167 (offset 19 lines).
done
All patches have been applied successfully. Have a nice day.
Somehow the system still crashes but at different lines - the cause seems to be the same:
System Information:
User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/119.0
FreeBSD 13.2-RELEASE-p3 stable/23.7-n254818-f155405f505 SMP amd64
OPNsense 23.7.7_3 f464e1901
Plugins os-api-backup-1.1 os-bind-1.27_1 os-cache-1.0_1 os-ddclient-1.16_1 os-dmidecode-1.1_1 os-hw-probe-1.0_1 os-iperf-1.0_1 os-realtek-re-1.0 os-shadowsocks-1.0_2 os-smart-2.2_3 os-theme-cicada-1.34_3 os-theme-rebellion-1.8.9 os-theme-vicuna-1.45_1 os-tor-1.9 os-upnp-1.5_4 os-wireguard-2.4_1 os-wol-2.4_2
Time Thu, 02 Nov 2023 13:46:12 +0100
OpenSSL 1.1.1w 11 Sep 2023
Python 3.9.18
PHP 8.2.11
PHP Errors:
[01-Nov-2023 18:51:09 Europe/Berlin] PHP Fatal error: Uncaught TypeError: Unsupported operand types: string / int in /usr/local/etc/inc/system.inc:826
Stack trace:
#0 /usr/local/etc/inc/filter.inc(808): get_memory()
#1 /usr/local/www/widgets/api/plugins/system.inc(94): default_state_size()
#2 /usr/local/www/widgets/api/plugins/system.inc(217): system_api_kernel()
#3 /usr/local/www/widgets/api/get.php(70): system_api()
#4 {main}
thrown in /usr/local/etc/inc/system.inc on line 826
[01-Nov-2023 20:56:50 Europe/Berlin] PHP Fatal error: Uncaught TypeError: Unsupported operand types: string / int in /usr/local/etc/inc/system.inc:826
Stack trace:
#0 /usr/local/etc/inc/filter.inc(808): get_memory()
#1 /usr/local/www/widgets/api/plugins/system.inc(94): default_state_size()
#2 /usr/local/www/widgets/api/plugins/system.inc(217): system_api_kernel()
#3 /usr/local/www/widgets/api/get.php(70): system_api()
#4 {main}
thrown in /usr/local/etc/inc/system.inc on line 826
dmesg.boot:
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.2-RELEASE-p3 stable/23.7-n254818-f155405f505 SMP amd64
FreeBSD clang version 14.0.5 (https://github.com/llvm/llvm-project.git llvmorg-14.0.5-0-gc12386ae247c)
VT(vga): resolution 640x480
CPU: Intel(R) N100 (806.40-MHz K8-class CPU)
Origin="GenuineIntel" Id=0xb06e0 Family=0x6 Model=0xbe Stepping=0
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x7ffafbbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0x239ca7eb<FSGSBASE,TSCADJ,BMI1,AVX2,FDPEXC,SMEP,BMI2,ERMS,INVPCID,NFPUSG,PQE,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PROCTRACE,SHA>
Structured Extended Features2=0x98c007bc<UMIP,PKU,OSPKE,WAITPKG,GFNI,VAES,VPCLMULQDQ,RDPID,MOVDIRI,MOVDIR64B>
Structured Extended Features3=0xfc184410<FSRM,MD_CLEAR,IBT,IBPB,STIBP,L1DFL,ARCH_CAP,CORE_CAP,SSBD>
XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
IA32_ARCH_CAPS=0x180fd6b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO,TAA_NO>
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
TSC: P-state invariant, performance statistics
real memory = 8589934592 (8192 MB)
avail memory = 8030289920 (7658 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I >
WARNING: L3 data cache covers more APIC IDs than a package (7 > 3)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-119
Launching APs: 1 3 2
wlan: mac acl policy registered
random: entropy device external interface
kbd1 at kbdmux0
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0.
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
smbios0: <System Management BIOS> at iomem 0x77c2a000-0x77c2a01e
smbios0: Version: 3.5, BCD Revision: 3.5
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256>
acpi0: <ALASKA A M I >
acpi0: Power Button (fixed)
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 19200000 Hz quality 950
Event timer "HPET" frequency 19200000 Hz quality 550
Event timer "HPET1" frequency 19200000 Hz quality 440
Event timer "HPET2" frequency 19200000 Hz quality 440
Event timer "HPET3" frequency 19200000 Hz quality 440
Event timer "HPET4" frequency 19200000 Hz quality 440
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0x6000-0x603f mem 0x6000000000-0x6000ffffff,0x4000000000-0x400fffffff at device 2.0 on pci0
vgapci0: Boot video device
xhci0: <XHCI (generic) USB 3.0 controller> mem 0x6001110000-0x600111ffff at device 13.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
xhci1: <XHCI (generic) USB 3.0 controller> mem 0x6001100000-0x600110ffff at device 20.0 on pci0
xhci1: 32 bytes context size, 64-bit DMA
usbus1 on xhci1
usbus1: 5.0Gbps Super Speed USB v3.0
pci0: <memory, RAM> at device 20.2 (no driver attached)
pci0: <simple comms> at device 22.0 (no driver attached)
ahci0: <AHCI SATA controller> port 0x6090-0x6097,0x6080-0x6083,0x6060-0x607f mem 0x80b00000-0x80b01fff,0x80b03000-0x80b030ff,0x80b02000-0x80b027ff at device 23.0 on pci0
ahci0: AHCI v1.31 with 2 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
pcib1: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <PCI-PCI bridge> at device 2.0 on pci2
pci3: <PCI bus> on pcib3
em0: <Intel(R) PRO/1000 PT 82571EB/82571GB (Quad Copper)> port 0x4020-0x403f mem 0x809a0000-0x809bffff,0x80980000-0x8099ffff at device 0.0 on pci3
em0: EEPROM V5.10-2
em0: Using 1024 TX descriptors and 1024 RX descriptors
em0: Using an MSI interrupt
em0: Ethernet address: 00:1b:21:b1:f0:09
em0: netmap queues/slots: TX 1/1024, RX 1/1024
em1: <Intel(R) PRO/1000 PT 82571EB/82571GB (Quad Copper)> port 0x4000-0x401f mem 0x80940000-0x8095ffff,0x80920000-0x8093ffff at device 0.1 on pci3
em1: EEPROM V5.10-2
em1: Using 1024 TX descriptors and 1024 RX descriptors
em1: Using an MSI interrupt
em1: Ethernet address: 00:1b:21:b1:f0:08
em1: netmap queues/slots: TX 1/1024, RX 1/1024
pcib4: <PCI-PCI bridge> at device 4.0 on pci2
pci4: <PCI bus> on pcib4
em2: <Intel(R) PRO/1000 PT 82571EB/82571GB (Quad Copper)> port 0x3020-0x303f mem 0x808a0000-0x808bffff,0x80880000-0x8089ffff at device 0.0 on pci4
em2: EEPROM V5.10-2
em2: Using 1024 TX descriptors and 1024 RX descriptors
em2: Using an MSI interrupt
em2: Ethernet address: 00:1b:21:b1:f0:0b
em2: netmap queues/slots: TX 1/1024, RX 1/1024
em3: <Intel(R) PRO/1000 PT 82571EB/82571GB (Quad Copper)> port 0x3000-0x301f mem 0x80840000-0x8085ffff,0x80820000-0x8083ffff at device 0.1 on pci4
em3: EEPROM V5.10-2
em3: Using 1024 TX descriptors and 1024 RX descriptors
em3: Using an MSI interrupt
em3: Ethernet address: 00:1b:21:b1:f0:0a
em3: netmap queues/slots: TX 1/1024, RX 1/1024
pcib5: <ACPI PCI-PCI bridge> at device 28.6 on pci0
pci5: <ACPI PCI bus> on pcib5
re0: <Realtek PCIe GbE Family Controller> port 0x5000-0x50ff mem 0x80a04000-0x80a04fff,0x80a00000-0x80a03fff at device 0.0 on pci5
re0: Using Memory Mapping!
re0: Using 1 MSI-X message
re0: ASPM disabled
re0: version:1.98.00
re0: Ethernet address: 9c:6b:00:26:c6:0a
I'm beginning to suspect these N100/Adler Lake things are broken by design and can only be fixed by microcode updates as discussed here https://forum.opnsense.org/index.php?topic=37055.0 (German thread) and that this in particular is just one of the friendlier side effects.
for documentation purposes
the following seems to fixed it for me: https://forum.opnsense.org/index.php?topic=36139.0
@xtrcode thanks for reporting back, let's close this one
@AdSchellevis do we want to roll back the change done?
@fichtner nah, converting 0 to '' isn't great anyway
Think I may have misread before. It's checking the array which contains a sysctl to be returned. If there is no sysctl we don't know what the caller expected. The code makes no difference.
yes, so get_syctl() should always return the array so it it's probably safe, but would it make more sense to return "null" here? I don't know if sysctls can be empty but it may gloss over a reading that way getting an expected (but empty) result.
ok, reverting is fine as well.
ok please don't hate me for 9d818e5 :) can leave this on devel
Important notices
Before you add a new report, we ask you kindly to acknowledge the following:
Describe the bug Since upgrading to an Intel N100 SoC (Asrock N100M) the system crashes daily. Probably related to the following pfSense bug: https://redmine.pfsense.org/issues/14648 Workaround: https://redmine.pfsense.org/projects/pfsense/repository/2/revisions/aed18fb07d387c90942b729c02fe460064310f5e/diff/src/usr/local/www/includes/functions.inc.php
To Reproduce
Run OPNsense on new Intel NXXX SoC
Expected behavior
No crashes
Describe alternatives you considered
Currently testing the following changes in system.inc:103-108
Screenshots
none
Relevant log files
Additional context
It appears to be related to the following pfsense bug: https://redmine.pfsense.org/issues/14648
Environment
Software version used and hardware type if relevant, e.g.:
OPNsense 23.7.7_3 (amd64, commit: f464e1901) Intel N100 (Board: ASROCK N100M) Network Intel® Quad NIC 82571EB/82571GB