Nuclei Crashes on a huge list of urls

Nuclei version:

2.7.7

Current Behavior:

When running nuclei on a list of targets more than 5K, it seems to crash after running for 10 minutes.

Expected Behavior:

It should not crash.

Steps To Reproduce:

nuclei -l urls.txt -t any.yaml -rl 40 -o any.txt
KILLED message appears after a while on bash.

Anything else:

Hi Team,

Hope All are going well !!

Yes observed the same as @IfrahIman said and I'm about to create this one !!

Though tried with all optimizations/rate-limits, Running file with 5k+ hosts nuclei is getting crashed/killed even on high VPS with 8 GB / 4 AMD vCPUs / 160 GB Disk / NYC3 - Debian 11 x64

Thanks & Regards, @zy9ard3

If one of you shares the dmesg output to understand why the kernel killed the process and a pprof profile for the process, I could look into this issue. https://stackoverflow.com/a/20672691

Hi @IfrahIman @zy9ard3

If your processes get killed at the end of the session, it is possible that you are using socket activation and it gets killed by systemd when it notices that the SSH session process exited. In that case there are two solutions. One is to avoid using socket activation by using ssh.service instead of ssh.socket. The other is to set KillMode=process in the Service section of ssh@.service.

The KillMode=process setting may also be useful with the classic ssh.service, as it avoids killing the SSH session process or the screen or tmux processes when the server gets stopped or restarted.

Could be that the server is deleting the tmux socket in /tmp. You can pass a custom socket directory when starting tmux.

Please see the following reference: http://rubenvarela.com/guides/recovering-tmux-when-its-socket-was-deleted

This is happening because nuclei eats up all the ram and hence gets killed/crashes. I have shared more info with screenshots on discord server. tested on different servers with different settings. all resulted in same outcome.

Having a memory profile would be super helpful. If you are kind enough to share one + the corresponding dmesg output, ping me and I'll be happy to contribute a fix for it. 😃

Guys I think this might have something to do with using the -o out.txt flag. I used nuclei without this flag and it seemed to run fine? please verify @demon1k @zy9ard3

Edit:

No it still crashed with the -o flag.

i don't use -o flag. i use -silent and let the 'notify' tool send output on my slack

same issue

nuclei -l test.txt -t /home/hal/nuclei-templates/

It just stops, never runs

If only one of you shared the dmesg log and a memory profile of the process, we could build a fix for this bug.

Hey there!

I have attached the dmesg log

[    0.000000] Linux version 5.15.0-46-generic (buildd@lcy02-amd64-115) (gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #49-Ubuntu SMP Thu Aug 4 18:03:25 UTC 2022 (Ubuntu 5.15.0-46.49-generic 5.15.39)
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.0-46-generic root=UUID=136735fa-5cc1-470f-9359-ee736e42f844 ro console=tty1 console=ttyS0 net.ifnames=0 biosdevname=0
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Hygon HygonGenuine
[    0.000000]   Centaur CentaurHauls
[    0.000000]   zhaoxin   Shanghai  
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: xstate_offset[5]: 1088, xstate_sizes[5]:   64
[    0.000000] x86/fpu: xstate_offset[6]: 1152, xstate_sizes[6]:  512
[    0.000000] x86/fpu: xstate_offset[7]: 1664, xstate_sizes[7]: 1024
[    0.000000] x86/fpu: xstate_offset[9]: 2688, xstate_sizes[9]:    8
[    0.000000] x86/fpu: Enabled xstate features 0x2e7, context size is 2696 bytes, using 'standard' format.
[    0.000000] signal: max sigframe size: 3632
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000003ffd7fff] usable
[    0.000000] BIOS-e820: [mem 0x000000003ffd8000-0x000000003fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.8 present.
[    0.000000] DMI: DigitalOcean Droplet/Droplet, BIOS 20171212 12/12/2017
[    0.000000] Hypervisor detected: KVM
[    0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
[    0.000000] kvm-clock: cpu 0, msr 31e01001, primary cpu clock
[    0.000003] kvm-clock: using sched offset of 1504620644 cycles
[    0.000010] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[    0.000018] tsc: Detected 2494.140 MHz processor
[    0.000832] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000835] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000842] last_pfn = 0x3ffd8 max_arch_pfn = 0x400000000
[    0.000909] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT  
[    0.010275] found SMP MP-table at [mem 0x000f5c60-0x000f5c6f]
[    0.010383] Using GB pages for direct mapping
[    0.010511] RAMDISK: [mem 0x34da7000-0x366cafff]
[    0.010539] ACPI: Early table checksum verification disabled
[    0.010587] ACPI: RSDP 0x00000000000F5A50 000014 (v00 BOCHS )
[    0.010606] ACPI: RSDT 0x000000003FFE1930 000038 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
[    0.010617] ACPI: FACP 0x000000003FFE172C 000074 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
[    0.010629] ACPI: DSDT 0x000000003FFE0040 0016EC (v01 BOCHS  BXPC     00000001 BXPC 00000001)
[    0.010633] ACPI: FACS 0x000000003FFE0000 000040
[    0.010638] ACPI: APIC 0x000000003FFE17A0 000078 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
[    0.010642] ACPI: HPET 0x000000003FFE1818 000038 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
[    0.010647] ACPI: SRAT 0x000000003FFE1850 0000B8 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
[    0.010652] ACPI: WAET 0x000000003FFE1908 000028 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
[    0.010655] ACPI: Reserving FACP table memory at [mem 0x3ffe172c-0x3ffe179f]
[    0.010657] ACPI: Reserving DSDT table memory at [mem 0x3ffe0040-0x3ffe172b]
[    0.010658] ACPI: Reserving FACS table memory at [mem 0x3ffe0000-0x3ffe003f]
[    0.010660] ACPI: Reserving APIC table memory at [mem 0x3ffe17a0-0x3ffe1817]
[    0.010661] ACPI: Reserving HPET table memory at [mem 0x3ffe1818-0x3ffe184f]
[    0.010662] ACPI: Reserving SRAT table memory at [mem 0x3ffe1850-0x3ffe1907]
[    0.010663] ACPI: Reserving WAET table memory at [mem 0x3ffe1908-0x3ffe192f]
[    0.010877] SRAT: PXM 0 -> APIC 0x00 -> Node 0
[    0.010882] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff]
[    0.010890] ACPI: SRAT: Node 0 PXM 0 [mem 0x00100000-0x3fffffff]
[    0.010893] NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0x3ffd7fff] -> [mem 0x00000000-0x3ffd7fff]
[    0.010910] NODE_DATA(0) allocated [mem 0x3ffae000-0x3ffd7fff]
[    0.011400] Zone ranges:
[    0.011406]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.011413]   DMA32    [mem 0x0000000001000000-0x000000003ffd7fff]
[    0.011415]   Normal   empty
[    0.011417]   Device   empty
[    0.011418] Movable zone start for each node
[    0.011424] Early memory node ranges
[    0.011425]   node   0: [mem 0x0000000000001000-0x000000000009efff]
[    0.011427]   node   0: [mem 0x0000000000100000-0x000000003ffd7fff]
[    0.011429] Initmem setup node 0 [mem 0x0000000000001000-0x000000003ffd7fff]
[    0.011449] On node 0, zone DMA: 1 pages in unavailable ranges
[    0.011619] On node 0, zone DMA: 97 pages in unavailable ranges
[    0.022783] On node 0, zone DMA32: 40 pages in unavailable ranges
[    0.023199] ACPI: PM-Timer IO Port: 0x608
[    0.023229] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[    0.023275] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
[    0.023280] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.023283] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
[    0.023290] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.023304] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
[    0.023306] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
[    0.023311] ACPI: Using ACPI (MADT) for SMP configuration information
[    0.023313] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[    0.023323] TSC deadline timer available
[    0.023325] smpboot: Allowing 1 CPUs, 0 hotplug CPUs
[    0.023366] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.023369] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x0009ffff]
[    0.023371] PM: hibernation: Registered nosave memory: [mem 0x000a0000-0x000effff]
[    0.023372] PM: hibernation: Registered nosave memory: [mem 0x000f0000-0x000fffff]
[    0.023375] [mem 0x40000000-0xfeffbfff] available for PCI devices
[    0.023376] Booting paravirtualized kernel on KVM
[    0.023380] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[    0.023393] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1
[    0.024968] percpu: Embedded 60 pages/cpu s208896 r8192 d28672 u2097152
[    0.024982] pcpu-alloc: s208896 r8192 d28672 u2097152 alloc=1*2097152
[    0.024985] pcpu-alloc: [0] 0 
[    0.025015] kvm-guest: stealtime: cpu 0, msr 3ec32080
[    0.025020] kvm-guest: PV spinlocks disabled, no host support
[    0.025030] Built 1 zonelists, mobility grouping on.  Total pages: 257752
[    0.025033] Policy zone: DMA32
[    0.025035] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.0-46-generic root=UUID=136735fa-5cc1-470f-9359-ee736e42f844 ro console=tty1 console=ttyS0 net.ifnames=0 biosdevname=0
[    0.025300] Unknown kernel command line parameters "BOOT_IMAGE=/boot/vmlinuz-5.15.0-46-generic biosdevname=0", will be passed to user space.
[    0.026114] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.026512] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.026574] mem auto-init: stack:off, heap alloc:on, heap free:off
[    0.029574] Memory: 957884K/1048024K available (16393K kernel code, 4382K rwdata, 10800K rodata, 3224K init, 6572K bss, 89880K reserved, 0K cma-reserved)
[    0.029605] random: get_random_u64 called from kmem_cache_open+0x2b/0x330 with crng_init=0
[    0.029820] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.029850] ftrace: allocating 50468 entries in 198 pages
[    0.061625] ftrace: allocated 198 pages with 4 groups
[    0.061986] rcu: Hierarchical RCU implementation.
[    0.061990] rcu:     RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=1.
[    0.061993]  Rude variant of Tasks RCU enabled.
[    0.061993]  Tracing variant of Tasks RCU enabled.
[    0.061995] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.061996] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[    0.066717] NR_IRQS: 524544, nr_irqs: 256, preallocated irqs: 16
[    0.066956] random: crng done (trusting CPU's manufacturer)
[    0.080180] Console: colour VGA+ 80x25
[    0.115770] printk: console [tty1] enabled
[    0.194764] printk: console [ttyS0] enabled
[    0.195482] ACPI: Core revision 20210730
[    0.196266] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns
[    0.197943] APIC: Switch to symmetric I/O mode setup
[    0.198991] x2apic enabled
[    0.199656] Switched APIC routing to physical x2apic.
[    0.201603] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.202690] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x23f39a1d859, max_idle_ns: 440795326830 ns
[    0.204290] Calibrating delay loop (skipped) preset value.. 4988.28 BogoMIPS (lpj=9976560)
[    0.205858] pid_max: default: 32768 minimum: 301
[    0.208332] LSM: Security Framework initializing
[    0.209126] landlock: Up and running.
[    0.210022] Yama: becoming mindful.
[    0.210952] AppArmor: AppArmor initialized
[    0.211793] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.212284] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.213256] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
[    0.214394] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0
[    0.216310] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[    0.218262] Spectre V2 : Mitigation: Enhanced IBRS
[    0.220291] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
[    0.222094] RETBleed: Mitigation: Enhanced IBRS
[    0.223302] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
[    0.224291] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp
[    0.228303] MMIO Stale Data: Vulnerable: Clear CPU buffers attempted, no microcode
[    0.259831] Freeing SMP alternatives memory: 40K
[    0.260576] smpboot: CPU0: DO-Premium-Intel (family: 0x6, model: 0x55, stepping: 0x7)
[    0.262198] Performance Events: unsupported p6 CPU model 85 no PMU driver, software events only.
[    0.263653] rcu: Hierarchical SRCU implementation.
[    0.264901] NMI watchdog: Perf NMI watchdog permanently disabled
[    0.265909] smp: Bringing up secondary CPUs ...
[    0.266620] smp: Brought up 1 node, 1 CPU
[    0.267266] smpboot: Max logical packages: 1
[    0.267972] smpboot: Total of 1 processors activated (4988.28 BogoMIPS)
[    0.268620] devtmpfs: initialized
[    0.270263] x86/mm: Memory block size: 128MB
[    0.271308] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.272305] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[    0.273259] pinctrl core: initialized pinctrl subsystem
[    0.274358] PM: RTC time: 17:35:27, date: 2022-09-08
[    0.275765] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.276588] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
[    0.277993] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.279572] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.280310] audit: initializing netlink subsys (disabled)
[    0.281650] thermal_sys: Registered thermal governor 'fair_share'
[    0.281653] thermal_sys: Registered thermal governor 'bang_bang'
[    0.282761] thermal_sys: Registered thermal governor 'step_wise'
[    0.283849] thermal_sys: Registered thermal governor 'user_space'
[    0.284291] thermal_sys: Registered thermal governor 'power_allocator'
[    0.285406] EISA bus registered
[    0.286877] cpuidle: using governor ladder
[    0.287448] cpuidle: using governor menu
[    0.288101] ACPI: bus type PCI registered
[    0.288294] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.289463] audit: type=2000 audit(1662658527.101:1): state=initialized audit_enabled=0 res=1
[    0.291267] PCI: Using configuration type 1 for base access
[    0.294146] Kprobes globally optimized
[    0.295159] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.296069] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.298025] ACPI: Added _OSI(Module Device)
[    0.298805] ACPI: Added _OSI(Processor Device)
[    0.299430] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.300066] ACPI: Added _OSI(Processor Aggregator Device)
[    0.300302] ACPI: Added _OSI(Linux-Dell-Video)
[    0.300992] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[    0.301971] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
[    0.303689] ACPI: 1 ACPI AML tables successfully acquired and loaded
[    0.305472] ACPI: Interpreter enabled
[    0.306045] ACPI: PM: (supports S0 S3 S4 S5)
[    0.306646] ACPI: Using IOAPIC for interrupt routing
[    0.307332] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    0.308475] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.311282] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    0.312104] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI EDR HPX-Type3]
[    0.312317] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
[    0.314027] acpiphp: Slot [3] registered
[    0.314606] acpiphp: Slot [4] registered
[    0.315184] acpiphp: Slot [5] registered
[    0.315775] acpiphp: Slot [6] registered
[    0.316316] acpiphp: Slot [7] registered
[    0.316898] acpiphp: Slot [8] registered
[    0.317481] acpiphp: Slot [9] registered
[    0.318105] acpiphp: Slot [10] registered
[    0.318685] acpiphp: Slot [11] registered
[    0.319305] acpiphp: Slot [12] registered
[    0.319933] acpiphp: Slot [13] registered
[    0.320304] acpiphp: Slot [14] registered
[    0.320883] acpiphp: Slot [15] registered
[    0.321475] acpiphp: Slot [16] registered
[    0.322059] acpiphp: Slot [17] registered
[    0.322633] acpiphp: Slot [18] registered
[    0.324331] acpiphp: Slot [19] registered
[    0.324984] acpiphp: Slot [20] registered
[    0.325582] acpiphp: Slot [21] registered
[    0.326231] acpiphp: Slot [22] registered
[    0.326841] acpiphp: Slot [23] registered
[    0.327593] acpiphp: Slot [24] registered
[    0.328320] acpiphp: Slot [25] registered
[    0.328944] acpiphp: Slot [26] registered
[    0.329570] acpiphp: Slot [27] registered
[    0.330170] acpiphp: Slot [28] registered
[    0.330766] acpiphp: Slot [29] registered
[    0.331378] acpiphp: Slot [30] registered
[    0.331964] acpiphp: Slot [31] registered
[    0.332309] PCI host bridge to bus 0000:00
[    0.332897] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.333717] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    0.334700] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    0.335627] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    0.336290] pci_bus 0000:00: root bus resource [mem 0x40000000-0xfebfffff window]
[    0.337303] pci_bus 0000:00: root bus resource [mem 0x100000000-0x17fffffff window]
[    0.338498] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000
[    0.340570] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100
[    0.341934] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180
[    0.345696] pci 0000:00:01.1: reg 0x20: [io  0xc200-0xc20f]
[    0.347713] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io  0x01f0-0x01f7]
[    0.348291] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io  0x03f6]
[    0.349136] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io  0x0170-0x0177]
[    0.350072] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io  0x0376]
[    0.351254] pci 0000:00:01.2: [8086:7020] type 00 class 0x0c0300
[    0.355651] pci 0000:00:01.2: reg 0x20: [io  0xc180-0xc19f]
[    0.357768] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000
[    0.359026] pci 0000:00:01.3: quirk: [io  0x0600-0x063f] claimed by PIIX4 ACPI
[    0.360033] pci 0000:00:01.3: quirk: [io  0x0700-0x070f] claimed by PIIX4 SMB
[    0.360958] pci 0000:00:02.0: [1b36:0100] type 00 class 0x030000
[    0.366332] pci 0000:00:02.0: reg 0x10: [mem 0xf8000000-0xfbffffff]
[    0.369824] pci 0000:00:02.0: reg 0x14: [mem 0xfc000000-0xfcffffff]
[    0.373805] pci 0000:00:02.0: reg 0x18: [mem 0xfd010000-0xfd011fff]
[    0.379162] pci 0000:00:02.0: reg 0x1c: [io  0xc1a0-0xc1bf]
[    0.386775] pci 0000:00:02.0: reg 0x30: [mem 0xfd000000-0xfd00ffff pref]
[    0.388161] pci 0000:00:02.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    0.389847] pci 0000:00:03.0: [1af4:1000] type 00 class 0x020000
[    0.392294] pci 0000:00:03.0: reg 0x10: [io  0xc1c0-0xc1df]
[    0.396997] pci 0000:00:03.0: reg 0x14: [mem 0xfd012000-0xfd012fff]
[    0.403398] pci 0000:00:03.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref]
[    0.409882] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000
[    0.412295] pci 0000:00:04.0: reg 0x10: [io  0xc1e0-0xc1ff]
[    0.414792] pci 0000:00:04.0: reg 0x14: [mem 0xfd013000-0xfd013fff]
[    0.421164] pci 0000:00:04.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref]
[    0.425221] pci 0000:00:05.0: [1af4:1004] type 00 class 0x010000
[    0.428985] pci 0000:00:05.0: reg 0x10: [io  0xc100-0xc13f]
[    0.431381] pci 0000:00:05.0: reg 0x14: [mem 0xfd014000-0xfd014fff]
[    0.436082] pci 0000:00:05.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref]
[    0.444472] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000
[    0.448298] pci 0000:00:06.0: reg 0x10: [io  0xc000-0xc07f]
[    0.453101] pci 0000:00:06.0: reg 0x14: [mem 0xfd015000-0xfd015fff]
[    0.458698] pci 0000:00:06.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref]
[    0.466557] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000
[    0.468293] pci 0000:00:07.0: reg 0x10: [io  0xc080-0xc0ff]
[    0.470617] pci 0000:00:07.0: reg 0x14: [mem 0xfd016000-0xfd016fff]
[    0.477116] pci 0000:00:07.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref]
[    0.483743] pci 0000:00:08.0: [1af4:1002] type 00 class 0x00ff00
[    0.485178] pci 0000:00:08.0: reg 0x10: [io  0xc140-0xc17f]
[    0.489985] pci 0000:00:08.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref]
[    0.504822] ACPI: PCI: Interrupt link LNKA configured for IRQ 10
[    0.505767] ACPI: PCI: Interrupt link LNKB configured for IRQ 10
[    0.506723] ACPI: PCI: Interrupt link LNKC configured for IRQ 11
[    0.508470] ACPI: PCI: Interrupt link LNKD configured for IRQ 11
[    0.509350] ACPI: PCI: Interrupt link LNKS configured for IRQ 9
[    0.510937] iommu: Default domain type: Translated 
[    0.511637] iommu: DMA domain TLB invalidation policy: lazy mode 
[    0.512719] SCSI subsystem initialized
[    0.513372] libata version 3.00 loaded.
[    0.513513] pci 0000:00:02.0: vgaarb: setting as boot VGA device
[    0.514325] pci 0000:00:02.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[    0.515480] pci 0000:00:02.0: vgaarb: bridge control possible
[    0.516254] vgaarb: loaded
[    0.516379] ACPI: bus type USB registered
[    0.516989] usbcore: registered new interface driver usbfs
[    0.517750] usbcore: registered new interface driver hub
[    0.518474] usbcore: registered new device driver usb
[    0.519204] pps_core: LinuxPPS API ver. 1 registered
[    0.519881] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.520297] PTP clock support registered
[    0.520977] EDAC MC: Ver: 3.0.0
[    0.522174] NetLabel: Initializing
[    0.522682] NetLabel:  domain hash size = 128
[    0.523292] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    0.524088] NetLabel:  unlabeled traffic allowed by default
[    0.524363] PCI: Using ACPI for IRQ routing
[    0.524955] PCI: pci_cache_line_size set to 64 bytes
[    0.525132] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff]
[    0.525137] e820: reserve RAM buffer [mem 0x3ffd8000-0x3fffffff]
[    0.525390] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[    0.526069] hpet0: 3 comparators, 64-bit 100.000000 MHz counter
[    0.530461] clocksource: Switched to clocksource kvm-clock
[    0.546383] VFS: Disk quotas dquot_6.6.0
[    0.547030] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.548142] AppArmor: AppArmor Filesystem Enabled
[    0.548852] pnp: PnP ACPI init
[    0.549691] pnp: PnP ACPI: found 4 devices
[    0.559978] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    0.561278] NET: Registered PF_INET protocol family
[    0.562091] IP idents hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.566532] tcp_listen_portaddr_hash hash table entries: 512 (order: 1, 8192 bytes, linear)
[    0.575192] TCP established hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.576345] TCP bind hash table entries: 8192 (order: 5, 131072 bytes, linear)
[    0.577339] TCP: Hash tables configured (established 8192 bind 8192)
[    0.578375] MPTCP token hash table entries: 1024 (order: 2, 24576 bytes, linear)
[    0.579411] UDP hash table entries: 512 (order: 2, 16384 bytes, linear)
[    0.580291] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes, linear)
[    0.581268] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.582059] NET: Registered PF_XDP protocol family
[    0.582752] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[    0.583568] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[    0.584388] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[    0.585311] pci_bus 0000:00: resource 7 [mem 0x40000000-0xfebfffff window]
[    0.586206] pci_bus 0000:00: resource 8 [mem 0x100000000-0x17fffffff window]
[    0.587177] pci 0000:00:01.0: PIIX3: Enabling Passive Release
[    0.587945] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[    0.588755] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[    0.609701] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    0.630764] pci 0000:00:01.2: quirk_usb_early_handoff+0x0/0x160 took 40195 usecs
[    0.631945] PCI: CLS 0 bytes, default 64
[    0.632619] Trying to unpack rootfs image as initramfs...
[    0.636421] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x23f39a1d859, max_idle_ns: 440795326830 ns
[    0.640705] Initialise system trusted keyrings
[    0.641425] Key type blacklist registered
[    0.644379] workingset: timestamp_bits=36 max_order=18 bucket_order=0
[    0.646628] zbud: loaded
[    0.647360] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.652604] fuse: init (API version 7.34)
[    0.653469] integrity: Platform Keyring initialized
[    0.672452] Key type asymmetric registered
[    0.673142] Asymmetric key parser 'x509' registered
[    0.673850] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
[    0.674951] io scheduler mq-deadline registered
[    0.675809] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    0.676973] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[    0.680350] ACPI: button: Power Button [PWRF]
[    0.721867] ACPI: \_SB_.LNKC: Enabled at IRQ 10
[    0.812084] ACPI: \_SB_.LNKA: Enabled at IRQ 10
[    0.859351] ACPI: \_SB_.LNKB: Enabled at IRQ 11
[    0.953118] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[    1.000704] 00:00: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[    1.005869] Linux agpgart interface v0.103
[    1.013275] loop: module loaded
[    1.013996] ata_piix 0000:00:01.1: version 2.13
[    1.020849] scsi host0: ata_piix
[    1.024530] scsi host1: ata_piix
[    1.025228] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc200 irq 14
[    1.026117] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc208 irq 15
[    1.027539] tun: Universal TUN/TAP device driver, 1.6
[    1.028303] PPP generic driver version 2.4.2
[    1.028990] VFIO - User Level meta-driver version: 0.3
[    1.030037] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    1.030917] ehci-pci: EHCI PCI platform driver
[    1.031567] ehci-platform: EHCI generic platform driver
[    1.036350] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    1.037255] ohci-pci: OHCI PCI platform driver
[    1.037913] ohci-platform: OHCI generic platform driver
[    1.038633] uhci_hcd: USB Universal Host Controller Interface driver
[    1.085855] uhci_hcd 0000:00:01.2: UHCI Host Controller
[    1.086645] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1
[    1.087741] uhci_hcd 0000:00:01.2: detected 2 ports
[    1.088625] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c180
[    1.092695] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 5.15
[    1.093912] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.094923] usb usb1: Product: UHCI Host Controller
[    1.095598] usb usb1: Manufacturer: Linux 5.15.0-46-generic uhci_hcd
[    1.096518] usb usb1: SerialNumber: 0000:00:01.2
[    1.101083] hub 1-0:1.0: USB hub found
[    1.101823] hub 1-0:1.0: 2 ports detected
[    1.102649] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
[    1.108805] serio: i8042 KBD port at 0x60,0x64 irq 1
[    1.109548] serio: i8042 AUX port at 0x60,0x64 irq 12
[    1.110439] mousedev: PS/2 mouse device common for all mice
[    1.111464] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1
[    1.112919] rtc_cmos 00:03: RTC can wake from S4
[    1.116832] rtc_cmos 00:03: registered as rtc0
[    1.117837] rtc_cmos 00:03: setting system clock to 2022-09-08T17:35:27 UTC (1662658527)
[    1.119072] rtc_cmos 00:03: alarms up to one day, y3k, 242 bytes nvram, hpet irqs
[    1.120156] i2c_dev: i2c /dev entries driver
[    1.124563] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.
[    1.126321] device-mapper: uevent: version 1.0.3
[    1.132743] device-mapper: ioctl: 4.45.0-ioctl (2021-03-22) initialised: dm-devel@redhat.com
[    1.134198] platform eisa.0: Probing EISA bus 0
[    1.134865] platform eisa.0: EISA: Cannot allocate resource for mainboard
[    1.135838] platform eisa.0: Cannot allocate resource for EISA slot 1
[    1.140388] platform eisa.0: Cannot allocate resource for EISA slot 2
[    1.141408] platform eisa.0: Cannot allocate resource for EISA slot 3
[    1.142256] platform eisa.0: Cannot allocate resource for EISA slot 4
[    1.143112] platform eisa.0: Cannot allocate resource for EISA slot 5
[    1.144036] platform eisa.0: Cannot allocate resource for EISA slot 6
[    1.148348] platform eisa.0: Cannot allocate resource for EISA slot 7
[    1.149461] platform eisa.0: Cannot allocate resource for EISA slot 8
[    1.150371] platform eisa.0: EISA: Detected 0 cards
[    1.151097] intel_pstate: CPU model not supported
[    1.156755] ledtrig-cpu: registered to indicate activity on CPUs
[    1.157991] drop_monitor: Initializing network drop monitor service
[    1.159165] NET: Registered PF_INET6 protocol family
[    1.263874] Freeing initrd memory: 25744K
[    1.272966] Segment Routing with IPv6
[    1.273584] In-situ OAM (IOAM) with IPv6
[    1.274217] NET: Registered PF_PACKET protocol family
[    1.275094] Key type dns_resolver registered
[    1.276008] No MBM correction factor available
[    1.276781] IPI shorthand broadcast: enabled
[    1.277602] sched_clock: Marking stable (1143387876, 133364991)->(1337763944, -61011077)
[    1.278881] registered taskstats version 1
[    1.279530] Loading compiled-in X.509 certificates
[    1.280917] Loaded X.509 cert 'Build time autogenerated kernel key: 8bf8b3bafb8f11680b7c33e638b6adc8d8cce374'
[    1.282717] Loaded X.509 cert 'Canonical Ltd. Live Patch Signing: 14df34d1a87cf37625abec039ef2bf521249b969'
[    1.284563] Loaded X.509 cert 'Canonical Ltd. Kernel Module Signing: 88f752e560a1e0737e31163a466ad7b70a850c19'
[    1.285965] blacklist: Loading compiled-in revocation X.509 certificates
[    1.286878] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing: 61482aa2830d0ab2ad5af10b7250da9033ddcef0'
[    1.288407] zswap: loaded using pool lzo/zbud
[    1.289190] Key type ._fscrypt registered
[    1.289763] Key type .fscrypt registered
[    1.290320] Key type fscrypt-provisioning registered
[    1.295152] Key type encrypted registered
[    1.295783] AppArmor: AppArmor sha1 policy hashing enabled
[    1.296556] ima: No TPM chip found, activating TPM-bypass!
[    1.297318] Loading compiled-in module X.509 certificates
[    1.298539] Loaded X.509 cert 'Build time autogenerated kernel key: 8bf8b3bafb8f11680b7c33e638b6adc8d8cce374'
[    1.301525] ima: Allocated hash algorithm: sha1
[    1.302250] ima: No architecture policies found
[    1.302970] evm: Initialising EVM extended attributes:
[    1.303721] evm: security.selinux
[    1.304245] evm: security.SMACK64
[    1.304811] evm: security.SMACK64EXEC
[    1.305446] evm: security.SMACK64TRANSMUTE
[    1.306044] evm: security.SMACK64MMAP
[    1.306614] evm: security.apparmor
[    1.307191] evm: security.ima
[    1.307655] evm: security.capability
[    1.308178] evm: HMAC attrs: 0x1
[    1.309103] PM:   Magic number: 6:452:592
[    1.309937] RAS: Correctable Errors collector initialized.
[    1.313227] Freeing unused decrypted memory: 2036K
[    1.315798] Freeing unused kernel image (initmem) memory: 3224K
[    1.320352] Write protecting the kernel read-only data: 30720k
[    1.322823] Freeing unused kernel image (text/rodata gap) memory: 2036K
[    1.325327] Freeing unused kernel image (rodata/data gap) memory: 1488K
[    1.339823] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    1.340752] Run /init as init process
[    1.341303]   with arguments:
[    1.341305]     /init
[    1.341306]   with environment:
[    1.341307]     HOME=/
[    1.341307]     TERM=linux
[    1.341308]     BOOT_IMAGE=/boot/vmlinuz-5.15.0-46-generic
[    1.341309]     biosdevname=0
[    1.552139] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4
[    1.553571] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3
[    1.557324] cryptd: max_cpu_qlen set to 1000
[    1.559244] scsi host2: Virtio SCSI HBA
[    1.564245] virtio_blk virtio3: [vda] 52428800 512-byte logical blocks (26.8 GB/25.0 GiB)
[    1.566626] AVX2 version of gcm_enc/dec engaged.
[    1.576834] AES CTR mode by8 optimization enabled
[    1.590678]  vda: vda1 vda14 vda15
[    1.598388] virtio_blk virtio4: [vdb] 932 512-byte logical blocks (477 kB/466 KiB)
[    1.856298] raid6: avx512x4 gen() 17965 MB/s
[    1.924293] raid6: avx512x4 xor()  7343 MB/s
[    1.992294] raid6: avx512x2 gen() 17722 MB/s
[    2.068308] raid6: avx512x2 xor() 23979 MB/s
[    2.136325] raid6: avx512x1 gen() 17568 MB/s
[    2.204297] raid6: avx512x1 xor() 22453 MB/s
[    2.272292] raid6: avx2x4   gen() 17189 MB/s
[    2.340296] raid6: avx2x4   xor()  7045 MB/s
[    2.408314] raid6: avx2x2   gen() 17764 MB/s
[    2.476307] raid6: avx2x2   xor() 16270 MB/s
[    2.544294] raid6: avx2x1   gen() 13433 MB/s
[    2.612313] raid6: avx2x1   xor()  9668 MB/s
[    2.680291] raid6: sse2x4   gen()  9266 MB/s
[    2.748290] raid6: sse2x4   xor()  6137 MB/s
[    2.816294] raid6: sse2x2   gen() 10633 MB/s
[    2.884305] raid6: sse2x2   xor()  6225 MB/s
[    2.952294] raid6: sse2x1   gen()  9789 MB/s
[    3.020309] raid6: sse2x1   xor()  4960 MB/s
[    3.021089] raid6: using algorithm avx512x4 gen() 17965 MB/s
[    3.021858] raid6: .... xor() 7343 MB/s, rmw enabled
[    3.022546] raid6: using avx512x2 recovery algorithm
[    3.025036] xor: automatically using best checksumming function   avx       
[    3.027107] async_tx: api initialized (async)
[    3.121937] Btrfs loaded, crc32c=crc32c-intel, zoned=yes, fsverity=yes
[    3.205769] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[    3.345211] systemd[1]: Inserted module 'autofs4'
[    3.361300] systemd[1]: systemd 251.3-1 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    3.365440] systemd[1]: Detected virtualization kvm.
[    3.366127] systemd[1]: Detected architecture x86-64.
[    3.371375] systemd[1]: Hostname set to <trent>.
[    3.614663] systemd[1]: Queued start job for default target Graphical Interface.
[    3.617302] systemd[1]: Created slice Slice /system/getty.
[    3.619587] systemd[1]: Created slice Slice /system/modprobe.
[    3.622009] systemd[1]: Created slice Slice /system/serial-getty.
[    3.624331] systemd[1]: Created slice Slice /system/systemd-fsck.
[    3.626620] systemd[1]: Created slice User and Session Slice.
[    3.628538] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[    3.631029] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[    3.633632] systemd[1]: Reached target Local Integrity Protected Volumes.
[    3.635631] systemd[1]: Reached target Slice Units.
[    3.637282] systemd[1]: Reached target Swaps.
[    3.640329] systemd[1]: Reached target Local Verity Protected Volumes.
[    3.642572] systemd[1]: Listening on Device-mapper event daemon FIFOs.
[    3.644987] systemd[1]: Listening on LVM2 poll daemon socket.
[    3.646907] systemd[1]: Listening on multipathd control socket.
[    3.648963] systemd[1]: Listening on Syslog Socket.
[    3.650801] systemd[1]: Listening on fsck to fsckd communication Socket.
[    3.652926] systemd[1]: Listening on initctl Compatibility Named Pipe.
[    3.655109] systemd[1]: Listening on Journal Audit Socket.
[    3.657027] systemd[1]: Listening on Journal Socket (/dev/log).
[    3.659076] systemd[1]: Listening on Journal Socket.
[    3.660932] systemd[1]: Listening on Network Service Netlink Socket.
[    3.663280] systemd[1]: Listening on udev Control Socket.
[    3.665819] systemd[1]: Listening on udev Kernel Socket.
[    3.669236] systemd[1]: Mounting Huge Pages File System...
[    3.674943] systemd[1]: Mounting POSIX Message Queue File System...
[    3.679825] systemd[1]: Mounting Kernel Debug File System...
[    3.684561] systemd[1]: Mounting Kernel Trace File System...
[    3.690415] systemd[1]: Starting Set the console keyboard layout...
[    3.697805] systemd[1]: Starting Create List of Static Device Nodes...
[    3.705955] systemd[1]: Starting Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
[    3.711339] systemd[1]: LXD - agent was skipped because of a failed condition check (ConditionPathExists=/dev/virtio-ports/org.linuxcontainers.lxd).
[    3.715483] systemd[1]: Starting Load Kernel Module configfs...
[    3.724213] systemd[1]: Starting Load Kernel Module drm...
[    3.738089] systemd[1]: Starting Load Kernel Module fuse...
[    3.743056] systemd[1]: OpenVSwitch configuration for cleanup was skipped because of a failed condition check (ConditionFileIsExecutable=/usr/bin/ovs-vsctl).
[    3.755653] systemd[1]: File System Check on Root Device was skipped because of a failed condition check (ConditionPathExists=!/run/initramfs/fsck-root).
[    3.763654] systemd[1]: Starting Journal Service...
[    3.772448] systemd[1]: Starting Load Kernel Modules...
[    3.784722] systemd[1]: Starting Remount Root and Kernel File Systems...
[    3.787742] systemd[1]: Repartition Root Disk was skipped because all trigger condition checks failed.
[    3.794265] systemd[1]: Starting Coldplug All udev Devices...
[    3.808400] systemd[1]: Mounted Huge Pages File System.
[    3.811247] systemd[1]: Mounted POSIX Message Queue File System.
[    3.818473] systemd[1]: Mounted Kernel Debug File System.
[    3.828956] systemd[1]: Mounted Kernel Trace File System.
[    3.840447] systemd[1]: Finished Set the console keyboard layout.
[    3.847813] systemd[1]: Finished Create List of Static Device Nodes.
[    3.855110] systemd[1]: modprobe@configfs.service: Deactivated successfully.
[    3.864824] systemd[1]: Finished Load Kernel Module configfs.
[    3.872563] systemd[1]: modprobe@fuse.service: Deactivated successfully.
[    3.874386] systemd[1]: Finished Load Kernel Module fuse.
[    3.884703] systemd[1]: Mounting FUSE Control File System...
[    3.895946] systemd[1]: Mounting Kernel Configuration File System...
[    3.903124] systemd[1]: Mounted FUSE Control File System.
[    3.910459] systemd[1]: Finished Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
[    3.914448] EXT4-fs (vda1): re-mounted. Opts: discard,errors=remount-ro. Quota mode: none.
[    3.920280] systemd[1]: modprobe@drm.service: Deactivated successfully.
[    3.928672] systemd[1]: Finished Load Kernel Module drm.
[    3.931039] systemd[1]: Finished Load Kernel Modules.
[    3.938419] systemd[1]: Finished Remount Root and Kernel File Systems.
[    3.946194] systemd[1]: Mounted Kernel Configuration File System.
[    3.958952] systemd[1]: Starting Device-Mapper Multipath Device Controller...
[    3.962855] systemd[1]: Platform Persistent Storage Archival was skipped because of a failed condition check (ConditionDirectoryNotEmpty=/sys/fs/pstore).
[    3.970952] systemd[1]: Starting Load/Save Random Seed...
[    3.978476] systemd[1]: Starting Apply Kernel Variables...
[    3.983809] systemd[1]: Starting Create System Users...
[    4.021875] systemd[1]: Finished Apply Kernel Variables.
[    4.023745] systemd[1]: Finished Load/Save Random Seed.
[    4.025092] systemd[1]: First Boot Complete was skipped because of a failed condition check (ConditionFirstBoot=yes).
[    4.037534] systemd[1]: Finished Create System Users.
[    4.040734] systemd[1]: Starting Create Static Device Nodes in /dev...
[    4.065091] systemd[1]: Started Journal Service.
[    4.108493] systemd-journald[336]: Received client request to flush runtime journal.
[    4.156666] loop0: detected capacity change from 0 to 8
[    4.161731] loop1: detected capacity change from 0 to 288968
[    4.166387] loop2: detected capacity change from 0 to 291304
[    4.178024] loop3: detected capacity change from 0 to 126888
[    4.183245] loop4: detected capacity change from 0 to 820832
[    4.188257] loop5: detected capacity change from 0 to 709216
[    4.194813] loop6: detected capacity change from 0 to 187776
[    4.197722] loop7: detected capacity change from 0 to 126896
[    4.219000] loop8: detected capacity change from 0 to 163736
[    4.219441] loop9: detected capacity change from 0 to 210912
[    4.219647] loop10: detected capacity change from 0 to 96176
[    5.079682] audit: type=1400 audit(1662658531.456:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=473 comm="apparmor_parser"
[    5.101071] audit: type=1400 audit(1662658531.480:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=474 comm="apparmor_parser"
[    5.101076] audit: type=1400 audit(1662658531.480:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=474 comm="apparmor_parser"
[    5.150730] audit: type=1400 audit(1662658531.528:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=475 comm="apparmor_parser"
[    5.150736] audit: type=1400 audit(1662658531.528:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=475 comm="apparmor_parser"
[    5.150738] audit: type=1400 audit(1662658531.528:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=475 comm="apparmor_parser"
[    5.150741] audit: type=1400 audit(1662658531.528:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/{,usr/}sbin/dhclient" pid=475 comm="apparmor_parser"
[    5.162572] audit: type=1400 audit(1662658531.540:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/man" pid=477 comm="apparmor_parser"
[    5.162579] audit: type=1400 audit(1662658531.540:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_filter" pid=477 comm="apparmor_parser"
[    5.321220] kauditd_printk_skb: 4 callbacks suppressed
[    5.321224] audit: type=1400 audit(1662658531.700:15): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/snap/snapd/16292/usr/lib/snapd/snap-confine" pid=492 comm="apparmor_parser"
[    5.321228] audit: type=1400 audit(1662658531.700:16): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/snap/snapd/16292/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=492 comm="apparmor_parser"
[    5.347557] audit: type=1400 audit(1662658531.724:17): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap-update-ns.chromium" pid=493 comm="apparmor_parser"
[    5.351582] audit: type=1400 audit(1662658531.728:18): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap-update-ns.lxd" pid=495 comm="apparmor_parser"
[    5.397673] audit: type=1400 audit(1662658531.776:19): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.chromium.chromedriver" pid=496 comm="apparmor_parser"
[    5.435368] audit: type=1400 audit(1662658531.812:20): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.chromium.chromium" pid=497 comm="apparmor_parser"
[    5.454183] audit: type=1400 audit(1662658531.832:21): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.chromium.hook.configure" pid=500 comm="apparmor_parser"
[    5.464957] audit: type=1400 audit(1662658531.844:22): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.activate" pid=502 comm="apparmor_parser"
[    5.478223] audit: type=1400 audit(1662658531.856:23): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.benchmark" pid=503 comm="apparmor_parser"
[    5.497506] audit: type=1400 audit(1662658531.876:24): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.buginfo" pid=506 comm="apparmor_parser"
[    6.464332] loop11: detected capacity change from 0 to 8
[78910.277925] loop11: detected capacity change from 0 to 129480
[88493.348455] nuclei invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[88493.348471] CPU: 0 PID: 12460 Comm: nuclei Not tainted 5.15.0-46-generic #49-Ubuntu
[88493.348475] Hardware name: DigitalOcean Droplet/Droplet, BIOS 20171212 12/12/2017
[88493.348480] Call Trace:
[88493.348482]  <TASK>
[88493.348498]  show_stack+0x52/0x5c
[88493.348532]  dump_stack_lvl+0x4a/0x63
[88493.348549]  dump_stack+0x10/0x16
[88493.348551]  dump_header+0x53/0x228
[88493.348558]  oom_kill_process.cold+0xb/0x10
[88493.348560]  out_of_memory+0x106/0x2e0
[88493.348580]  __alloc_pages_slowpath.constprop.0+0x97e/0xa40
[88493.348587]  __alloc_pages+0x311/0x330
[88493.348589]  alloc_pages+0x9e/0x1e0
[88493.348594]  __page_cache_alloc+0x7e/0x90
[88493.348596]  pagecache_get_page+0x152/0x590
[88493.348598]  ? page_cache_ra_unbounded+0x166/0x210
[88493.348600]  filemap_fault+0x488/0xab0
[88493.348604]  ? filemap_map_pages+0x309/0x400
[88493.348606]  __do_fault+0x39/0x120
[88493.348610]  do_read_fault+0xeb/0x160
[88493.348612]  do_fault+0xa0/0x2e0
[88493.348614]  handle_pte_fault+0x1cd/0x240
[88493.348616]  __handle_mm_fault+0x3c7/0x700
[88493.348619]  handle_mm_fault+0xd8/0x2c0
[88493.348621]  do_user_addr_fault+0x1c9/0x670
[88493.348633]  exc_page_fault+0x77/0x170
[88493.348644]  asm_exc_page_fault+0x26/0x30
[88493.348650] RIP: 0033:0x453917
[88493.348661] Code: Unable to access opcode bytes at RIP 0x4538ed.
[88493.348662] RSP: 002b:000000c0000871a8 EFLAGS: 00010212
[88493.348664] RAX: 0000000000068f80 RBX: fffffffffffffffa RCX: 0000000002423ac0
[88493.348666] RDX: 00000000000004ca RSI: 0000000000008003 RDI: 00000000021130a0
[88493.348667] RBP: 000000c0000871b8 R08: 00000000000004cb R09: 00000000021156f8
[88493.348668] R10: 00007fb21f2b7a58 R11: 000000c00fe8b980 R12: 000000c000087260
[88493.348669] R13: 0000000000000000 R14: 000000c000002ea0 R15: ffffffffffffffff
[88493.348672]  </TASK>
[88493.348680] Mem-Info:
[88493.348684] active_anon:235 inactive_anon:205106 isolated_anon:0
                active_file:27 inactive_file:1 isolated_file:0
                unevictable:6946 dirty:0 writeback:0
                slab_reclaimable:5468 slab_unreclaimable:8987
                mapped:2295 shmem:248 pagetables:1136 bounce:0
                kernel_misc_reclaimable:0
                free:12152 free_pcp:234 free_cma:0
[88493.348691] Node 0 active_anon:940kB inactive_anon:820424kB active_file:108kB inactive_file:4kB unevictable:27784kB isolated(anon):0kB isolated(file):0kB mapped:9180kB dirty:0kB writeback:0kB shmem:992kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 4096kB writeback_tmp:0kB kernel_stack:2304kB pagetables:4544kB all_unreclaimable? no
[88493.348695] Node 0 DMA free:4400kB min:720kB low:900kB high:1080kB reserved_highatomic:0KB active_anon:0kB inactive_anon:10712kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[88493.348699] lowmem_reserve[]: 0 920 920 920 920
[88493.348702] Node 0 DMA32 free:44208kB min:44332kB low:55412kB high:66492kB reserved_highatomic:0KB active_anon:940kB inactive_anon:809712kB active_file:108kB inactive_file:4kB unevictable:27784kB writepending:0kB present:1032032kB managed:977352kB mlocked:27784kB bounce:0kB free_pcp:936kB local_pcp:936kB free_cma:0kB
[88493.348706] lowmem_reserve[]: 0 0 0 0 0
[88493.348709] Node 0 DMA: 6*4kB (UM) 11*8kB (UM) 2*16kB (UM) 7*32kB (UM) 5*64kB (UM) 5*128kB (UM) 4*256kB (UM) 0*512kB 2*1024kB (UM) 0*2048kB 0*4096kB = 4400kB
[88493.348719] Node 0 DMA32: 190*4kB (UME) 253*8kB (ME) 141*16kB (ME) 214*32kB (ME) 135*64kB (UME) 67*128kB (ME) 27*256kB (ME) 12*512kB (M) 2*1024kB (UM) 0*2048kB 0*4096kB = 44208kB
[88493.348730] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[88493.348736] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[88493.348737] 2572 total pagecache pages
[88493.348738] 0 pages in swap cache
[88493.348738] Swap cache stats: add 0, delete 0, find 0/0
[88493.348739] Free swap  = 0kB
[88493.348740] Total swap = 0kB
[88493.348741] 262006 pages RAM
[88493.348741] 0 pages HighMem/MovableOnly
[88493.348742] 13828 pages reserved
[88493.348742] 0 pages hwpoisoned
[88493.348743] Tasks state (memory values in pages):
[88493.348743] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[88493.348749] [    336]     0   336    19785      837   188416        0          -250 systemd-journal
[88493.348752] [    345]     0   345    72365     6750   118784        0         -1000 multipathd
[88493.348757] [    353]     0   353     3230     1280    65536        0         -1000 systemd-udevd
[88493.348759] [    464]   100   464     4540      697    77824        0             0 systemd-network
[88493.348761] [    470]   101   470    11805     7591   131072        0             0 systemd-resolve
[88493.348762] [    471]   103   471    22567      558    77824        0             0 systemd-timesyn
[88493.348766] [    525]   102   525     2469      900    65536        0          -900 dbus-daemon
[88493.348768] [    529]     0   529    58723      651    86016        0             0 polkitd
[88493.348769] [    532]     0   532    55532      609    73728        0             0 rsyslogd
[88493.348771] [    535]     0   535   329600     2640   249856        0          -900 snapd
[88493.348773] [    536]     0   536     4420      613    73728        0             0 systemd-logind
[88493.348774] [    537]     0   537    98352     1006   135168        0             0 udisksd
[88493.348776] [    549]     0   549    79342     1088   106496        0             0 ModemManager
[88493.348777] [    572]     0   572    26982     2894   110592        0             0 unattended-upgr
[88493.348779] [    661]     0   661   324829      210   167936        0          -900 droplet-agent
[88493.348780] [    665]     0   665     1095      573    40960        0             0 cron
[88493.348782] [    685]     0   685      801      219    40960        0             0 agetty
[88493.348783] [    686]     0   686      812      213    49152        0             0 agetty
[88493.348785] [    687]     0   687     3958      979    73728        0         -1000 sshd
[88493.348787] [    727]     0   727     4804     1075    77824        0           100 systemd
[88493.348788] [    728]     0   728    26036      993    90112        0           100 (sd-pam)
[88493.348791] [    763]     0   763     1912      886    57344        0             0 tmux: server
[88493.348792] [    764]     0   764     1804     1091    49152        0             0 bash
[88493.348793] [    837]     0   837     1805     1063    57344        0             0 bash
[88493.348795] [   2534]     0  2534    73685     1060   172032        0             0 packagekitd
[88493.348797] [  10088]     0 10088     4701     1409    73728        0             0 sshd
[88493.348799] [  10095]     0 10095     1403      950    45056        0             0 bash
[88493.348800] [  12092]     0 12092     1805      801    57344        0             0 bash
[88493.348802] [  12457]     0 12457   357387   178058  1650688        0             0 nuclei
[88493.348803] [  12934]     0 12934     4694     1232    77824        0             0 sshd
[88493.348805] [  12942]     0 12942     1403      933    49152        0             0 bash
[88493.348807] [  13049]     0 13049     1403      716    49152        0             0 bash
[88493.348808] [  13051]     0 13051     1436      633    49152        0             0 tmux: client
[88493.348810] [  13332]     0 13332      722      131    45056        0             0 apt.systemd.dai
[88493.348812] [  13337]     0 13337      722      402    40960        0             0 apt.systemd.dai
[88493.348813] [  13343]     0 13343    24166     3223   229376        0             0 apt-get
[88493.348815] [  13345]     0 13345     4419     1190    81920        0             0 sshd
[88493.348816] [  13347]   109 13347     3960      553    77824        0             0 sshd
[88493.348818] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-2.scope,task=nuclei,pid=12457,uid=0
[88493.348843] Out of memory: Killed process 12457 (nuclei) total-vm:1429548kB, anon-rss:712232kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1612kB oom_score_adj:0

@IfrahIman a memory profile + the exact commit your nuclei was built from would be super helpful. Will have a look at this during the next week but it looks like a goroutine leak preventing the GC from collecting its resources.

@jimen0 can you please share the command to get memory profile + the exact commit your nuclei ? I'll paste that in terminal and share the output.

@IfrahIman you can run nuclei with -profile-mem <profile.pprof> flag and share the profile.pprof file with us which will help us identify the leak.

Hey @IfrahIman - what @Ice3man543 said plus the output of $ go version -m <path/to/your/nuclei/binary>, it should look like this:

<path/to/your/nuclei/binary> go1.19
    path    github.com/projectdiscovery/nuclei/v2/cmd/nuclei
    mod github.com/projectdiscovery/nuclei/v2   (devel) 
    dep git.mills.io/prologic/smtpd v0.0.0-20210710122116-a525b76c287a  h1:3i+FJ7IpSZHL+VAjtpQeZCRhrpP0odl5XfoLBY4fxJ8=
    ...
    dep google.golang.org/protobuf  v1.28.1 h1:d0NfwRgPtno5B1Wa6L2DAG+KivqkdutMf1UhdNx175w=
    build   -compiler=gc
    build   CGO_ENABLED=0
    build   GOARCH=amd64
    build   GOOS=darwin
    build   GOAMD64=v1
    build   vcs=git
    build   vcs.revision=01e2e1df9a4064095b4bb6853bb093a3e7c930f7
    build   vcs.time=2022-09-08T10:08:58Z
    build   vcs.modified=true

The above output will allow me to build nuclei using the same Go version yours uses and from the same git commit vcs.revision yours was built. That, coupled with the memory profile you'll share will let me pinpoint the exact line of code causing the allocations.

Thanks!

Okay, there's definitely a goroutine leakage on fileutil package TimeoutReader type which is used when nuclei is configured to read from stdin: https://sourcegraph.com/github.com/projectdiscovery/nuclei@dev/-/blob/v2/pkg/core/inputs/hybrid/hmap.go?L75-78&subtree=true

Here is a small reproducer for the goroutine leakage: https://go.dev/play/p/IKCff54NBuz

The above leak alone counts for ~140/146 leaked goroutines on a nuclei process I ran. The other leaked goroutines seem to come from github.com/karlseguin/ccache@v2.0.3+incompatible/cache.go:167 itself or how Nuclei uses it (see https://github.com/karlseguin/ccache/issues/74#issuecomment-1200098662). My guess is that fixing both of these should resolve this issue.

cc @Ice3man543

Hello, I'll paste the pprof file when nuclei finishes the scanning, meanwhile Go version:


        path    github.com/projectdiscovery/nuclei/v2/cmd/nuclei
        mod     github.com/projectdiscovery/nuclei/v2   v2.7.7  h1:1gGoyvrsuhTDOKbImHzN7FVNLHDgSwLQwF97swHhL8k=
        dep     git.mills.io/prologic/smtpd     v0.0.0-20210710122116-a525b76c287a      h1:3i+FJ7IpSZHL+VAjtpQeZCRhrpP0odl5XfoLBY4fxJ8=
        dep     github.com/DataDog/gostackparse v0.5.0  h1:jb72P6GFHPHz2W0onsN51cS3FkaMDcjb0QzgxxA4gDk=
        dep     github.com/Knetic/govaluate     v3.0.1-0.20171022003610-9aa49832a739+incompatible       h1:1G1pk05UrOh0NlF1oeaaix1x8XzrfjIDK47TY0Zehcw=
        dep     github.com/Mzack9999/go-http-digest-auth-client v0.6.1-0.20220414142836-eb8883508809    h1:ZbFL+BDfBqegi+/Ssh7im5+aQfBRx6it+kHnC7jaDU8=
        dep     github.com/Mzack9999/ldapserver v1.0.2-0.20211229000134-b44a0d6ad0dd    h1:RTWs+wEY9efxTKK5aFic5C5KybqQelGcX+JdM69KoTo=
        dep     github.com/akrylysov/pogreb     v0.10.1 h1:FqlR8VR7uCbJdfUob916tPM+idpKgeESDXOA1K0DK4w=
        dep     github.com/alecthomas/jsonschema        v0.0.0-20211022214203-8b29eab41725      h1:NjwIgLQlD46o79bheVG4SCdRnnOz4XtgUN1WABX5DLA=
        dep     github.com/alecthomas/template  v0.0.0-20190718012654-fb15b899a751      h1:JYp7IbQjafoB+tBA3gMyHYHrpOtNuDiK/uB5uXxq5wM=
        dep     github.com/alecthomas/units     v0.0.0-20211218093645-b94a6e3cc137      h1:s6gZFSlWYmbqAuRjVTiNNhvNRfY2Wxp9nhfyel4rklc=
        dep     github.com/andres-erbsen/clock  v0.0.0-20160526145045-9e14626cd129      h1:MzBOUgng9orim59UnfUTLRjMpd09C5uEVQ6RPGeCaVI=
        dep     github.com/andygrunwald/go-jira v1.16.0 h1:PU7C7Fkk5L96JvPc6vDVIrd99vdPnYudHu4ju2c2ikQ=
        dep     github.com/antchfx/htmlquery    v1.2.5  h1:1lXnx46/1wtv1E/kzmH8vrfMuUKYgkdDBA9pIdMJnk4=
        dep     github.com/antchfx/xmlquery     v1.3.12 h1:6TMGpdjpO/P8VhjnaYPXuqT3qyJ/VsqoyNTmJzNBTQ4=
        dep     github.com/antchfx/xpath        v1.2.1  h1:qhp4EW6aCOVr5XIkT+l6LJ9ck/JsUH/yyauNgTQkBF8=
        dep     github.com/apex/log     v1.9.0  h1:FHtw/xuaM8AgmvDDTI9fiwoAL25Sq2cxojnZICUU8l0=
        dep     github.com/asaskevich/govalidator       v0.0.0-20210307081110-f21760c49a8d      h1:Byv0BzEl3/e6D5CLfI0j/7hiIEtvGVFPCZ7Ei2oq8iQ=
        dep     github.com/aws/aws-sdk-go       v1.44.83        h1:7+Rtc2Eio6EKUNoZeMV/IVxzVrY5oBQcNPtCcgIHYJA=
        dep     github.com/aymerick/douceur     v0.2.0  h1:Mv+mAeH1Q+n9Fr+oyamOlAkUNPWPlA8PPGR0QAaYuPk=
        dep     github.com/bits-and-blooms/bitset       v1.2.0  h1:Kn4yilvwNtMACtf1eYDlG8H77R07mZSPbMjLyS07ChA=
        dep     github.com/bits-and-blooms/bloom/v3     v3.0.1  h1:Inlf0YXbgehxVjMPmCGv86iMCKMGPPrPSHtBF5yRHwA=
        dep     github.com/blang/semver v3.5.1+incompatible     h1:cQNTCjp13qL8KC3Nbxr/y2Bqb63oX6wdnnjpJbkM4JQ=
        dep     github.com/bluele/gcache        v0.0.2  h1:WcbfdXICg7G/DGBh1PFfcirkWOQV+v077yF1pSy3DGw=
        dep     github.com/c4milo/unpackit      v0.1.0  h1:91pWJ6B3svZ4LOE+p3rnyucRK5fZwBdF/yQ/pcZO31I=
        dep     github.com/caddyserver/certmagic        v0.16.1 h1:rdSnjcUVJojmL4M0efJ+yHXErrrijS4YYg3FuwRdJkI=
        dep     github.com/cnf/structhash       v0.0.0-20201127153200-e1b16c1ebc08      h1:ox2F0PSMlrAAiAdknSRMDrAr8mfxPCfSZolH+/qQnyQ=
        dep     github.com/corpix/uarand        v0.2.0  h1:U98xXwud/AVuCpkpgfPF7J5TQgr7R5tqT8VZP5KWbzE=
        dep     github.com/dimchansky/utfbom    v1.1.1  h1:vV6w1AhK4VMnhBno/TPVCoK9U/LP0PkLCS9tbxHdi/U=
        dep     github.com/docker/go-units      v0.4.0  h1:3uh0PgVws3nIA0Q+MwDC8yjEPf9zjRfZZWXZYDct3Tw=
        dep     github.com/dsnet/compress       v0.0.1  h1:PlZu0n3Tuv04TzpfPbrnI0HW/YwodEXDS+oPKahKF0Q=
        dep     github.com/fatih/structs        v1.1.0  h1:Q7juDM0QtcnhCpeyLGQKyg4TOIghuNXrkL32pHAUMxo=
        dep     github.com/go-playground/locales        v0.14.0 h1:u50s323jtVGugKlcYeyzC0etD1HifMjqmJqb8WugfUU=
        dep     github.com/go-playground/universal-translator   v0.18.0 h1:82dyy6p4OuJq4/CByFNOn/jYrnRPArHwAcmLoJZxyho=
        dep     github.com/go-playground/validator/v10  v10.11.0        h1:0W+xRM511GY47Yy3bZUbJVitCNg2BOGlCyvTqsp/xIw=
        dep     github.com/go-rod/rod   v0.109.0        h1:h16UAw8AWRTcZ2bPfk23o8L1DBW692RUJzet49XD1rk=
        dep     github.com/goburrow/cache       v0.1.4  h1:As4KzO3hgmzPlnaMniZU9+VmoNYseUhuELbxy9mRBfw=
        dep     github.com/gobwas/httphead      v0.1.0  h1:exrUm0f4YX0L7EBwZHuCF4GDp8aJfVeBrlLQrs6NqWU=
        dep     github.com/gobwas/pool  v0.2.1  h1:xfeeEhW7pwmX8nuLVlqbzVc7udMDrwetjEv+TZIz1og=
        dep     github.com/gobwas/ws    v1.1.0  h1:7RFti/xnNkMJnrK7D1yQ/iCIB5OrrY/54/H930kIbHA=
        dep     github.com/golang-jwt/jwt/v4    v4.4.2  h1:rcc4lwaZgFMCZ5jxF9ABolDcIHdBytAFgqFPbSJQAYs=
        dep     github.com/golang/groupcache    v0.0.0-20200121045136-8c9f03a8e57e      h1:1r7pUrabqp18hOBcwBwiTsbnFeTZHV9eER/QT5JVZxY=
        dep     github.com/golang/snappy        v0.0.4  h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM=
        dep     github.com/google/go-github     v17.0.0+incompatible    h1:N0LgJ1j65A7kfXrZnUDaYCs/Sf4rEjNlfyDHW9dolSY=
        dep     github.com/google/go-querystring        v1.1.0  h1:AnCroh3fv4ZBgVIf1Iwtovgjaw/GiKJo8M8yD/fhyJ8=
        dep     github.com/google/uuid  v1.3.0  h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
        dep     github.com/gorilla/css  v1.0.0  h1:BQqNyPTi50JCFMTw/b67hByjMVXZRwGha6wxVGkeihY=
        dep     github.com/gosuri/uilive        v0.0.4  h1:hUEBpQDj8D8jXgtCdBu7sWsy5sbW/5GhuO8KBwJ2jyY=
        dep     github.com/gosuri/uiprogress    v0.0.1  h1:0kpv/XY/qTmFWl/SkaJykZXrBBzwwadmW8fRb7RJSxw=
        dep     github.com/h2non/filetype       v1.1.3  h1:FKkx9QbD7HR/zjK1Ia5XiBsq9zdLi5Kf3zGyFTAFkGg=
        dep     github.com/hashicorp/errwrap    v1.0.0  h1:hLrqtEDnRye3+sgx6z4qVLNuviH3MR5aQ0ykNJa/UYA=
        dep     github.com/hashicorp/go-cleanhttp       v0.5.2  h1:035FKYIWjmULyFRBKPs8TBQoi0x6d9G4xc9neXJWAZQ=
        dep     github.com/hashicorp/go-multierror      v1.1.1  h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo=
        dep     github.com/hashicorp/go-retryablehttp   v0.7.1  h1:sUiuQAnLlbvmExtFQs72iFW/HXeUn8Z1aJLQ4LJJbTQ=
        dep     github.com/hashicorp/go-version v1.6.0  h1:feTTfFNnjP967rlCxM/I9g701jU+RN74YKx2mOkIeek=
        dep     github.com/iancoleman/orderedmap        v0.0.0-20190318233801-ac98e3ecb4b0      h1:i462o439ZjprVSFSZLZxcsoAe592sZB1rci2Z8j4wdk=
        dep     github.com/itchyny/gojq v0.12.8 h1:Zxcwq8w4IeR8JJYEtoG2MWJZUv0RGY6QqJcO1cqV8+A=
        dep     github.com/itchyny/timefmt-go   v0.1.3  h1:7M3LGVDsqcd0VZH2U+x393obrzZisp7C0uEe921iRkU=
        dep     github.com/jmespath/go-jmespath v0.4.0  h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
        dep     github.com/json-iterator/go     v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
        dep     github.com/karlseguin/ccache    v2.0.3+incompatible     h1:j68C9tWOROiOLWTS/kCGg9IcJG+ACqn5+0+t8Oh83UU=
        dep     github.com/klauspost/compress   v1.15.8 h1:JahtItbkWjf2jzm/T+qgMxkP9EMHsqEUA6vCMGmXvhA=
        dep     github.com/klauspost/cpuid/v2   v2.0.11 h1:i2lw1Pm7Yi/4O6XCSyJWqEHI2MDw2FzUK6o/D21xn2A=
        dep     github.com/klauspost/pgzip      v1.2.5  h1:qnWYvvKqedOF2ulHpMG72XQol4ILEJ8k2wwRl/Km8oE=
        dep     github.com/leodido/go-urn       v1.2.1  h1:BqpAaACuzVSgi/VLzGZIobT2z4v53pjosyNd9Yv6n/w=
        dep     github.com/libdns/libdns        v0.2.1  h1:Wu59T7wSHRgtA0cfxC+n1c/e+O3upJGWytknkmFEDis=
        dep     github.com/logrusorgru/aurora   v2.0.3+incompatible     h1:tOpm7WcpBTn4fjmVfgpQq0EfczGlG91VSDkswnjF5A8=
        dep     github.com/lor00x/goldap        v0.0.0-20180618054307-a546dffdd1a3      h1:wIONC+HMNRqmWBjuMxhatuSzHaljStc4gjDeKycxy0A=
        dep     github.com/mattn/go-runewidth   v0.0.13 h1:lTGmDsbAYt5DmK6OnoV7EuIF1wEIFAcxld6ypU4OSgU=
        dep     github.com/mholt/acmez  v1.0.2  h1:C8wsEBIUVi6e0DYoxqCcFuXtwc4AWXL/jgcDjF7mjVo=
        dep     github.com/mholt/archiver       v3.1.1+incompatible     h1:1dCVxuqs0dJseYEhi5pl7MYPH9zDa1wBi7mF09cbNkU=
        dep     github.com/microcosm-cc/bluemonday      v1.0.19 h1:OI7hoF5FY4pFz2VA//RN8TfM0YJ2dJcl4P4APrCWy6c=
        dep     github.com/miekg/dns    v1.1.50 h1:DQUfb9uc6smULcREF09Uc+/Gd46YWqJd5DbpPE9xkcA=
        dep     github.com/mitchellh/go-homedir v1.1.0  h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
        dep     github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd      h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
        dep     github.com/modern-go/reflect2   v1.0.2  h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
        dep     github.com/nwaples/rardecode    v1.1.2  h1:Cj0yZY6T1Zx1R7AhTbyGSALm44/Mmq+BAPc4B/p/d3M=
        dep     github.com/olekukonko/tablewriter       v0.0.5  h1:P2Ga83D34wi1o9J6Wh1mRuqd4mF/x/lgBS7N7AbDhec=
        dep     github.com/openrdap/rdap        v0.9.1-0.20191017185644-af93e7ef17b7    h1:3Xn/CN6GVY+7mVuGgt5bfp0F9JwcWqnvwfb23Jf8Vxg=
        dep     github.com/owenrumney/go-sarif/v2       v2.1.2  h1:PMDK7tXShJ9zsB7bfvlpADH5NEw1dfA9xwU8Xtdj73U=
        dep     github.com/pierrec/lz4  v2.6.1+incompatible     h1:9UY3+iC23yxF0UfGaYrGplQ+79Rg+h/q9FV9ix19jjM=
        dep     github.com/pkg/errors   v0.9.1  h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
        dep     github.com/projectdiscovery/blackrock   v0.0.0-20210415162320-b38689ae3a2e      h1:7bwaFH1jvtOo5ndhTQgoA349ozhX+1dc4b6tbaPnBOA=
        dep     github.com/projectdiscovery/clistats    v0.0.8  h1:tjmWb15mqsPf/yrQXVHLe2ThZX/5+mgKSfZBKWWLh20=
        dep     github.com/projectdiscovery/cryptoutil  v1.0.0  h1:5rQfnWDthJ5ZFcqze+rmT1N7l1HJQ6EB26MrjaYB7I0=
        dep     github.com/projectdiscovery/fastdialer  v0.0.16-0.20220609092737-7e3c7dcecf16   h1:oinl80GbA16c6Et6YxcYw/QfwsZgRE0ZswxL43/MNhY=
        dep     github.com/projectdiscovery/filekv      v0.0.0-20210915124239-3467ef45dd08      h1:NwD1R/du1dqrRKN3SJl9kT6tN3K9puuWFXEvYF2ihew=
        dep     github.com/projectdiscovery/fileutil    v0.0.0-20220705195237-01becc2a8963      h1:4o97N9ftX1J3iKlIRVMPVOVZs4qbCczJvoFF2WA40t4=
        dep     github.com/projectdiscovery/folderutil  v0.0.0-20220215113126-add60a1e8e08      h1:m1pgJisawU7zP9lKGktOEk6KNrNAR7e4Q07Kt3ox0NM=
        dep     github.com/projectdiscovery/goflags     v0.0.8-0.20220610073650-5d31a8c159e3    h1:2cUSicd6wkd7A3P+tjjAYiPmD99FtWp4k2curJkcrJ0=
        dep     github.com/projectdiscovery/gologger    v1.1.4  h1:qWxGUq7ukHWT849uGPkagPKF3yBPYAsTtMKunQ8O2VI=
        dep     github.com/projectdiscovery/hmap        v0.0.2-0.20210917080408-0fd7bd286bfa    h1:9sZWFUAshIa/ea0RKjGRuuZiS5PzYXAFjTRUnSbezr0=
        dep     github.com/projectdiscovery/interactsh  v1.0.6  h1:nuJmJ9zSUiG5tcgwgtkFdXpYghuB+i+OuvohXSCPhxo=
        dep     github.com/projectdiscovery/iputil      v0.0.0-20220620153941-036d511e4097      h1:hPFYljvvgbvH+oTPtixCTruvR8uecCBON4C74xOUMFY=
        dep     github.com/projectdiscovery/mapcidr     v1.0.0  h1:4ctYAgkxaLeky3o3c+qMgHoUjytYy6DME2fjbgiB1Ag=
        dep     github.com/projectdiscovery/networkpolicy       v0.0.1  h1:RGRuPlxE8WLFF9tdKSjTsYiTIKHNHW20Kl0nGGiRb1I=
        dep     github.com/projectdiscovery/nuclei-updatecheck-api      v0.0.0-20211006155443-c0a8d610a4df      h1:CvTNAUD5JbLMqpMFoGNgfk2gOcN0NC57ICu0+oK84vs=
        dep     github.com/projectdiscovery/rawhttp     v0.1.1  h1:VRpPAGwgLzKyApUbVqW2NjCMPxrjAmRkeIS+WPKwhFI=
        dep     github.com/projectdiscovery/retryabledns        v1.0.13 h1:Ogfv0fl3Nszb+Nq2S2qQmE+PJDlStVxyLkmTotMohZY=
        dep     github.com/projectdiscovery/retryablehttp-go    v1.0.3-0.20220506110515-811d938bd26d    h1:VR+tDkedzHIp1pGKIDcfPFt7J8KjcjxGsJvBAP6RXFQ=
        dep     github.com/projectdiscovery/sliceutil   v0.0.0-20220511171050-c7d9bc5cadd9      h1:dQO4FNxOhkZqZHW5nUh6f1NXIRUyoZ57bXETXjsyfNk=
        dep     github.com/projectdiscovery/stringsutil v0.0.0-20220612082425-0037ce9f89f3      h1:EbP+rVR7NPsFKfwhzshmXP2GMjqBpaMZurJzQ5PYvbY=
        dep     github.com/projectdiscovery/urlutil     v0.0.0-20210525140139-b874f06ad921      h1:EgaxpJm7+lKppfAHkFHs+S+II0lodp4Gu3leZCCkWlc=
        dep     github.com/projectdiscovery/wappalyzergo        v0.0.56 h1:774LsI8tAUAYROQhTX9VsxoQN2kAC5m9CcNj3BHsvTs=
        dep     github.com/projectdiscovery/yamldoc-go  v1.0.3-0.20211126104922-00d2c6bb43b6    h1:DvWRQpw7Ib2CRL3ogYm/BWM+X0UGPfz1n9Ix9YKgFM8=
        dep     github.com/remeh/sizedwaitgroup v1.0.0  h1:VNGGFwNo/R5+MJBf6yrsr110p0m4/OX4S3DCy7Kyl5E=
        dep     github.com/rivo/uniseg  v0.2.0  h1:S1pD9weZBuJdFmowNwbpi7BJ8TNftyUImj/0WQi72jY=
        dep     github.com/rs/xid       v1.4.0  h1:qd7wPTDkN6KQx2VmMBLrpHkiyQwgFXRnkOLacUiaSNY=
        dep     github.com/saintfish/chardet    v0.0.0-20120816061221-3af4cd4741ca      h1:NugYot0LIVPxTvN8n+Kvkn6TrbMyxQiuvKdEwFdR9vI=
        dep     github.com/segmentio/ksuid      v1.0.4  h1:sBo2BdShXjmcugAMwjugoGUdUV0pcxY5mW4xKRn3v4c=
        dep     github.com/shirou/gopsutil/v3   v3.22.7 h1:flKnuCMfUUrO+oAvwAd6GKZgnPzr098VA/UJ14nhJd4=
        dep     github.com/spaolacci/murmur3    v1.1.0  h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI=
        dep     github.com/spf13/cast   v1.5.0  h1:rj3WzYc11XZaIZMPKmwP96zkFEnnAmV8s6XbB2aY32w=
        dep     github.com/syndtr/goleveldb     v1.0.0  h1:fBdIW9lB4Iz0n9khmH8w27SJ3QEJ7+IgjPEwGSZiFdE=
        dep     github.com/tj/go-update v2.2.5-0.20200519121640-62b4b798fd68+incompatible       h1:guTq1YxwB8XSILkI9q4IrOmrCOS6Hc1L3hmOhi4Swcs=
        dep     github.com/tklauser/go-sysconf  v0.3.10 h1:IJ1AZGZRWbY8T5Vfk04D9WOA5WSejdflXxP03OUqALw=
        dep     github.com/tklauser/numcpus     v0.4.0  h1:E53Dm1HjH1/R2/aoCtXtPgzmElmn51aOkhCFSuZq//o=
        dep     github.com/trivago/tgo  v1.0.7  h1:uaWH/XIy9aWYWpjm2CU3RpcqZXmX2ysQ9/Go+d9gyrM=
        dep     github.com/ulikunitz/xz v0.5.10 h1:t92gobL9l3HE202wg3rlk19F6X+JOxl9BBrCCMYEYd8=
        dep     github.com/ulule/deepcopier     v0.0.0-20200430083143-45decc6639b6      h1:TtyC78WMafNW8QFfv3TeP3yWNDG+uxNkk9vOrnDu6JA=
        dep     github.com/valyala/bytebufferpool       v1.0.0  h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
        dep     github.com/valyala/fasttemplate v1.2.1  h1:TVEnxayobAdVkhQfrfes2IzOB6o+z4roRkPF52WA1u4=
        dep     github.com/weppos/publicsuffix-go       v0.15.1-0.20210928183822-5ee35905bd95   h1:DyAZOw3JsVd6LJHqhl4MpKQdYQEmat0C6pPPwom39Ow=
        dep     github.com/xanzy/go-gitlab      v0.73.1 h1:UMagqUZLJdjss1SovIC+kJCH4k2AZWXl58gJd38Y/hI=
        dep     github.com/xi2/xz       v0.0.0-20171230120015-48954b6210f8      h1:nIPpBwaJSVYIxUFsDv3M8ofmx9yWTog9BfvIu0q41lo=
        dep     github.com/yl2chen/cidranger    v1.0.2  h1:lbOWZVCG1tCRX4u24kuM1Tb4nHqWkDxwLdoS+SevawU=
        dep     github.com/ysmood/goob  v0.4.0  h1:HsxXhyLBeGzWXnqVKtmT9qM7EuVs/XOgkX7T6r1o1AQ=
        dep     github.com/ysmood/gson  v0.7.1  h1:zKL2MTGtynxdBdlZjyGsvEOZ7dkxaY5TH6QhAbTgz0Q=
        dep     github.com/ysmood/leakless      v0.8.0  h1:BzLrVoiwxikpgEQR0Lk8NyBN5Cit2b1z+u0mgL4ZJak=
        dep     github.com/zmap/rc2     v0.0.0-20131011165748-24b9757f5521      h1:kKCF7VX/wTmdg2ZjEaqlq99Bjsoiz7vH6sFniF/vI4M=
        dep     github.com/zmap/zcrypto v0.0.0-20211005224000-2d0ffdec8a9b      h1:iYQzlljG1dOXBtsJGyzFC/wBK5qUCWs1eLCr/UcJYPA=
        dep     go.etcd.io/bbolt        v1.3.6  h1:/ecaJf0sk1l4l6V4awd65v2C3ILy7MSj+s/x1ADCIMU=
        dep     go.uber.org/atomic      v1.9.0  h1:ECmE8Bn/WFTYwEW/bpKD3M8VtR/zQVbavAoalC1PYyE=
        dep     go.uber.org/multierr    v1.8.0  h1:dg6GjLku4EH+249NNmoIciG9N/jURbDG+pFlTkhzIC8=
        dep     go.uber.org/ratelimit   v0.2.0  h1:UQE2Bgi7p2B85uP5dC2bbRtig0C+OeNRnNEafLjsLPA=
        dep     go.uber.org/zap v1.21.0 h1:WefMeulhovoZ2sYXz7st6K0sLj7bBhpiFaud4r4zST8=
        dep     goftp.io/server/v2      v2.0.0  h1:FF8JKXXKDxAeO1uXEZz7G+IZwCDhl19dpVIlDtp3QAg=
        dep     golang.org/x/crypto     v0.0.0-20220210151621-f4118a5b28e2      h1:XdAboW3BNMv9ocSCOk/u1MFioZGzCNkiJZ19v9Oe3Ig=
        dep     golang.org/x/net        v0.0.0-20220805013720-a33c5aa5df48      h1:N9Vc/rorQUDes6B9CNdIxAn5jODGj2wzfrei2x4wNj4=
        dep     golang.org/x/oauth2     v0.0.0-20220722155238-128564f6959c      h1:q3gFqPqH7NVofKo3c3yETAP//pPI+G5mvB7qqj1Y5kY=
        dep     golang.org/x/sys        v0.0.0-20220728004956-3c1f35247d10      h1:WIoqL4EROvwiPdUtaip4VcDdpZ4kha7wBWZrbVKCIZg=
        dep     golang.org/x/text       v0.3.7  h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
        dep     golang.org/x/time       v0.0.0-20220722155302-e5dcc9cfc0b9      h1:ftMN5LMiBFjbzleLqtoBZk7KdJwhuybIU+FckUHgoyQ=
        dep     gopkg.in/alecthomas/kingpin.v2  v2.2.6  h1:jMFz6MfLP0/4fUyZle81rXUoxOBFi19VUFKVDOQfozc=
        dep     gopkg.in/corvus-ch/zbase32.v1   v1.0.0  h1:K4u1NprbDNvKPczKfHLbwdOWHTZ0zfv2ow71H1nRnFU=
        dep     gopkg.in/yaml.v2        v2.4.0  h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
        dep     gopkg.in/yaml.v3        v3.0.1  h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
        dep     moul.io/http2curl       v1.0.0  h1:6XwpyZOYsgZJrU8exnG87ncVkU1FVCcTRpwzOkTDUi8=
        build   -compiler=gc
        build   CGO_ENABLED=1
        build   CGO_CFLAGS=
        build   CGO_CPPFLAGS=
        build   CGO_CXXFLAGS=
        build   CGO_LDFLAGS=
        build   GOARCH=amd64
        build   GOOS=linux
        build   GOAMD64=v1```

Hello Team

the command: cat urls.txt | nuclei -s low,medium,high,critical -o output.txt -profile-mem test.pprof does not seem to be storing the debug info the test.pprof file. I ran it 2 times. The process gets killed as usual but the test.pprof file is empty

@IfrahIman can you run with a smaller target list? The profile is empty because it is written at the end of execution and the execution doesn't complete since we OOM and crash.

You can run nuclei with the -ep flag to enable the pprof handlers and then run $ go tool pprof -png http://localhost:8086/debug/pprof/heap > out_n.png (8086 port is hardcoded here https://github.com/projectdiscovery/nuclei/blob/1010cca84e62e04cd675debfce20ce96d2e9cd3c/v2/internal/runner/runner.go#L74 ) a couple of times (once each 3 or 5 minutes for example). These images should show top heap allocations.

Anyways, I think fixing the goroutine leakage I described above, the missing Close() calls on ccache and this other leak should resolve this issue. These account roughly for all goroutines on my ~10 minutes running Nuclei process.

goroutine profile: total 903
 561 @ 0x1039836 0x1068235 0x1f41f1c 0x1f43c7e 0x1f6a63e 0x1f69bad 0x1f73b9f 0x2093abe 0x106b3a1
 #   0x1068234   time.Sleep+0x134                                            /usr/local/go/src/runtime/time.go:195
 #   0x1f41f1b   github.com/andres-erbsen/clock.(*clock).Sleep+0x1b                          /Users/miguel/pkg/mod/github.com/andres-erbsen/clock@v0.0.0-20160526145045-9e14626cd129/clock.   go:40
 #   0x1f43c7d   go.uber.org/ratelimit.(*atomicLimiter).Take+0x1bd                           /Users/miguel/pkg/mod/go.uber.org/ratelimit@v0.2.0/limiter_atomic.go:108
 #   0x1f6a63d   github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*Request).ExecuteWithResults.func1+0x6dd  /Users/miguel/research/projectdiscovery/nuclei/v2/pkg/protocols/http/    request.go:279
 #   0x1f69bac   github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*Request).ExecuteWithResults+0x36c        /Users/miguel/research/projectdiscovery/nuclei/v2/pkg/protocols/http/    request.go:333
 #   0x1f73b9e   github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/executer.(*Executer).Execute+0x1be       /Users/miguel/research/projectdiscovery/nuclei/v2/pkg/protocols/common/  executer/executer.go:67
 #   0x2093abd   github.com/projectdiscovery/nuclei/v2/pkg/core.(*Engine).executeModelWithInput.func2.1+0x11d        /Users/miguel/research/projectdiscovery/nuclei/v2/pkg/core/execute.go:   142

 152 @ 0x1039836 0x1005a2c 0x10055dd 0x13d97ba 0x106b3a1
 #   0x13d97b9   github.com/projectdiscovery/fileutil.TimeoutReader.Read.func1+0x39  /Users/miguel/pkg/mod/github.com/projectdiscovery/fileutil@v0.0.0-20220705195237-01becc2a8963/           timeout_reader.go:31

Heap roughly about the same time looked like this: out_99

cc @Ice3man543

Hi, Here is the pprof output:

$pprof -top /root/pprof/pprof.nuclei.alloc_objects.alloc_space.inuse_objects.inuse_space.006.pb.gz                                                       
File: nuclei                                                                                                                                                                                 
Type: inuse_space                                                                                                                                                                            
Time: Sep 13, 2022 at 12:00am (UTC)                                                                                                                                                          
Showing nodes accounting for 85354.43kB, 88.77% of 96151.70kB total                                                                                                                          
Dropped 662 nodes (cum <= 480.76kB)                                                                                                                                                          
      flat  flat%   sum%        cum   cum%                                                                                                                                                   
   12288kB 12.78% 12.78% 12302.37kB 12.79%  github.com/syndtr/goleveldb/leveldb/memdb.New                                                                                                    
12014.99kB 12.50% 25.28% 12014.99kB 12.50%  bytes.makeSlice                                                                                                                                  
 8738.55kB  9.09% 34.36%  8738.55kB  9.09%  reflect.New                                                                                                                                      
 5274.31kB  5.49% 39.85% 11776.44kB 12.25%  crypto/x509.parseCertificate                                                                                                                     
 4480.04kB  4.66% 44.51%  4480.04kB  4.66%  gopkg.in/yaml%2ev2.(*parser).scalar                                                                                                              
 4232.07kB  4.40% 48.91% 13577.61kB 14.12%  crypto/tls.(*Conn).readHandshake                                                                                                                 
 3906.01kB  4.06% 52.97%  3906.01kB  4.06%  bufio.NewReaderSize (inline)                                                                                                                     
 3872.92kB  4.03% 57.00%  3872.92kB  4.03%  github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators.MergeMaps (inline)                                                         
 3457.38kB  3.60% 60.60%  3457.38kB  3.60%  bufio.NewWriterSize (inline)                                                                                                                     
 2240.06kB  2.33% 62.93%  2240.06kB  2.33%  strings.(*Builder).grow (inline)                                                                                                                 
 1932.96kB  2.01% 64.94% 13695.69kB 14.24%  github.com/projectdiscovery/nuclei/v2/pkg/parsers.ParseTemplate

2240.06kB  2.33% 62.93%  2240.06kB  2.33%  strings.(*Builder).grow (inline)                                                                                                       [132/1917] 1932.96kB  2.01% 64.94% 13695.69kB 14.24%  github.com/projectdiscovery/nuclei/v2/pkg/parsers.ParseTemplate                                                                                  
 1857.06kB  1.93% 66.87%  1857.06kB  1.93%  crypto/x509/pkix.(*Name).FillFromRDNSequence                                                                                                     
 1833.54kB  1.91% 68.77%  1833.54kB  1.91%  regexp/syntax.(*compiler).inst (inline)                                                                                                          
 1757.80kB  1.83% 70.60%  1757.80kB  1.83%  vendor/golang.org/x/crypto/cryptobyte.(*String).ReadASN1ObjectIdentifier                                                                         
 1623.70kB  1.69% 72.29%  1990.63kB  2.07%  io.ReadAll                                                                                                                                       
 1512.09kB  1.57% 73.86%  2129.34kB  2.21%  github.com/projectdiscovery/nuclei/v2/pkg/model/types/stringslice.(*StringSlice).UnmarshalYAML                                                   
 1337.04kB  1.39% 75.25%  1337.04kB  1.39%  bytes.Replace                                                                                                                                    
 1204.01kB  1.25% 76.51%  1204.01kB  1.25%  math/big.nat.make (inline)                                                                                                                       
 1140.28kB  1.19% 77.69% 12479.52kB 12.98%  github.com/projectdiscovery/nuclei/v2/pkg/templates.Parse                                                                                        
  904.27kB  0.94% 78.63%   904.27kB  0.94%  github.com/syndtr/goleveldb/leveldb/memdb.(*DB).Put                                                                                              
  872.57kB  0.91% 79.54% 38835.59kB 40.39%  net/http.(*Transport).dialConn                                                                                                                   
  871.61kB  0.91% 80.45%   979.99kB  1.02%  github.com/projectdiscovery/nuclei/v2/pkg/templates.(*Template).compileProtocolRequests                                                          
  808.29kB  0.84% 81.29%   808.29kB  0.84%  reflect.unsafe_NewArray                                                                                                                          
  658.62kB  0.68% 81.97%   658.62kB  0.68%  reflect.mapassign_faststr                                                                                                                        
  618.58kB  0.64% 82.62%   618.58kB  0.64%  crypto/tls.Client (inline)                                                                                                                       
  516.20kB  0.54% 83.15%   673.99kB   0.7%  github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/replacer.Replace                                                                      
  515.63kB  0.54% 83.69%   515.63kB  0.54%  crypto/aes.(*aesCipherGCM).NewGCM                                                                                                                
  506.06kB  0.53% 84.22%  1722.64kB  1.79%  crypto/x509.parseName                                                                                                                            
  493.44kB  0.51% 84.73% 12283.19kB 12.77%  github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*Request).executeRequest                                                               
  473.08kB  0.49% 85.22%   587.69kB  0.61%  github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*Request).responseToDSLMap                                                             
  453.46kB  0.47% 85.69%   521.53kB  0.54%  sync.(*Map).Store                                                                                                                                
  438.60kB  0.46% 86.15%   983.24kB  1.02%  crypto/x509.processExtensions                                                                                                                    
  288.95kB   0.3% 86.45%   797.22kB  0.83%  github.com/projectdiscovery/nuclei/v2/pkg/templates/cache.(*Templates).Store (inline)                                                            
  288.09kB   0.3% 86.75%  1939.44kB  2.02%  io.copyBuffer                                                                                                                                    
  251.44kB  0.26% 87.01%   563.59kB  0.59%  crypto/hmac.New                                                                                                                                  
  232.06kB  0.24% 87.25%   488.98kB  0.51%  net/http.(*Transport).getConn                                                                                                                    
  216.97kB  0.23% 87.48%  3799.36kB  3.95%  github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*Request).Compile                                                                      
  195.64kB   0.2% 87.68%  2370.61kB  2.47%  regexp.compile                                                                                                                                   
  193.66kB   0.2% 87.88%   493.51kB  0.51%  encoding/gob.(*Decoder).compileDec                                                                                                               
  136.45kB  0.14% 88.02%  1061.76kB  1.10%  crypto/x509.parsePublicKey                                                                                                                       
  111.59kB  0.12% 88.14%   510.03kB  0.53%  encoding/gob.(*Decoder).getDecEnginePtr                                                                                                          
  109.48kB  0.11% 88.25%   530.28kB  0.55%  vendor/golang.org/x/crypto/hkdf.Expand                                                                                                           
   99.48kB   0.1% 88.36%   824.59kB  0.86%  github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*requestGenerator).handleRawWithPayloads                                               
   92.53kB 0.096% 88.45%  1005.52kB  1.05%  crypto/tls.(*cipherSuiteTLS13).expandLabel                     

99.48kB   0.1% 88.36%   824.59kB  0.86%  github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*requestGenerator).handleRawWithPayloads                                      [99/1917]   92.53kB 0.096% 88.45%  1005.52kB  1.05%  crypto/tls.(*cipherSuiteTLS13).expandLabel                                                                                                       
   88.52kB 0.092% 88.55% 14969.33kB 15.57%  github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/executer.(*Executer).Execute                                                          
   50.23kB 0.052% 88.60%  9580.48kB  9.96%  gopkg.in/yaml%2ev2.(*decoder).sequence                                                                                                           
   36.14kB 0.038% 88.64%   917.52kB  0.95%  crypto/tls.aeadAESGCMTLS13                                                                                                                       
   24.26kB 0.025% 88.66% 28194.24kB 29.32%  crypto/tls.(*Conn).handshakeContext                                                                                                              
   24.23kB 0.025% 88.69%   839.22kB  0.87%  github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*requestGenerator).makeHTTPRequestFromModel                                            
   16.76kB 0.017% 88.70% 30900.61kB 32.14%  github.com/projectdiscovery/fastdialer/fastdialer.(*Dialer).DialTLS                                                                              
    8.35kB 0.0087% 88.71%   852.11kB  0.89%  github.com/projectdiscovery/fastdialer/fastdialer.(*Dialer).GetDNSDataFromCache                                                                 
    8.09kB 0.0084% 88.72% 12390.46kB 12.89%  github.com/syndtr/goleveldb/leveldb.openDB                                                                                                      
    8.08kB 0.0084% 88.73%  3211.17kB  3.34%  github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.dumpResponseWithRedirectChain                                                          
    8.06kB 0.0084% 88.74%   716.70kB  0.75%  net/http.(*Transport).roundTrip                                                                                                                 
    8.05kB 0.0084% 88.75%  2065.81kB  2.15%  net/http/httputil.DumpResponse                                                                                                                  
    8.03kB 0.0084% 88.75% 12003.01kB 12.48%  crypto/tls.(*Conn).verifyServerCertificate                                                                                                      
    4.03kB 0.0042% 88.76% 12019.02kB 12.50%  bytes.(*Buffer).grow                                                                                                                            
    4.01kB 0.0042% 88.76%  5342.11kB  5.56%  crypto/tls.(*Conn).readFromUntil                                                                                                                
    4.01kB 0.0042% 88.77%   812.31kB  0.84%  reflect.MakeSlice                                                                                                                               
    4.01kB 0.0042% 88.77%  2057.76kB  2.14%  net/http.(*Response).Write                                                                                                                      
         0     0% 88.77%  5338.10kB  5.55%  bytes.(*Buffer).Grow (inline)                                                                                                                    
         0     0% 88.77%  1739.57kB  1.81%  bytes.(*Buffer).ReadFrom                                                                                                                         
         0     0% 88.77%  4793.89kB  4.99%  bytes.(*Buffer).Write                                                                                                                            
         0     0% 88.77%  1337.04kB  1.39%  bytes.ReplaceAll (inline)                                                                                                                        
         0     0% 88.77%   515.63kB  0.54%  crypto/cipher.NewGCM (inline)                                                                                                                    
         0     0% 88.77%   515.63kB  0.54%  crypto/cipher.newGCMWithNonceAndTagSize                                                                                                          
         0     0% 88.77% 28194.24kB 29.32%  crypto/tls.(*Conn).HandshakeContext (inline)                                                                                                     
         0     0% 88.77% 28161.88kB 29.29%  crypto/tls.(*Conn).clientHandshake                                                                                                               
         0     0% 88.77%  9679.42kB 10.07%  crypto/tls.(*Conn).readRecord (inline)                                                                                                           
         0     0% 88.77%  9679.42kB 10.07%  crypto/tls.(*Conn).readRecordOrCCS                                                                                                               
         0     0% 88.77%   569.25kB  0.59%  crypto/tls.(*cipherSuiteTLS13).deriveSecret                                                                                                      
         0     0% 88.77%  1608.02kB  1.67%  crypto/tls.(*clientHandshakeState).doFullHandshake                                                                                               
         0     0% 88.77%  1693.85kB  1.76%  crypto/tls.(*clientHandshakeState).handshake                                                                                                     
         0     0% 88.77%   676.60kB   0.7%  crypto/tls.(*clientHandshakeStateTLS13).establishHandshakeKeys                                                                                   
         0     0% 88.77% 26277.23kB 27.33%  crypto/tls.(*clientHandshakeStateTLS13).handshake                                                                                                
         0     0% 88.77%    24029kB 24.99%  crypto/tls.(*clientHandshakeStateTLS13).readServerCertificate                                                                                    
         0     0% 88.77%   918.49kB  0.96%  crypto/tls.(*clientHandshakeStateTLS13).readServerFinished 

    0     0% 88.77%    24029kB 24.99%  crypto/tls.(*clientHandshakeStateTLS13).readServerCertificate                                                                           [66/1917]         0     0% 88.77%   918.49kB  0.96%  crypto/tls.(*clientHandshakeStateTLS13).readServerFinished                                                                                       
         0     0% 88.77%   575.92kB   0.6%  crypto/tls.(*clientHandshakeStateTLS13).sendClientFinished                                                                                       
         0     0% 88.77%  1284.18kB  1.34%  crypto/tls.(*halfConn).setTrafficSecret                                                                                                          
         0     0% 88.77% 28830.07kB 29.98%  crypto/tls.DialWithDialer (inline)                                                                                                               
         0     0% 88.77% 28830.07kB 29.98%  crypto/tls.dial                                                                                                                                  
         0     0% 88.77% 11776.44kB 12.25%  crypto/x509.ParseCertificate                                                                                                                     
         0     0% 88.77%   724.32kB  0.75%  crypto/x509.parseExtension                                                                                                                       
         0     0% 88.77%   733.35kB  0.76%  encoding/gob.(*Decoder).Decode                                                                                                                   
         0     0% 88.77%   733.35kB  0.76%  encoding/gob.(*Decoder).DecodeValue                                                                                                              
         0     0% 88.77%   699.89kB  0.73%  encoding/gob.(*Decoder).decodeValue                                                                                                              
         0     0% 88.77% 30883.84kB 32.12%  github.com/projectdiscovery/fastdialer/fastdialer.(*Dialer).DialTLSWithConfig (inline)                                                           
         0     0% 88.77%   872.25kB  0.91%  github.com/projectdiscovery/fastdialer/fastdialer.(*Dialer).GetDNSData                                                                           
         0     0% 88.77% 30946.42kB 32.19%  github.com/projectdiscovery/fastdialer/fastdialer.(*Dialer).dial                                                                                 
         0     0% 88.77%  8304.43kB  8.64%  github.com/projectdiscovery/fastdialer/fastdialer.NewDialer                                                                                      
         0     0% 88.77%   920.32kB  0.96%  github.com/projectdiscovery/hmap/store/disk.(*LevelDB).Set                                                                                       
         0     0% 88.77%   920.32kB  0.96%  github.com/projectdiscovery/hmap/store/disk.(*LevelDB).set                                                                                       
         0     0% 88.77% 12474.51kB 12.97%  github.com/projectdiscovery/hmap/store/disk.OpenLevelDB                                                                                          
         0     0% 88.77%   920.32kB  0.96%  github.com/projectdiscovery/hmap/store/hybrid.(*HybridMap).Set                                                                                   
         0     0% 88.77% 12474.51kB 12.97%  github.com/projectdiscovery/hmap/store/hybrid.New                                                                                                
         0     0% 88.77% 26465.84kB 27.53%  github.com/projectdiscovery/nuclei/v2/internal/runner.(*Runner).RunEnumeration                                                                   
         0     0% 88.77%  4305.74kB  4.48%  github.com/projectdiscovery/nuclei/v2/internal/runner.New                                                                                        
         0     0% 88.77%  8308.62kB  8.64%  github.com/projectdiscovery/nuclei/v2/internal/runner.ParseOptions                                                                               
         0     0% 88.77% 26441.23kB 27.50%  github.com/projectdiscovery/nuclei/v2/pkg/catalog/loader.(*Store).Load                                                                           
         0     0% 88.77% 26473.54kB 27.53%  github.com/projectdiscovery/nuclei/v2/pkg/catalog/loader.(*Store).LoadTemplates                                                                  
         0     0% 88.77% 14969.33kB 15.57%  github.com/projectdiscovery/nuclei/v2/pkg/core.(*Engine).executeModelWithInput.func2.1                                                           
         0     0% 88.77%  4211.41kB  4.38%  github.com/projectdiscovery/nuclei/v2/pkg/core/inputs/hybrid.New                                                                                 
         0     0% 88.77%  2625.75kB  2.73%  github.com/projectdiscovery/nuclei/v2/pkg/operators.(*Operators).Compile                                                                         
         0     0% 88.77%  1001.51kB  1.04%  github.com/projectdiscovery/nuclei/v2/pkg/operators/extractors.(*Extractor).CompileExtractors                                                    
         0     0% 88.77%  1624.24kB  1.69%  github.com/projectdiscovery/nuclei/v2/pkg/operators/matchers.(*Matcher).CompileMatchers                                                          
         0     0% 88.77% 13715.68kB 14.26%  github.com/projectdiscovery/nuclei/v2/pkg/parsers.LoadTemplate                                                                                   
         0     0% 88.77%  4085.67kB  4.25%  github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/executer.(*Executer).Compile                                                          
         0     0% 88.77%   673.99kB   0.7%  github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/expressions.Evaluate (inline)                                                         
         0     0% 88.77%   673.99kB   0.7%  github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/expressions.evaluate                                                                  
         0     0% 88.77%  8308.62kB  8.64%  github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/protocolinit.Init 

     0     0% 88.77%   673.99kB   0.7%  github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/expressions.evaluate                                                         [33/1917]         0     0% 88.77%  8308.62kB  8.64%  github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/protocolinit.Init                                                                     
         0     0% 88.77%  8304.43kB  8.64%  github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/protocolstate.Init                                                                    
         0     0% 88.77% 14880.82kB 15.48%  github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*Request).ExecuteWithResults                                                           
         0     0% 88.77% 14755.35kB 15.35%  github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*Request).ExecuteWithResults.func1                                                     
         0     0% 88.77%  2189.42kB  2.28%  github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*requestGenerator).Make                                                                
         0     0% 88.77%   833.18kB  0.87%  github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*requestGenerator).makeHTTPRequestFromRaw                                              
         0     0% 88.77%  1350.55kB  1.40%  github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.normalizeResponseBody                                                                   
         0     0% 88.77% 12642.98kB 13.15%  github.com/projectdiscovery/nuclei/v2/pkg/templates.(*Template).UnmarshalYAML                                                                    
         0     0% 88.77%   781.40kB  0.81%  github.com/projectdiscovery/retryabledns.(*DNSData).Unmarshal                                                                                    
         0     0% 88.77%   598.97kB  0.62%  github.com/projectdiscovery/retryablehttp-go.(*Client).Do                                                                                        
         0     0% 88.77%   908.29kB  0.94%  github.com/syndtr/goleveldb/leveldb.(*Batch).putMem                                                                                              
         0     0% 88.77%   908.29kB  0.94%  github.com/syndtr/goleveldb/leveldb.(*DB).Put (inline)                                                                                           
         0     0% 88.77% 12302.37kB 12.79%  github.com/syndtr/goleveldb/leveldb.(*DB).mpoolGet                                                                                               
         0     0% 88.77% 12382.37kB 12.88%  github.com/syndtr/goleveldb/leveldb.(*DB).newMem                                                                                                 
         0     0% 88.77%   908.29kB  0.94%  github.com/syndtr/goleveldb/leveldb.(*DB).putRec                                                                                                 
         0     0% 88.77% 12382.37kB 12.88%  github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal                                                                                         
         0     0% 88.77%   908.29kB  0.94%  github.com/syndtr/goleveldb/leveldb.(*DB).writeLocked                                                                                            
         0     0% 88.77% 12470.46kB 12.97%  github.com/syndtr/goleveldb/leveldb.Open                                                                                                         
         0     0% 88.77% 12474.51kB 12.97%  github.com/syndtr/goleveldb/leveldb.OpenFile                                                                                                     
         0     0% 88.77% 12642.98kB 13.15%  gopkg.in/yaml%2ev2.(*decoder).callUnmarshaler                                                                                                    
         0     0% 88.77% 12630.96kB 13.14%  gopkg.in/yaml%2ev2.(*decoder).callUnmarshaler.func1                                                                                              
         0     0% 88.77% 12642.98kB 13.15%  gopkg.in/yaml%2ev2.(*decoder).document                                                                                                           
         0     0% 88.77% 12630.96kB 13.14%  gopkg.in/yaml%2ev2.(*decoder).mapping                                                                                                            
         0     0% 88.77% 12630.96kB 13.14%  gopkg.in/yaml%2ev2.(*decoder).mappingStruct                                                                                                      
         0     0% 88.77% 12642.98kB 13.15%  gopkg.in/yaml%2ev2.(*decoder).prepare                                                                                                            
         0     0% 88.77%   658.62kB  0.68%  gopkg.in/yaml%2ev2.(*decoder).setMapIndex                                                                                                        
         0     0% 88.77% 12642.98kB 13.15%  gopkg.in/yaml%2ev2.(*decoder).unmarshal                                                                                                          
         0     0% 88.77%  4616.31kB  4.80%  gopkg.in/yaml%2ev2.(*parser).document                                                                                                            
         0     0% 88.77%  4616.31kB  4.80%  gopkg.in/yaml%2ev2.(*parser).mapping                                                                                                             
         0     0% 88.77%  4616.31kB  4.80%  gopkg.in/yaml%2ev2.(*parser).parse                                                                                                               
         0     0% 88.77%  2706.63kB  2.81%  gopkg.in/yaml%2ev2.(*parser).sequence                                                                                                            
         0     0% 88.77%  6001.29kB  6.24%  gopkg.in/yaml%2ev2.Unmarshal (inline)                                                                                                            
         0     0% 88.77%    11258kB 11.71%  gopkg.in/yaml%2ev2.UnmarshalStrict (inline)                                                                                                      
         0     0% 88.77% 17259.29kB 17.95%  gopkg.in/yaml%2ev2.unmarshal   

    0     0% 88.77%    11258kB 11.71%  gopkg.in/yaml%2ev2.UnmarshalStrict (inline)                                                                                              [0/1917]         0     0% 88.77% 17259.29kB 17.95%  gopkg.in/yaml%2ev2.unmarshal                                                                                                                     
         0     0% 88.77%  1939.44kB  2.02%  io.Copy (inline)                                                                                                                                 
         0     0% 88.77% 39039.68kB 40.60%  main.main                                                                                                                                        
         0     0% 88.77%  1034.25kB  1.08%  math/big.(*Int).SetBytes (inline)                                                                                                                
         0     0% 88.77%  1034.25kB  1.08%  math/big.nat.setBytes                                                                                                                            
         0     0% 88.77%   570.84kB  0.59%  net/http.(*Client).Do (inline)                                                                                                                   
         0     0% 88.77%   570.84kB  0.59%  net/http.(*Client).do                                                                                                                            
         0     0% 88.77%   546.10kB  0.57%  net/http.(*Client).send                                                                                                                          
         0     0% 88.77%   716.70kB  0.75%  net/http.(*Transport).RoundTrip (partial-inline)                                                                                                 
         0     0% 88.77% 30900.61kB 32.14%  net/http.(*Transport).customDialTLS                                                                                                              
         0     0% 88.77% 38835.59kB 40.39%  net/http.(*Transport).dialConnFor                                                                                                                
         0     0% 88.77%   554.56kB  0.58%  net/http.(*persistConn).writeLoop                                                                                                                
         0     0% 88.77%  1939.44kB  2.02%  net/http.(*transferWriter).doBodyCopy                                                                                                            
         0     0% 88.77%  1939.44kB  2.02%  net/http.(*transferWriter).writeBody                                                                                                             
         0     0% 88.77%   546.10kB  0.57%  net/http.send                                                                                                                                    
         0     0% 88.77%   658.62kB  0.68%  reflect.Value.SetMapIndex                                                                                                                        
         0     0% 88.77%  2370.61kB  2.47%  regexp.Compile (inline)                                                                                                                          
         0     0% 88.77%  1754.78kB  1.83%  regexp/syntax.(*compiler).compile                                                                                                                
         0     0% 88.77%  1490.57kB  1.55%  regexp/syntax.(*compiler).rune                                                                                                                   
         0     0% 88.77%  1855.88kB  1.93%  regexp/syntax.Compile                                                                                                                            
         0     0% 88.77%   664.76kB  0.69%  runtime.doInit                                                                                                                                   
         0     0% 88.77% 39414.61kB 40.99%  runtime.main                                                                                                                                     
         0     0% 88.77%  2240.06kB  2.33%  strings.(*Builder).Grow (inline)                                                                                                                 
         0     0% 88.77%   827.97kB  0.86%  strings.Join                                                                                                                                     
         0     0% 88.77%   700.52kB  0.73%  strings.Replace                                                                                                                                  
         0     0% 88.77%   676.15kB   0.7%  strings.ReplaceAll (inline)                                                                                                                      
         0     0% 88.77%   613.25kB  0.64%  strings.ToLower                                                                                                                                  
         0     0% 88.77%  1021.94kB  1.06%  vendor/golang.org/x/crypto/cryptobyte.(*String).ReadASN1Integer                                                                                  
         0     0% 88.77%  1021.94kB  1.06%  vendor/golang.org/x/crypto/cryptobyte.(*String).readASN1BigInt

@IfrahIman can you share alloc_space data as well for the profile or attach it here?

Am running into a crash situation where the Linux OOM daemon kills nuclei while in the middle of scanning 1.3 million URLs. Here is the relevant dmesg output.

[6598432.427587] 2577 total pagecache pages
[6598432.427591] 2032 pages in swap cache
[6598432.427593] Swap cache stats: add 17148651, delete 17144550, find 16876979/19534460
[6598432.427594] Free swap  = 0kB
[6598432.427595] Total swap = 7812496kB
[6598432.427596] 2097017 pages RAM
[6598432.427604] 0 pages HighMem/MovableOnly
[6598432.427605] 59975 pages reserved
[6598432.427606] 0 pages cma reserved
[6598432.427607] 0 pages hwpoisoned
[6598432.427608] Tasks state (memory values in pages):
[6598432.427609] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
*** SNIP ***
[6598432.428355] [  15731]     0 15731     3485       38    73728      326             0 sshd
[6598432.428363] [  15830]     0 15830     2161        5    57344      460             0 bash
[6598432.428366] [  29904]     0 29904     3486        1    61440      368             0 sshd
[6598432.428368] [  30002]     0 30002     2194       17    57344      488             0 bash
[6598432.428371] [  55305]     0 55305     3485        0    65536      366             0 sshd
[6598432.428374] [  55406]     0 55406     2194       19    57344      474             0 bash
[6598432.428376] [  55437]     0 55437  3312167  1587153 25341952  1552329             0 nuclei
[6598432.428379] [  54518]     0 54518     3620        0    65536      559             0 sshd
[6598432.428382] [  54618]     0 54618     2433       11    65536      729             0 bash
[6598432.428384] [  10050]     0 10050   311062     1402   307200     2896          -900 snapd
[6598432.428387] [  23836]     0 23836     1964        3    49152      411             0 nano
[6598432.428390] [  34635]     0 34635     1760        3    53248       63             0 screen
[6598432.428393] [  38472]     0 38472     1369       16    57344        0             0 sleep
[6598432.428396] [  50954]     0 50954      395        1    36864        0             0 sleep
[6598432.428399] [  61476]     0 61476   234557    79929  1241088        0             0 python3
[6598432.428401] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-16382.scope,task=nuclei,pid=55437,uid=0
[6598432.428468] Out of memory: Killed process 55437 (nuclei) total-vm:13248668kB, anon-rss:6348612kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:24748kB oom_score_adj:0
[6598433.548751] oom_reaper: reaped process 55437 (nuclei), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Ubuntu VPS running kernel 5.4.0-121-generic with 8GB of RAM and 80GB disk. go version go1.18.3 linux/amd64.

pprof.nuclei.alloc_objects.alloc_space.inuse_objects.inuse_space.011.pb.gz pprof.nuclei.alloc_objects.alloc_space.inuse_objects.inuse_space.012.pb.gz

Hi,

I've attached the two pprof profiles, one was generated when nuclei started and the other after 15 minutes.

Best,

Not reproducible with docker, virtualbox and qemu, memory consumption is always under 300mb. While proceeding with vps testing (probably cloud hookings might interfere with memory management?), dozens of runs anyway brought up interesting crashes to investigate (reference https://github.com/projectdiscovery/nuclei/issues/2641):

Crash 1

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   2.7.8-dev

        projectdiscovery.io

[WRN] Use with caution. You are responsible for your actions.
[WRN] Developers assume no liability and are not responsible for any misuse or damage.
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0x9e2c47]

goroutine 1 [running]:
github.com/syndtr/goleveldb/leveldb.(*DB).writeLocked(0xc0001ea1a0, 0xc0002c34c0, 0xc0002c34c0, 0x1, 0x0)
    /home/user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_write.go:239 +0x587
github.com/syndtr/goleveldb/leveldb.(*DB).putRec(0xc0001ea1a0, 0x1, {0xc0018afc80, 0x21, 0x30}, {0xc001881dc8, 0x2, 0x8}, 0x30?)
    /home/user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_write.go:362 +0x466
github.com/syndtr/goleveldb/leveldb.(*DB).Put(...)
    /home/user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_write.go:372
github.com/projectdiscovery/hmap/store/disk.(*LevelDB).set(0xc0003bf6e0, {0xc0018afc80, 0x21, 0x30}, {0x0, 0x0, 0xc0018afb90?}, 0x21?)
    /home/user/go/pkg/mod/github.com/projectdiscovery/hmap@v0.0.2/store/disk/leveldb.go:90 +0x1b0
github.com/projectdiscovery/hmap/store/disk.(*LevelDB).Set(0xc000223800?, {0xc0018afb90?, 0x494?}, {0x0, 0x0, 0x0}, 0x1000000000010?)
    /home/user/go/pkg/mod/github.com/projectdiscovery/hmap@v0.0.2/store/disk/leveldb.go:95 +0x6c
github.com/projectdiscovery/hmap/store/hybrid.(*HybridMap).Set(0xc0018afb90?, {0xc0018afb90?, 0x3224ce0?}, {0x0?, 0x0?, 0xc0000021a0?})
    /home/user/go/pkg/mod/github.com/projectdiscovery/hmap@v0.0.2/store/hybrid/hybrid.go:206 +0xc4
github.com/projectdiscovery/nuclei/v2/pkg/core/inputs/hybrid.(*Input).normalizeStoreInputValue(0xc0003bf700, {0xc0018afb90?, 0x21?})
    /home/user/go/src/github.com/projectdiscovery/nuclei/v2/pkg/core/inputs/hybrid/hmap.go:113 +0x85
github.com/projectdiscovery/nuclei/v2/pkg/core/inputs/hybrid.(*Input).scanInputFromReader(0x7fffcf1f6439?, {0x2637a60?, 0xc000124658?})
    /home/user/go/src/github.com/projectdiscovery/nuclei/v2/pkg/core/inputs/hybrid/hmap.go:96 +0x95
github.com/projectdiscovery/nuclei/v2/pkg/core/inputs/hybrid.(*Input).initializeInputSources(0x0?, 0x32250e0)
    /home/user/go/src/github.com/projectdiscovery/nuclei/v2/pkg/core/inputs/hybrid/hmap.go:86 +0xf7
github.com/projectdiscovery/nuclei/v2/pkg/core/inputs/hybrid.New(0x32250e0)
    /home/user/go/src/github.com/projectdiscovery/nuclei/v2/pkg/core/inputs/hybrid/hmap.go:51 +0x17b
github.com/projectdiscovery/nuclei/v2/internal/runner.New(0x32250e0)
    /home/user/go/src/github.com/projectdiscovery/nuclei/v2/internal/runner/runner.go:164 +0x8dc
main.main()
    /home/user/go/src/github.com/projectdiscovery/nuclei/v2/cmd/nuclei/main.go:71 +0x4b5
exit status 2

Crash 2

  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   2.7.8-dev

        projectdiscovery.io

[WRN] Use with caution. You are responsible for your actions.
[WRN] Developers assume no liability and are not responsible for any misuse or damage.
[WRN] Could not load template /home/user/nuclei-templates/misconfiguration/proxy/open-proxy-localhost.yaml: the template was excluded
[WRN] Could not load template /home/user/nuclei-templates/fuzzing/adminer-panel-fuzz.yaml: the template was excluded
[WRN] Could not load template /home/user/nuclei-templates/misconfiguration/proxy/open-proxy-internal.yaml: the template was excluded
[WRN] Could not load template /home/user/nuclei-templates/misconfiguration/proxy/open-proxy-portscan.yaml: the template was excluded
[WRN] Headless flag is required for headless template /home/user/nuclei-templates/headless/dvwa-headless-automatic-login.yaml
[WRN] Could not load template /home/user/nuclei-templates/vulnerabilities/wordpress/wordpress-wp-cron.yaml: the template was excluded
[WRN] Could not load template /home/user/nuclei-templates/misconfiguration/gitlab/gitlab-user-enum.yaml: the template was excluded
[WRN] Could not load template /home/user/nuclei-templates/fuzzing/wordpress-weak-credentials.yaml: the template was excluded
[WRN] Headless flag is required for headless template /home/user/nuclei-templates/headless/postmessage-outgoing-tracker.yaml
[WRN] Headless flag is required for headless template /home/user/nuclei-templates/headless/postmessage-tracker.yaml
[WRN] Headless flag is required for headless template /home/user/nuclei-templates/headless/prototype-pollution-check.yaml
[WRN] Could not load template /home/user/nuclei-templates/fuzzing/xff-403-bypass.yaml: the template was excluded
[WRN] Could not load template /home/user/nuclei-templates/fuzzing/wordpress-plugins-detect.yaml: the template was excluded
[WRN] Could not load template /home/user/nuclei-templates/cves/2019/CVE-2019-17382.yaml: the template was excluded
[WRN] Could not load template /home/user/nuclei-templates/vulnerabilities/grafana/grafana-file-read.yaml: the template was excluded
[WRN] Could not load template /home/user/nuclei-templates/vulnerabilities/wordpress/wp-xmlrpc-brute-force.yaml: the template was excluded
[WRN] Could not load template /home/user/nuclei-templates/fuzzing/linux-lfi-fuzzing.yaml: the template was excluded
[WRN] Could not load template /home/user/nuclei-templates/fuzzing/wordpress-themes-detect.yaml: the template was excluded
[WRN] Could not load template /home/user/nuclei-templates/cves/2020/CVE-2020-16139.yaml: the template was excluded
[WRN] Headless flag is required for headless template /home/user/nuclei-templates/headless/window-name-domxss.yaml
[WRN] Could not load template /home/user/nuclei-templates/miscellaneous/ntlm-directories.yaml: the template was excluded
[WRN] Could not load template /home/user/nuclei-templates/fuzzing/mdb-database-file.yaml: the template was excluded
[WRN] Headless flag is required for headless template /home/user/nuclei-templates/headless/screenshot.yaml
[WRN] Headless flag is required for headless template /home/user/nuclei-templates/cves/2022/CVE-2022-0776.yaml
[WRN] Could not load template /home/user/nuclei-templates/fuzzing/cache-poisoning-fuzz.yaml: the template was excluded
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0x1440a29]

goroutine 1 [running]:
github.com/projectdiscovery/nuclei/v2/pkg/catalog/loader.(*Store).LoadTemplates(0xc00067db00, {0xc001ae36e0?, 0x0?, 0x0?})
    /home/user/go/src/github.com/projectdiscovery/nuclei/v2/pkg/catalog/loader/loader.go:284 +0x3a9
github.com/projectdiscovery/nuclei/v2/pkg/catalog/loader.(*Store).Load(0xc00067db00)
    /home/user/go/src/github.com/projectdiscovery/nuclei/v2/pkg/catalog/loader/loader.go:163 +0x2e
github.com/projectdiscovery/nuclei/v2/internal/runner.(*Runner).RunEnumeration(0xc00020f5e0)
    /home/user/go/src/github.com/projectdiscovery/nuclei/v2/internal/runner/runner.go:396 +0x1315
main.main()
    /home/user/go/src/github.com/projectdiscovery/nuclei/v2/cmd/nuclei/main.go:98 +0x73f
exit status 2

Crash 3 unexpectedsignal.txt

Crash 4

 go run . -l list.txt -o ev.txt -r resolvers.txt |& tee -a log2.txt 

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   2.7.8-dev

        projectdiscovery.io

[WRN] Use with caution. You are responsible for your actions.
[WRN] Developers assume no liability and are not responsible for any misuse or damage.
[INF] Using Nuclei Engine 2.7.8-dev (development)
[INF] Using Nuclei Templates 9.2.1 (latest)
[INF] Templates added in last update: 0
[INF] Templates loaded for scan: 4023
[INF] Templates clustered: 698 (Reduced 6390000 HTTP Requests)
[INF] Using Interactsh Server: oast.site
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xb5b677]

goroutine 21781 [running]:
github.com/projectdiscovery/nuclei/v2/pkg/operators/extractors.(*Extractor).ExtractRegex(0xc003ffc120, {0xc006f4ec00, 0x74})
    /home/vagrant/go/src/github.com/projectdiscovery/nuclei/v2/pkg/operators/extractors/extract.go:21 +0xd7
github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*Request).Extract(0x58?, 0x18c48c0?, 0xc003ffc120)
    /home/vagrant/go/src/github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/operators.go:66 +0xab
github.com/projectdiscovery/nuclei/v2/pkg/operators.(*Operators).Execute(0xc003ad5520, 0xc00b1227e0, 0xc00774a9e8, 0xc00774a9d8, 0x0)
    /home/vagrant/go/src/github.com/projectdiscovery/nuclei/v2/pkg/operators/operators.go:207 +0x1a3
github.com/projectdiscovery/nuclei/v2/pkg/templates.(*ClusterExecuter).Execute.func1(0xc00b122810)
    /home/vagrant/go/src/github.com/projectdiscovery/nuclei/v2/pkg/templates/cluster.go:189 +0x131
github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*Request).ExecuteWithResults.func1.1(0xc00b122810)
    /home/vagrant/go/src/github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/request.go:296 +0x2cf
github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*Request).executeRequest(0xc0076fb040, {0xc00d4582d0, 0x2a}, 0xc008f4cfa0, 0x2a?, 0x0, 0xc00774b890, 0x0?)
    /home/vagrant/go/src/github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/request.go:615 +0x28d4
github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*Request).ExecuteWithResults.func1({0xc0077864f0, 0xb}, 0x0, 0xc001f152c0)
    /home/vagrant/go/src/github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/request.go:281 +0x7a5
github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*Request).ExecuteWithResults(0xc0076fb040, {0xc00d4582d0, 0x2a}, 0xc001f152c0, 0xc001f15290, 0xc00a69f290)
    /home/vagrant/go/src/github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/request.go:333 +0x36d
github.com/projectdiscovery/nuclei/v2/pkg/templates.(*ClusterExecuter).Execute(0xc005760600, {0xc00d4582d0, 0x2a})
    /home/vagrant/go/src/github.com/projectdiscovery/nuclei/v2/pkg/templates/cluster.go:187 +0xbd
github.com/projectdiscovery/nuclei/v2/pkg/core.(*Engine).executeModelWithInput.func2.1(0x3ef37d0?, 0xc0?, {0xc00d4582d0?, 0xc004034360?})
    /home/vagrant/go/src/github.com/projectdiscovery/nuclei/v2/pkg/core/execute.go:142 +0x11e
created by github.com/projectdiscovery/nuclei/v2/pkg/core.(*Engine).executeModelWithInput.func2
    /home/vagrant/go/src/github.com/projectdiscovery/nuclei/v2/pkg/core/execute.go:129 +0x6ae
exit status 2

Crash 5

$ go run . -l list.txt -o ev.txt -r resolvers.txt |& tee -a log2.txt 

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   2.7.8-dev

        projectdiscovery.io

[WRN] Use with caution. You are responsible for your actions.
[WRN] Developers assume no liability and are not responsible for any misuse or damage.
[INF] Using Nuclei Engine 2.7.8-dev (development)
[INF] Using Nuclei Templates 9.2.1 (latest)
[INF] Templates added in last update: 0
[INF] Templates loaded for scan: 4023
[INF] Templates clustered: 698 (Reduced 6390000 HTTP Requests)
[INF] Using Interactsh Server: oast.online
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x12e8711]

goroutine 34208 [running]:
github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators.MergeMaps(...)
    /home/vagrant/go/src/github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators/maps.go:53
github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*requestGenerator).Make(0xc005569630, {0x2640010, 0xc002880540}, {0xc0052e17d0, 0x2a}, {0xc004e245d0?, 0x26?}, 0x0?, 0xc00a8717a8)
    /home/vagrant/go/src/github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/build_request.go:97 +0x871
github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*Request).ExecuteWithResults.func1({0xc004e245d0, 0x26}, 0x0, 0xc005f8bd70)
    /home/vagrant/go/src/github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/request.go:260 +0x385
github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*Request).ExecuteWithResults(0xc004bb1ba0, {0xc0052e17d0, 0x2a}, 0xc005f8bd70, 0xc005f8bda0, 0xc005f8be00)
    /home/vagrant/go/src/github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/request.go:333 +0x36d
github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/executer.(*Executer).Execute(0xc00488f080, {0xc0052e17d0, 0x2a})
    /home/vagrant/go/src/github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/executer/executer.go:67 +0x1ae
github.com/projectdiscovery/nuclei/v2/pkg/core.(*Engine).executeModelWithInput.func2.1(0x4b097d0?, 0xc0?, {0xc0052e17d0?, 0xc001806480?})
    /home/vagrant/go/src/github.com/projectdiscovery/nuclei/v2/pkg/core/execute.go:142 +0x11e
created by github.com/projectdiscovery/nuclei/v2/pkg/core.(*Engine).executeModelWithInput.func2
    /home/vagrant/go/src/github.com/projectdiscovery/nuclei/v2/pkg/core/execute.go:129 +0x6ae
exit status 2

Memory leaks under investigation in https://github.com/projectdiscovery/team-backlogs/issues/208

@Mzack9999

May I know the count of input list please !!

As I'm still facing the issue even on today itself running on 8 GB VPS

Now For a temporary solution, I extended the swap memory upto 3 GB and even running nuclei for 60k+ hosts with individual severities ( nuclei -l urls.txt -s high -dr -o out.txt ), but still also reached upto 90%++ ( 7.5 GB of 8 GB (RAM) + 2.5 GB of 3 GB (SWAP) ) of memory

Thanks & Regards, @zy9ard3

@zy9ard3 sure, I was testing with 1 million targets input list and the machine I tested with had the following settings:

Ram: 1Gb
OS: Ubuntu 22.04

Which cloud provider are you using - aws, google, digital ocean, ... ?

Thanks!

@Mzack9999

Thanks for the Clarifications !!

I'm using Digital Ocean with 8 GB / 4 AMD vCPUs / 160 GB Disk / NYC3 - Debian 11 x64

Thanks & Regards, @zy9ard3

I've tested it with VULTR and DIGITAL OCEAN vps and both crashed. My specs: 2 GB RAM UBUNTU 20

Best,

Hi, the same behaviour is noticed in aws kali-linux instance

[117708.965342] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-324.scope,task=nuclei,pid=2095424,uid=1000 [117708.991183] Out of memory: Killed process 2095424 (nuclei) total-vm:1726800kB, anon-rss:579544kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:1340kB oom_score_adj:0 [117711.154619] oom_reaper: reaped process 2095424 (nuclei), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Best regards,

Hello,

Is this issue fixed? I am still having the same issue on a new droplet of digitalocean issue with higher specs.

Best,

@IfrahIman, it should be fixed within the dev branch and not yet released on master. If you can still reproduce this with the dev branch, please let me know, and I'll have another look. Thanks!

This should be fixed with latest release of nuclei.

I have same problem. After a while it crash. I think the only option that works without any crash is: -c 1 of course all goes slow af in this way

Hi @everyone

Now it's 2023 still digital ocean has this Issue I just started nuclei to scan 2K+ hosts and after some minutes nuclei scan got stuck and writing no response or output or anything. I have tried with silent and /dev/null as well nothing worked. how to fix this ! I tried google cloud last year with 1 gb ram and 1 cpu core and never faced this type of issue. Any one fixed this issue in digital ocean ? Please help.

Thanks

@yashahmed101 Did u use non-default templates for ur scan? I got "killed" in nuclei by using old templates, so try to check if default templates work for u

Is this fixed? still having same problem... If you run it in a docker container or on VirtualBox , it wont freeze?

projectdiscovery / nuclei