Vegeta is crashing my machine on high requests per second

ambition-consulting commented 4 years ago

Version and Runtime

vegeta -version
Version: 12.8.3
Commit:
Runtime: go1.14 darwin/amd64
Date: 2020-03-25T14:41:40Z

Expected Behaviour

I expected my system to not crash.

Actual Behaviour

I got a OSX crash, after reboot, I get:

panic(cpu 6 caller 0xffffff8012ec1f4f): assertion failed: ifp->if_sndbyte_unsent >= 0, file: /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-6153.141.1/bsd/netinet/in_pcb.c, line: 3977 Backtrace (CPU 6), Frame : Return Address 0xffffffa3e35bb870 : 0xffffff801271a65d 0xffffffa3e35bb8c0 : 0xffffff8012854a75 0xffffffa3e35bb900 : 0xffffff80128465fe 0xffffffa3e35bb950 : 0xffffff80126c0a40 0xffffffa3e35bb970 : 0xffffff8012719d27 0xffffffa3e35bba70 : 0xffffff801271a117 0xffffffa3e35bbac0 : 0xffffff8012ec1abc 0xffffffa3e35bbb30 : 0xffffff8012ec1f4f 0xffffffa3e35bbb40 : 0xffffff8012ce05ed 0xffffffa3e35bbb50 : 0xffffff8012ce01c3 0xffffffa3e35bbbb0 : 0xffffff8012ce08fa 0xffffffa3e35bbbe0 : 0xffffff8012b513c5 0xffffffa3e35bbcd0 : 0xffffff8012cd333b 0xffffffa3e35bbdf0 : 0xffffff8012ca7efc 0xffffffa3e35bbe20 : 0xffffff8012ca14f0 0xffffffa3e35bbee0 : 0xffffff8012ca12b0 0xffffffa3e35bbf40 : 0xffffff8012d84c37 0xffffffa3e35bbfa0 : 0xffffff80126c1206

BSD process name corresponding to current thread: vegeta Boot args: chunklist-security-epoch=0 -chunklist-no-rev2-dev

Mac OS version: 19G2021

Kernel version: Darwin Kernel Version 19.6.0: Thu Jun 18 20:49:00 PDT 2020; root:xnu-6153.141.1~1/RELEASE_X86_64 Kernel UUID: 1D3A0F3D-D908-397B-BD16-8F48C0823A2E Kernel slide: 0x0000000012400000 Kernel text base: 0xffffff8012600000 __HIB text base: 0xffffff8012500000 System model name: MacBookPro16,2 (Mac-5F9802EFE386AA28) System shutdown begun: NO

Steps to Reproduce

Run ramp-requests.py multiple times

ambition-consulting commented 4 years ago

I also increased my maxfiles, macproc etc. following https://gist.github.com/tombigel/d503800a282fcadbee14b537735d202c

tsenart commented 4 years ago

What's your macOS version? Never seen this before. Maybe recompiling vegeta with latest Go in your machine solves it? Can you try?

ambition-consulting commented 4 years ago

Latest Catalina in a MacBook Pro 13. Can compile also, yes. Currently it is running from brew installation.

tsenart commented 4 years ago

If you do a simple test, without ramp-requests.py, does it also crash?

ambition-consulting commented 4 years ago

Nope, simple tests have never crashed before. Ramp-Request will also be succcessful for 2-3 iterations.

I am using https://apps.apple.com/us/app/amphetamine/ to keep the session alive. I will try running it without as well.

The applications I am testing are not on the same machine, so I don't think there is any other program causing resource starvation.

ambition-consulting commented 4 years ago

Still crashing despite having compiled it myself. Amphetamine is also not involved.

ambition-consulting commented 4 years ago

Also, after several thousands of attacks, my osx will not accept inbound or outbound network communication anymore, despite having high ulimits and lsof only showing 4000 IPv4 entries:

% ulimit -aS -t: cpu time (seconds) unlimited -f: file size (blocks) unlimited -d: data seg size (kbytes) unlimited -s: stack size (kbytes) 8192 -c: core file size (blocks) 0 -v: address space (kbytes) unlimited -l: locked-in-memory size (kbytes) unlimited -u: processes 2048 -n: file descriptors 200000

Disabling and enabling Wifi helps with that problem. Not sure if it is related to the crash though.

ambition-consulting commented 4 years ago

FYI, here is the list of my installations - I only use brew:

brew list bash-completion fribidi go jq libtiff openssl@1.1 python@3.8 webp cairo fswatch gobject-introspection ldns lua p7zip qt xz docker gd graphite2 libcbor lzo pango readline docker-machine gdbm harfbuzz libcerf maven pcre rename dos2unix gettext htop libffi ncurses pixman sqlite fontconfig glib icu4c libfido2 oniguruma pkg-config vegeta freetype gnuplot jpeg libpng openjdk pstree watch

brew list --cask docker google-chrome intellij-idea iterm2 microsoft-teams

ambition-consulting commented 4 years ago

I wanted to monitor lsof, but only found sloth, which is no longer accepted by osx for security/bullshit reasons. Any other idea on how to debug this problem?

ambition-consulting commented 4 years ago

FYI after I replaced my WIFI network connection with regulat network cable, the crashing has stopped - so far. I still experience open file descriptor starvation on network connections though.

tsenart commented 1 year ago

Closing this due to staleness.

tsenart / vegeta