Higher benchmarks in Vim's startuptime

[weirdsmiley]

I am using hyperfine 1.17.0 to test vim's startup time. And the results are almost 2s more than what time or vim --startuptime startuptime.log are showing. I have tried not spawing the shell (-N), not loading any vim plugins, features etc. but still these benchmarks are persistent.

$ hyperfine "vim -N -u NONE -i NONE -X -cq"
Benchmark 1: vim -N -u NONE -i NONE -X -cq
  Time (mean ± σ):      2.048 s ±  0.005 s    [User: 0.027 s, System: 0.016 s]
  Range (min … max):    2.042 s …  2.055 s    10 runs

$ time vim -N -u NONE -i NONE -X -cq

real    0m0.034s
user    0m0.020s
sys     0m0.014s

vim --startuptime startuptime.log also shows smaller number than hyperfine (note: this loads the plugins). A detailed log is here.

Also, strace shows a similar picture as time with systime being at around 10ms. Detailed log is here.

[sharkdp]

Hm, interesting. Thank you for reporting this. Can you please try with --output=inherit. Maybe vim is doing something different if it doesn't detect an interactive terminal.

[sharkdp]

Something weird is definitely going on

▶ vim -N -u NONE -i NONE -X -cq > /dev/null
▶ echo $?
0

▶ vim -N -u NONE -i NONE -X -cq            
[prints some garbage to the terminal]
▶ echo $?                       
1

[sharkdp]

Oh wait, vim is neovim for me. With plain VIM, I can reproduce this. The problem seems to be STDIN, not STDOUT. Try running

time vim -N -u NONE -i NONE -X -cq < /dev/null

and you should see the same behavior with time.

Vim is printing a warning (Input is not from a terminal), and then exiting after 2s.

[sharkdp]

Look at this: here are the mysterious 2 seconds (actually 2005 ms, to be precise): https://github.com/vim/vim/blob/4c0089d696b8d1d5dc40568f25ea5738fa5bbffb/src/main.c#L2802-L2807

[sharkdp]

I think you can maybe add -s /dev/null to fix it. But not sure if that's what you really want to benchmark.

▶ hyperfine -N 'vim.basic -N -u NONE -i NONE -X -cq -s /dev/null'
Benchmark 1: vim.basic -N -u NONE -i NONE -X -cq -s /dev/null
  Time (mean ± σ):       4.1 ms ±   1.3 ms    [User: 2.1 ms, System: 1.8 ms]
  Range (min … max):     2.5 ms …   7.8 ms    382 runs

[weirdsmiley]

Look at this: here are the mysterious 2 seconds (actually 2005 ms, to be precise): https://github.com/vim/vim/blob/4c0089d696b8d1d5dc40568f25ea5738fa5bbffb/src/main.c#L2802-L2807

Wow! Let me take this info to the mailing list.

I think you can maybe add -s /dev/null to fix it. But not sure if that's what you really want to benchmark.

Yes, this helps. Now it is much more accurate (as compared to time and --startuptime).

Thanks a lot! I am closing this.