Gallopsled / pwntools

CTF framework and exploit development library
http://pwntools.com
Other
12.16k stars 1.72k forks source link

Python3 startup time approaching 500ms #1847

Open heapcrash opened 3 years ago

heapcrash commented 3 years ago

It looks like we've regressed pretty far on import times for Python3.

$ hyperfine 'python -c "from pwn import *"'
Benchmark #1: python -c "from pwn import *"
  Time (mean ± σ):     455.5 ms ± 181.3 ms    [User: 201.0 ms, System: 219.1 ms]
  Range (min … max):   385.5 ms … 970.9 ms    10 runs

The most offending modules can be seen with the following command. It looks like we can shave off a lot of time by deferring imports of requests, site, urllib3, and ssl. We should investigate more places that we can defer imports until they're actually needed.

python3 -X importtime -c 'from pwn import *' |& awk '{print $5 "|" $7}' | sort -n | tail -20
36699|urllib3
37312|pwnlib.util
38026|pwnlib.constants
38644|pwnlib.elf.corefile
38938|pwnlib.elf
39584|pwnlib.shellcraft
42324|OpenSSL.crypto
43896|OpenSSL
43917|OpenSSL.SSL
47755|requests.compat
47956|requests._internal_utils
52052|requests.utils
63720|urllib3.contrib.pyopenssl
84509|site
170627|pwnlib
177054|requests
353090|pwn.toplevel
366794|pwn
Arusekk commented 3 years ago

Requests could be a proxy object that would import the actual module on demand (pwntools does not even use it anywhere, but exports it in the namespace on a "you would need it anyway" basis). But SSL imports could be deferred too. Also, rpyc is a time consumer, and is only used by object-oriented GDB procedures. See #1753 for some work already done.