cli: investigate Invenio CLI speed-up

diegodelemos commented 4 years ago

Currently, the Invenio CLI runs very slowly, we should:

[x] Profile it, find the bottleneck document how to speed it up
[ ] Implement the speed up implementation (only if time permits)

Note that:

The CLI referenced here is invenio not invenio-cli.
A similar investigation/job has been done in REANA, the outputs can be consulted here.
Documenting the process is very important as the lessons learned could be applied to other repos such as https://github.com/inveniosoftware/invenio-cli.

FlorianCassayre commented 4 years ago

My attempt: https://codimd.web.cern.ch/rYl3C1qDSfWxjkfrfMOpng

KonstantinaStoikou commented 4 years ago

TL;DR

Modules with functions that have the longer running times:

pyparsing.py
core.py
cli.py
__init__
frozen _bootstrap module

Actions to be taken:

probably lazy loading of modules

Report on profiling

Used the native tool cProfile. Only the first X lines are shown of the output of cProfile.

ncalls: number of calls for this function
totime/percall: time spent inside the body of the function (by eg. loops, I/O, ...). Further calls are excluded from this count (percall is the average time per call, totime is the sum for all calls)
cumtime/percall: time spent between the time the function is called and until it returns (same thing for cumulative / per call)

`invenio --help`

Cumulative time per function

``` 4810229 function calls (4567098 primitive calls) in 2.905 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 1706/1 0.035 0.000 2.907 2.907 {built-in method builtins.exec} 1 0.000 0.000 2.907 2.907 invenio:3() 1 0.000 0.000 2.529 2.529 core.py:827(__call__) 1 0.000 0.000 2.529 2.529 cli.py:567(main) 1 0.000 0.000 2.529 2.529 core.py:716(main) 1 0.000 0.000 2.529 2.529 core.py:680(make_context) 1 0.000 0.000 2.528 2.528 core.py:1207(parse_args) 1 0.000 0.000 2.528 2.528 core.py:1039(parse_args) 2 0.000 0.000 2.528 1.264 core.py:1619(handle_parse_result) 2 0.000 0.000 2.528 1.264 core.py:108(invoke_param_callback) 1 0.000 0.000 2.528 2.528 core.py:948(show_help) 1 0.000 0.000 2.528 2.528 core.py:566(get_help) 1 0.000 0.000 2.528 2.528 core.py:969(get_help) 1 0.000 0.000 2.528 2.528 core.py:989(format_help) 1 0.000 0.000 2.528 2.528 core.py:1135(format_options) 1 0.000 0.000 2.527 2.527 core.py:1179(format_commands) 1 0.000 0.000 2.527 2.527 cli.py:548(list_commands) 44 0.000 0.000 2.331 0.053 __init__.py:2432(load) 9 0.000 0.000 2.000 0.222 ext.py:9() 11 0.000 0.000 1.836 0.167 cli.py:513(_load_plugin_commands) 44 0.001 0.000 1.412 0.032 __init__.py:2457(require) 47 0.023 0.000 1.343 0.029 __init__.py:716(resolve) 1314/43 0.006 0.000 1.322 0.031 :966(_find_and_load) 1309/42 0.004 0.000 1.321 0.031 :936(_find_and_load_unlocked) 1209/34 0.005 0.000 1.317 0.039 :651(_load_unlocked) 1151/34 0.003 0.000 1.316 0.039 :672(exec_module) 1499/34 0.001 0.000 1.315 0.039 :211(_call_with_frames_removed) 419/90 0.001 0.000 1.218 0.014 {built-in method builtins.__import__} 3040 0.004 0.000 1.090 0.000 __init__.py:2736(requires) 2918 0.001 0.000 1.010 0.000 __init__.py:3020(_dep_map) 88 0.003 0.000 1.008 0.011 __init__.py:3028(_compute_dependencies) 44 0.000 0.000 0.919 0.021 __init__.py:2447(resolve) 1828 0.003 0.000 0.911 0.000 __init__.py:3077(parse_requirements) 971 0.007 0.000 0.905 0.001 __init__.py:3100(__init__) 11716/11715 0.003 0.000 0.905 0.000 {method 'extend' of 'list' objects} 971 0.007 0.000 0.881 0.001 requirements.py:88(__init__) 1660/973 0.003 0.000 0.854 0.001 pyparsing.py:1608(parseString) 130808/973 0.225 0.000 0.849 0.001 pyparsing.py:1370(_parseNoCache) 16909/973 0.048 0.000 0.845 0.001 pyparsing.py:3397(parseImpl) 1 0.000 0.000 0.759 0.759 __init__.py:402() 1 0.000 0.000 0.757 0.757 loaders.py:8() 15057/2326 0.052 0.000 0.735 0.000 pyparsing.py:3547(parseImpl) 3 0.000 0.000 0.691 0.230 cli.py:368(load_app) 1 0.000 0.000 0.691 0.691 cli.py:103(call_factory) 1 0.000 0.000 0.691 0.691 app.py:143(create_cli_app) 2/1 0.000 0.000 0.691 0.691 app.py:86(_create_app) 4 0.000 0.000 0.686 0.172 app.py:221(_loader) ```

Total time per function

``` 4810163 function calls (4567032 primitive calls) in 3.100 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 130808/973 0.258 0.000 0.936 0.001 pyparsing.py:1370(_parseNoCache) 73817/72146 0.094 0.000 0.136 0.000 pyparsing.py:372(__init__) 9143/1590 0.093 0.000 0.262 0.000 sre_parse.py:470(_parse) 1151 0.090 0.000 0.090 0.000 {built-in method marshal.loads} 15921 0.081 0.000 0.202 0.000 version.py:198(__init__) 15087/1544 0.075 0.000 0.228 0.000 sre_compile.py:64(_compile) 3987/3935 0.074 0.000 0.266 0.000 {built-in method builtins.__build_class__} 546590/544849 0.073 0.000 0.082 0.000 {built-in method builtins.isinstance} 15057/2326 0.060 0.000 0.801 0.000 pyparsing.py:3547(parseImpl) 16909/973 0.054 0.000 0.932 0.001 pyparsing.py:3397(parseImpl) 23008 0.042 0.000 0.042 0.000 {method 'search' of '_sre.SRE_Pattern' objects} 63171 0.041 0.000 0.062 0.000 pyparsing.py:2431(parseImpl) 21390 0.038 0.000 0.052 0.000 pyparsing.py:420(__setitem__) 83592 0.037 0.000 0.042 0.000 pyparsing.py:1351(preParse) 22420/8970 0.035 0.000 0.044 0.000 sre_parse.py:173(getwidth) 1706/1 0.035 0.000 3.102 3.102 {built-in method builtins.exec} 14544 0.035 0.000 0.041 0.000 version.py:343(_cmpkey) 73817 0.032 0.000 0.052 0.000 pyparsing.py:363(__new__) 7693 0.031 0.000 0.031 0.000 {built-in method posix.stat} 94214 0.031 0.000 0.031 0.000 sre_parse.py:232(__next) 27948 0.031 0.000 0.078 0.000 enum.py:803(__and__) 13645 0.030 0.000 0.080 0.000 pyparsing.py:663(__iadd__) 73367 0.027 0.000 0.039 0.000 sre_parse.py:163(__getitem__) 3708 0.027 0.000 0.039 0.000 sre_compile.py:250(_optimize_charset) 61289 0.026 0.000 0.026 0.000 {built-in method __new__ of type object at 0x9d12c0} 322184/316055 0.026 0.000 0.028 0.000 {built-in method builtins.len} 302618 0.026 0.000 0.026 0.000 {method 'append' of 'list' objects} 62354 0.024 0.000 0.024 0.000 pyparsing.py:209(__init__) 47 0.024 0.001 1.413 0.030 __init__.py:716(resolve) 59376 0.023 0.000 0.025 0.000 enum.py:517(__new__) 6497 0.022 0.000 0.028 0.000 pyparsing.py:668() 29511 0.021 0.000 0.021 0.000 {method 'match' of '_sre.SRE_Pattern' objects} 59390 0.020 0.000 0.049 0.000 enum.py:267(__call__) 5762/1544 0.020 0.000 0.268 0.000 sre_parse.py:407(_parse_sub) 77827 0.020 0.000 0.043 0.000 sre_parse.py:253(get) 6808 0.018 0.000 0.609 0.000 re.py:286(_compile) 155263 0.018 0.000 0.018 0.000 {method 'group' of '_sre.SRE_Match' objects} 5614 0.017 0.000 0.049 0.000 markers.py:249(default_environment) 2365 0.017 0.000 0.071 0.000 :1233(find_spec) 40974/38560 0.016 0.000 0.049 0.000 {built-in method builtins.hasattr} 5979/5614 0.015 0.000 0.084 0.000 markers.py:214(_evaluate_markers) 46623/46615 0.015 0.000 0.015 0.000 {built-in method builtins.getattr} ```

`invenio --version`

Cumulative time per function

``` Python 3.6.9 Flask 1.1.2 Werkzeug 0.16.1 542071 function calls (533323 primitive calls) in 0.373 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 283/1 0.008 0.000 0.373 0.373 {built-in method builtins.exec} 1 0.000 0.000 0.373 0.373 invenio:3() 305/1 0.001 0.000 0.372 0.372 :966(_find_and_load) 305/1 0.001 0.000 0.372 0.372 :936(_find_and_load_unlocked) 335/2 0.000 0.000 0.372 0.186 :211(_call_with_frames_removed) 288/2 0.001 0.000 0.372 0.186 :651(_load_unlocked) 249/2 0.001 0.000 0.372 0.186 :672(exec_module) 49/4 0.000 0.000 0.285 0.071 {built-in method builtins.__import__} 2 0.000 0.000 0.283 0.141 ext.py:9() 1 0.000 0.000 0.283 0.283 __init__.py:168() 1 0.000 0.000 0.139 0.139 __init__.py:16() 2 0.000 0.000 0.119 0.060 __init__.py:11() 741/717 0.002 0.000 0.100 0.000 __init__.py:2038(find_on_path) 993 0.002 0.000 0.093 0.000 re.py:286(_compile) 209 0.000 0.000 0.091 0.000 re.py:231(compile) 1 0.000 0.000 0.089 0.089 cli.py:9() 203 0.001 0.000 0.089 0.000 sre_compile.py:557(compile) 1 0.000 0.000 0.088 0.088 factory.py:9() 4 0.000 0.000 0.086 0.021 __init__.py:477(get_distribution) 3 0.000 0.000 0.084 0.028 factory.py:69(app_class) 4 0.000 0.000 0.082 0.021 __init__.py:355(get_provider) 3 0.000 0.000 0.082 0.027 __init__.py:892(require) 3 0.000 0.000 0.082 0.027 __init__.py:716(resolve) 3 0.000 0.000 0.081 0.027 __init__.py:963(__init__) 3 0.000 0.000 0.081 0.027 __init__.py:1005(scan) 999/992 0.013 0.000 0.069 0.000 {built-in method builtins.__build_class__} 1 0.000 0.000 0.063 0.063 __init__.py:14() 2 0.000 0.000 0.050 0.025 __init__.py:3236(_call_aside) 1 0.000 0.000 0.050 0.050 __init__.py:3253(_initialize_master_working_set) 203 0.000 0.000 0.047 0.000 sre_compile.py:542(_code) 1344 0.003 0.000 0.047 0.000 __init__.py:2125(distributions_from_metadata) 66 0.002 0.000 0.044 0.001 {built-in method builtins.sorted} 57 0.000 0.000 0.044 0.001 __init__.py:2012(_by_version_descending) 1827/203 0.009 0.000 0.043 0.000 sre_compile.py:64(_compile) 203 0.001 0.000 0.040 0.000 sre_parse.py:844(parse) 1 0.000 0.000 0.038 0.038 test.py:10() 20 0.000 0.000 0.038 0.002 __init__.py:612(add_entry) 691/203 0.002 0.000 0.035 0.000 sre_parse.py:407(_parse_sub) 249 0.001 0.000 0.035 0.000 :743(get_code) 953/215 0.012 0.000 0.035 0.000 sre_parse.py:470(_parse) 1 0.000 0.000 0.031 0.031 requirements.py:4() 684 0.001 0.000 0.030 0.000 __init__.py:2027(_by_version) ```

Total time per function

``` Python 3.6.9 Flask 1.1.2 Werkzeug 0.16.1 542071 function calls (533323 primitive calls) in 0.368 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 249 0.021 0.000 0.021 0.000 {built-in method marshal.loads} 999/992 0.013 0.000 0.069 0.000 {built-in method builtins.__build_class__} 3098 0.012 0.000 0.030 0.000 version.py:198(__init__) 953/215 0.012 0.000 0.035 0.000 sre_parse.py:470(_parse) 761 0.011 0.000 0.011 0.000 {built-in method posix.listdir} 2903 0.010 0.000 0.010 0.000 {built-in method posix.stat} 686 0.009 0.000 0.012 0.000 sre_compile.py:250(_optimize_charset) 283/1 0.009 0.000 0.369 0.369 {built-in method builtins.exec} 1827/203 0.009 0.000 0.043 0.000 sre_compile.py:64(_compile) 7979 0.007 0.000 0.019 0.000 enum.py:803(__and__) 17230 0.006 0.000 0.007 0.000 enum.py:517(__new__) 20380 0.005 0.000 0.005 0.000 sre_parse.py:232(__next) 38363/38362 0.005 0.000 0.005 0.000 {built-in method builtins.isinstance} 3099 0.005 0.000 0.005 0.000 {method 'search' of '_sre.SRE_Pattern' objects} 17244 0.005 0.000 0.016 0.000 enum.py:267(__call__) 702 0.005 0.000 0.020 0.000 :1233(find_spec) 1721 0.005 0.000 0.005 0.000 version.py:343(_cmpkey) ```

`invenio access list`

Cumulative time per function

``` admin-access: superuser-access: 4809020 function calls (4565885 primitive calls) in 2.844 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 1705/1 0.034 0.000 2.846 2.846 {built-in method builtins.exec} 1 0.000 0.000 2.846 2.846 invenio:3() 1 0.000 0.000 2.476 2.476 core.py:827(__call__) 1 0.000 0.000 2.476 2.476 cli.py:567(main) 1 0.000 0.000 2.476 2.476 core.py:716(main) 2/1 0.000 0.000 2.476 2.476 core.py:1221(invoke) 44 0.000 0.000 2.306 0.052 __init__.py:2432(load) 9 0.000 0.000 1.936 0.215 ext.py:9() 2 0.000 0.000 1.819 0.909 core.py:1292(resolve_command) 1 0.000 0.000 1.819 1.819 cli.py:526(get_command) 1 0.000 0.000 1.819 1.819 cli.py:513(_load_plugin_commands) 44 0.001 0.000 1.415 0.032 __init__.py:2457(require) 47 0.023 0.000 1.321 0.028 __init__.py:716(resolve) 1313/42 0.006 0.000 1.285 0.031 :966(_find_and_load) 1308/41 0.004 0.000 1.284 0.031 :936(_find_and_load_unlocked) 1208/33 0.004 0.000 1.280 0.039 :651(_load_unlocked) 1150/33 0.003 0.000 1.279 0.039 :672(exec_module) 1498/33 0.001 0.000 1.278 0.039 :211(_call_with_frames_removed) 419/90 0.001 0.000 1.185 0.013 {built-in method builtins.__import__} 3040 0.004 0.000 1.099 0.000 __init__.py:2736(requires) 2918 0.001 0.000 1.028 0.000 __init__.py:3020(_dep_map) 88 0.003 0.000 1.027 0.012 __init__.py:3028(_compute_dependencies) 1828 0.003 0.000 0.922 0.001 __init__.py:3077(parse_requirements) 971 0.008 0.000 0.916 0.001 __init__.py:3100(__init__) 11714/11713 0.003 0.000 0.916 0.000 {method 'extend' of 'list' objects} 971 0.007 0.000 0.892 0.001 requirements.py:88(__init__) 44 0.000 0.000 0.890 0.020 __init__.py:2447(resolve) 1660/973 0.004 0.000 0.865 0.001 pyparsing.py:1608(parseString) 130808/973 0.232 0.000 0.860 0.001 pyparsing.py:1370(_parseNoCache) 16909/973 0.048 0.000 0.856 0.001 pyparsing.py:3397(parseImpl) 15057/2326 0.052 0.000 0.747 0.000 pyparsing.py:3547(parseImpl) 1 0.000 0.000 0.731 0.731 __init__.py:402() 1 0.000 0.000 0.729 0.729 loaders.py:8() 3 0.000 0.000 0.657 0.219 core.py:1060(invoke) 4/3 0.000 0.000 0.657 0.219 core.py:572(invoke) ```

Total time per function

``` admin-access: superuser-access: 4809008 function calls (4565873 primitive calls) in 2.921 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 130808/973 0.261 0.000 0.898 0.001 pyparsing.py:1370(_parseNoCache) 1150 0.087 0.000 0.087 0.000 {built-in method marshal.loads} 15921 0.082 0.000 0.205 0.000 version.py:198(__init__) 9143/1590 0.079 0.000 0.221 0.000 sre_parse.py:470(_parse) 73817/72146 0.078 0.000 0.121 0.000 pyparsing.py:372(__init__) 546738/544997 0.068 0.000 0.078 0.000 {built-in method builtins.isinstance} 15087/1544 0.062 0.000 0.194 0.000 sre_compile.py:64(_compile) 3986/3934 0.058 0.000 0.259 0.000 {built-in method builtins.__build_class__} 15057/2326 0.054 0.000 0.776 0.000 pyparsing.py:3547(parseImpl) 16909/973 0.051 0.000 0.894 0.001 pyparsing.py:3397(parseImpl) 21390 0.045 0.000 0.060 0.000 pyparsing.py:420(__setitem__) 23008 0.042 0.000 0.042 0.000 {method 'search' of '_sre.SRE_Pattern' objects} 63171 0.038 0.000 0.057 0.000 pyparsing.py:2431(parseImpl) 14544 0.036 0.000 0.042 0.000 version.py:343(_cmpkey) 1705/1 0.035 0.000 2.923 2.923 {built-in method builtins.exec} 83592 0.034 0.000 0.040 0.000 pyparsing.py:1351(preParse) 73817 0.032 0.000 0.051 0.000 pyparsing.py:363(__new__) 7690 0.030 0.000 0.030 0.000 {built-in method posix.stat} 13645 0.029 0.000 0.069 0.000 pyparsing.py:663(__iadd__) 22420/8970 0.028 0.000 0.035 0.000 sre_parse.py:173(getwidth) ```

`invenio routes`

Cumulative time per function

``` Endpoint Methods Rule ------------------------------- --------- ---------------------------------------------- actionroles.action_view POST /admin/actionroles/action/ actionroles.ajax_lookup GET /admin/actionroles/ajax/lookup/ ... 4811901 function calls (4568770 primitive calls) in 2.931 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 1705/1 0.035 0.000 2.933 2.933 {built-in method builtins.exec} 1 0.000 0.000 2.933 2.933 invenio:3() 1 0.000 0.000 2.555 2.555 core.py:827(__call__) 1 0.000 0.000 2.555 2.555 cli.py:567(main) 1 0.000 0.000 2.555 2.555 core.py:716(main) 1 0.000 0.000 2.555 2.555 core.py:1221(invoke) 44 0.000 0.000 2.381 0.054 __init__.py:2432(load) 9 0.000 0.000 2.007 0.223 ext.py:9() 1 0.000 0.000 1.858 1.858 core.py:1292(resolve_command) 1 0.000 0.000 1.858 1.858 cli.py:526(get_command) 1 0.000 0.000 1.858 1.858 cli.py:513(_load_plugin_commands) 44 0.001 0.000 1.451 0.033 __init__.py:2457(require) 47 0.023 0.000 1.379 0.029 __init__.py:716(resolve) 1313/42 0.006 0.000 1.330 0.032 :966(_find_and_load) 1308/41 0.004 0.000 1.329 0.032 :936(_find_and_load_unlocked) 1208/33 0.004 0.000 1.325 0.040 :651(_load_unlocked) 1150/33 0.003 0.000 1.324 0.040 :672(exec_module) 1498/33 0.001 0.000 1.323 0.040 :211(_call_with_frames_removed) 419/90 0.001 0.000 1.231 0.014 {built-in method builtins.__import__} 3040 0.004 0.000 1.104 0.000 __init__.py:2736(requires) 2918 0.001 0.000 1.030 0.000 __init__.py:3020(_dep_map) 88 0.003 0.000 1.029 0.012 __init__.py:3028(_compute_dependencies) 44 0.000 0.000 0.930 0.021 __init__.py:2447(resolve) 1828 0.003 0.000 0.920 0.001 __init__.py:3077(parse_requirements) 11714/11713 0.003 0.000 0.913 0.000 {method 'extend' of 'list' objects} 971 0.008 0.000 0.913 0.001 __init__.py:3100(__init__) 971 0.008 0.000 0.888 0.001 requirements.py:88(__init__) 1660/973 0.003 0.000 0.860 0.001 pyparsing.py:1608(parseString) 130808/973 0.220 0.000 0.855 0.001 pyparsing.py:1370(_parseNoCache) 16909/973 0.049 0.000 0.851 0.001 pyparsing.py:3397(parseImpl) 1 0.000 0.000 0.761 0.761 __init__.py:402() 1 0.000 0.000 0.759 0.759 loaders.py:8() 15057/2326 0.054 0.000 0.740 0.000 pyparsing.py:3547(parseImpl) ```

Total time per function

``` Endpoint Methods Rule ------------------------------- --------- ---------------------------------------------- actionroles.action_view POST /admin/actionroles/action/ actionroles.ajax_lookup GET /admin/actionroles/ajax/lookup/ actionroles.ajax_update POST /admin/actionroles/ajax/update/ ... 4811883 function calls (4568752 primitive calls) in 3.158 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 130808/973 0.249 0.000 0.898 0.001 pyparsing.py:1370(_parseNoCache) 1150 0.097 0.000 0.097 0.000 {built-in method marshal.loads} 15921 0.091 0.000 0.226 0.000 version.py:198(__init__) 9143/1590 0.084 0.000 0.238 0.000 sre_parse.py:470(_parse) 73817/72146 0.079 0.000 0.120 0.000 pyparsing.py:372(__init__) 547057/545316 0.073 0.000 0.085 0.000 {built-in method builtins.isinstance} 15087/1544 0.066 0.000 0.205 0.000 sre_compile.py:64(_compile) 15057/2326 0.064 0.000 0.779 0.000 pyparsing.py:3547(parseImpl) 3986/3934 0.064 0.000 0.269 0.000 {built-in method builtins.__build_class__} 16909/973 0.051 0.000 0.894 0.001 pyparsing.py:3397(parseImpl) 23008 0.045 0.000 0.045 0.000 {method 'search' of '_sre.SRE_Pattern' objects} 21390 0.044 0.000 0.059 0.000 pyparsing.py:420(__setitem__) 14544 0.040 0.000 0.046 0.000 version.py:343(_cmpkey) 63171 0.039 0.000 0.058 0.000 pyparsing.py:2431(parseImpl) 1705/1 0.039 0.000 3.161 3.161 {built-in method builtins.exec} 83592 0.035 0.000 0.040 0.000 pyparsing.py:1351(preParse) 7690 0.033 0.000 0.033 0.000 {built-in method posix.stat} 73817 0.031 0.000 0.049 0.000 pyparsing.py:363(__new__) 22420/8970 0.030 0.000 0.038 0.000 sre_parse.py:173(getwidth) 47 0.029 0.001 1.412 0.030 __init__.py:716(resolve) 94214 0.028 0.000 0.028 0.000 sre_parse.py:232(__next) 13645 0.028 0.000 0.070 0.000 pyparsing.py:663(__iadd__) 27948 0.028 0.000 0.072 0.000 enum.py:803(__and__) 1 0.027 0.027 0.029 0.029 cli.py:899(routes_command) 322448/316319 0.025 0.000 0.026 0.000 {built-in method builtins.len} 61289 0.025 0.000 0.025 0.000 {built-in method __new__ of type object at 0x9d12c0} 3708 0.024 0.000 0.035 0.000 sre_compile.py:250(_optimize_charset) 73367 0.024 0.000 0.035 0.000 sre_parse.py:163(__getitem__) 302386 0.024 0.000 0.024 0.000 {method 'append' of 'list' objects} 62354 0.023 0.000 0.023 0.000 pyparsing.py:209(__init__) 59376 0.021 0.000 0.023 0.000 enum.py:517(__new__) 29511 0.021 0.000 0.021 0.000 {method 'match' of '_sre.SRE_Pattern' objects} 155263 0.020 0.000 0.020 0.000 {method 'group' of '_sre.SRE_Match' objects} ```

Conclusions

pyparsing.py has the longest total times (time spend inside the body of the function, call to other functions excluded) and it also significantly contibutes to the cumulative time.
core.py (of click package) has long cumulative times especially in the --help command.
cli.py has long cumulative times as well.
frozen _bootstrap module has long cumulative times too.
__init__

Full report codiMD here.

topless commented 4 years ago

Draft codiMD report

KonstantinaStoikou commented 4 years ago

Removing wsgi loop

`invenio --help`

Before:

Time: ~2.05s

Graph:

After:

Time: ~1.92s

Graph:

`invenio routes`

Before:

Time: ~2.00s

Graph:

After:

Time: ~1.88s

Graph:

Observations

invenio-base registers a cli application itself under inveniomanage command. The same functionality is also provided when invenio-app creates the cli. Should we contain the instantiation of apps only in the invenio-app and use invenio-base for their implementation?
Whenever we execute a cli command, the create_app_factory is invoked three times, for creating the apis, for the ui and finally to create the app which is a combination of the previous two.
We can possibly use the Indico approach for lazy loading.
Caching the applications instead of reinstantiating them, possibly complicated solution due to caching validation. Maybe caching only some parts of the app would be easier.
Needs contribution from architects on how to restructure the application initialization.

inveniosoftware / invenio