Profiling and performance improvements

[krackers]

seq 1 235976 | python3 -m cProfile (which pyp) "p | pp[-1]"

shows a lot of time (~20 out of 28 seconds) is spent in string_splitter. Some of this might be unavoidable due to python slowness, but I wonder if there's some cleverness we can do to avoid such heavy performance?

The slowdown is the particular line

split_variables = dict((x, PypList([PypStr(y) for y in split_variables_raw[x]])) for x in  split_variables_raw)

It's not clear to me what exactly it's doing. If that is removed, then we get the following heavy hitters:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   235977    1.415    0.000    7.143    0.000 pyp:1940(process_line)
   235978    0.758    0.000    2.128    0.000 pyp:1561(update_history)
   235982    0.704    0.000    0.724    0.000 {built-in method builtins.eval}
   235977    0.627    0.000    0.743    0.000 pyp:1385(translate_preset_variables)
   235978    0.603    0.000    0.603    0.000 {built-in method maketrans}
  4955517    0.539    0.000    0.539    0.000 pyp:1982(<genexpr>)
   235977    0.363    0.000    0.881    0.000 pyp:1218(string_splitter)
   235977    0.335    0.000    0.817    0.000 pyp:447(__init__)
   235977    0.256    0.000    1.146    0.000 pyp:1484(safe_eval)
   471954    0.253    0.000    0.442    0.000 posixpath.py:100(split)
   471957    0.253    0.000    0.253    0.000 {method 'update' of 'dict' objects}
   235977    0.216    0.000    0.363    0.000 pyp:1204(all_meta_split)
   235979    0.208    0.000    8.508    0.000 pyp:1935(<genexpr>)
  2595764    0.194    0.000    0.194    0.000 {method 'split' of 'str' objects}
   235976    0.175    0.000    0.257    0.000 pyp:1688(format_history)
   235978    0.173    0.000    0.173    0.000 {method 'isatty' of '_io.TextIOWrapper' objects}
   235978    0.148    0.000    0.261    0.000 pyp:1647(flatten_list)

[krackers]

Found a way to reduce time of builtin.eval. Python can precompile string to bytecode, with compile which can be passed to eval. Doing this basically optimizes as much as possible I think, only remaining is translate_preset_variables and string_splitter which is going to be unavoidable. Maybe one could optimize by seeing if the variable is used in the expression first, but I don't really know if that would be any faster.

[thepyedpiper]

Great! Thank you. I’ll check out tomorrow.

T

On Tue, Jan 9, 2024 at 7:14 PM krackers @.***> wrote:

Found a way to reduce time of builtin.eval. Python can precompile string to bytecode, with compile which can be passed to eval. Doing this basically optimizes as much as possible I think, only remaining is translate_preset_variables and string_splitter which is going to be unavoidable

— Reply to this email directly, view it on GitHub https://github.com/thepyedpiper/pyp/issues/6#issuecomment-1884124835, or unsubscribe https://github.com/notifications/unsubscribe-auth/AYBHA75WL673NCDZYRZT3STYNYBSDAVCNFSM6AAAAAA37OBJQSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBUGEZDIOBTGU . You are receiving this because you are subscribed to this thread.Message ID: @.***>