zenlc2000 / pyp3

Pyed Piper tool by Toby Rosen at Sony Imageworks converted to Python 3
34 stars 13 forks source link

Evaluate pyp_dev_1.3 and flatten speed patch #15

Open bobpaul opened 7 years ago

bobpaul commented 7 years ago

https://code.google.com/archive/p/pyp/issues/29 has a patch that speeds up flatten_list() dramatically.

Is codebase originally based on the pyp_dev code (I found it posted in 2 different bug reports)? Regardless, the flatten_list() in this repo looks different than the proposed patch, so it would be good to take a look I think.

bobpaul commented 7 years ago

In issue 25 there's pyp_dev, upon which pyp3 was based. pyp_dev_1.3 from issue 29 is definitely newer. I've merged the changes in and pushed them to branch _pyp2.3dev.

One thing I am unsure of: Previously, PowerPyp.flatten_list() simply wrapped PowerPyp.Pyp.flatten_list(). Now PowerPyp.flatten_list() has the newer, faster version. Is there a reason Pyp.flatten_list() still uses the slower version? As far as I can tell, the only place PowerPipe.flatten_list() is called is inside PowerPipe.oneline(). Everywhere else explicitly calls into Pyp.flatten_list()

If this is a a special case function that can only be used in one spot, it probably should have a different name

bobpaul commented 7 years ago

Following the bug report in issue #29, I created the following test:

$ for ((i=0; $i<500000; i=$i+1)); do echo $i >> 500000.txt; done
$ time wc 500000.txt
 500000  500000 3388890

real    0m0.089s
user    0m0.092s
sys 0m0.003s
$ time cat 500000.txt | python2 pyp_dev_1.3 "len(pp)"
500000

real    0m9.611s
user    0m9.374s
sys 0m0.246s
$ time cat 500000.txt | python3 ./pyp3 "len(pp)" #from pyp2.3_dev branch
500000

real    0m7.989s
user    0m7.815s
sys 0m0.180s
$ time cat 500000.txt | python3 ./pyp3 "len(pp)" #from master
500000

real    0m8.128s
user    0m7.936s
sys 0m0.200s

idk. I'm not really seeing a speed difference as is. But then maybe it's not being used.