Open bobpaul opened 7 years ago
In issue 25 there's pyp_dev, upon which pyp3 was based. pyp_dev_1.3 from issue 29 is definitely newer. I've merged the changes in and pushed them to branch _pyp2.3dev.
One thing I am unsure of: Previously, PowerPyp.flatten_list() simply wrapped PowerPyp.Pyp.flatten_list(). Now PowerPyp.flatten_list() has the newer, faster version. Is there a reason Pyp.flatten_list() still uses the slower version? As far as I can tell, the only place PowerPipe.flatten_list() is called is inside PowerPipe.oneline(). Everywhere else explicitly calls into Pyp.flatten_list()
If this is a a special case function that can only be used in one spot, it probably should have a different name
Following the bug report in issue #29, I created the following test:
$ for ((i=0; $i<500000; i=$i+1)); do echo $i >> 500000.txt; done
$ time wc 500000.txt
500000 500000 3388890
real 0m0.089s
user 0m0.092s
sys 0m0.003s
$ time cat 500000.txt | python2 pyp_dev_1.3 "len(pp)"
500000
real 0m9.611s
user 0m9.374s
sys 0m0.246s
$ time cat 500000.txt | python3 ./pyp3 "len(pp)" #from pyp2.3_dev branch
500000
real 0m7.989s
user 0m7.815s
sys 0m0.180s
$ time cat 500000.txt | python3 ./pyp3 "len(pp)" #from master
500000
real 0m8.128s
user 0m7.936s
sys 0m0.200s
idk. I'm not really seeing a speed difference as is. But then maybe it's not being used.
https://code.google.com/archive/p/pyp/issues/29 has a patch that speeds up flatten_list() dramatically.
Is codebase originally based on the pyp_dev code (I found it posted in 2 different bug reports)? Regardless, the flatten_list() in this repo looks different than the proposed patch, so it would be good to take a look I think.