I think regex for searching context is ineffective. ([^\n]*) means matching everything (except newline) while it might not even match our link regex in the middle.
I change it to offset search based on the link instead (We know cli only searches for link and is very fast)
Result
I think it's depends on the context too. I have gotten like x2-x4 better performance. On nu.nl seems to give x10. Performance can be varied on CPU/disk/OS so more test (both better/worse) is always appreciated.
New version
$ time (python linkfinder.py -i https://www.nu.nl/ -d)
real 0m37.334s
user 0m16.256s
sys 0m0.092s
Old version
$ time (python linkfinder.py -i https://www.nu.nl/ -d)
real 6m36.491s
user 5m58.722s
sys 0m0.083s
More
I also fixed random crash from python2 (I'm not sure how it happens, more like issue with python2 regex + ordering). Does not affect python3.
I use this code to confirm that everything is ok during transit between old and new version (Note there will always be dupe from different URL when using -d mode)
import re
d = {}
regex = re.compile("
<a href='([^'\n]*)' class='text'>")
f = open("new_today.html", "r")
w1 = open("tmp1", "w+")
content = f.read()
items = re.findall(regex, content)
for item in items:
if item in d:
print("Dupe %s" % (item))
d[item] = 1
w1.write(item + "\n")
f = open("old_today.html", "r")
w2 = open("tmp2", "w+")
content = f.read()
items = re.findall(regex, content)
for item in items:
if item not in d:
print("Miss from new %s" % (item))
d[item] = 2
w2.write(item + "\n")
for item in d:
if d[item] == 1:
print("Miss from old %s" % (item))
w1.close()
w2.close()
print("Done")
# Further action
- I would love any recommendation, test, feedback before we merge into main. I don't want anything to crash :P
Btw, because the way of our original regex:
I have to use m.group(1) instead of m.group(0) during re.finditer because 0 will also return double quotes/single quote. I already wrote the description but still feeling unsure.
Should I go back to regex_group_name or stay like this or any better options ?
Either choice will not affect our original LinkFinder, but it will affect anyone who decide to import our module and use their own custom regex.
Wow! This is indeed a lot faster. Thanks for the insight with why the regex was slow. It makes a lot of sense but didn't realise at the beginning. Can also confirm it works so will merge.
Issue
Searching link with context is too slow
Cause and solution
Result
New version
Old version
More
d = {} regex = re.compile("
f = open("new_today.html", "r") w1 = open("tmp1", "w+") content = f.read() items = re.findall(regex, content) for item in items: if item in d: print("Dupe %s" % (item)) d[item] = 1 w1.write(item + "\n")
f = open("old_today.html", "r") w2 = open("tmp2", "w+") content = f.read() items = re.findall(regex, content) for item in items: if item not in d: print("Miss from new %s" % (item)) d[item] = 2 w2.write(item + "\n")
for item in d: if d[item] == 1: print("Miss from old %s" % (item))
w1.close() w2.close() print("Done")
Btw, because the way of our original regex: I have to use
m.group(1)instead ofm.group(0)duringre.finditerbecause 0 will also return double quotes/single quote. I already wrote the description but still feeling unsure. Should I go back to regex_group_name or stay like this or any better options ?Either choice will not affect our original LinkFinder, but it will affect anyone who decide to import our module and use their own custom regex.
Wow! This is indeed a lot faster. Thanks for the insight with why the regex was slow. It makes a lot of sense but didn't realise at the beginning. Can also confirm it works so will merge.