Add a handful of parsing performance boosts

eregs / regulations-parser

Parser for U.S. federal regulations and other regulatory information

Creative Commons Zero v1.0 Universal

36 stars 40 forks source link

Add a handful of parsing performance boosts #340

Closed cmc333333 closed 7 years ago

cmc333333 commented 7 years ago

I used kcachegrind to inspect the results of a run on 27 CFR 478's full history and tried to optimize the biggest pain points. This led to four changes:

Caching the results of singular/plural lookups
Refraining from fetching Entry.contents for all entries
Allow QuickSearchable's to have hard-coded regexes (to be more descriptive)
Re-use QuickSearchable's parsed regex

coveralls commented 7 years ago

Coverage increased (+0.02%) to 91.744% when pulling fad5f2b1f54a66ad779788f887f77a63cea834a4 on cmc333333:speedups into e96412ed7d76201dd7fafad2a75cf69603af1880 on eregs:master.

cmc333333 commented 7 years ago

Comparing https://travis-ci.org/eregs/regulations-parser/builds/180844504 and https://travis-ci.org/eregs/regulations-parser/builds/183490176, I see a 20-25% drop in execution time.

coveralls commented 7 years ago

Coverage increased (+0.04%) to 91.768% when pulling 32f94e058f33672c05dbcb9051a402d584fe4f4d on cmc333333:speedups into e96412ed7d76201dd7fafad2a75cf69603af1880 on eregs:master.

cmc333333 commented 7 years ago

I'm more confident merging this now that ATF is regression tested. We're still seeing 20-25% improvements