Use generators in the parser

Queens-Hacks / qcumber-scraper

Scrapes SOLUS and generates structured data

3 stars 6 forks source link

Use generators in the parser #24

Open pR0Ps opened 9 years ago

pR0Ps commented 9 years ago

A lot of the data the parser gives back is gone through with for x in data loops.

Making the parser yield items instead of having to initially compute all of them and store them in memory would probably be more efficient.

uniphil commented 9 years ago

Awesome. I would love to see a comparisson of the mem usage before/after this change! Maybe I'll have some time to try it next weekend, if no one gets to it first.