search

benjmor / tabroom_auto_summarize

Generates AI-powered summaries of meets from tabroom.com data
3 stars 1 forks source link

See if we can speed up scrape_entry_record runs by using `requests` instead of `selenium` #32

Closed bmorrissirromb closed 8 months ago

bmorrissirromb commented 8 months ago

Scraping entry records is the current bottleneck since it involves so many page loads. Could using simple requests speed up this process to the point where larger tournaments could be supported?

Example output from a curl:

<span class="twothirds nospace">
                                        <h4 class="nospace semibold">
                                                        Saketh Sripada
&amp;                                                   Laaiq Shameer
                                        </h4>

                                        <h6 class="full nospace martop semibold bluetext">
                                                Allen HS Lowery Freshman Center Sripada & Shameer
                                        </h6>
                        </span>
benjmor commented 8 months ago

So far it's like 0.1 seconds per entry difference (0.45 vs 0.55s). Not where the bottleneck is.