search

KaylaCrush / advocacy-maps

The good governance project (GGP) is a non-partisan democracy reform group.
https://mapletestimony.org
MIT License
0 stars 1 forks source link

change page.tables datatype from dataframe to list of list #20

Closed KaylaCrush closed 1 year ago

KaylaCrush commented 1 year ago

Right now, every time we get a table we convert it from a list of lists to a pandas dataframe. But pandas is slow, and ultimately I need it to be saved to a database, not a pandas dataframe.

test the speed difference between using pandas and a list of lists

KaylaCrush commented 1 year ago

hey past kayla, pandas is doing a lot of work for us and it's not THAT slow come on.

KaylaCrush commented 1 year ago

did this as part of the beautiful soup refactor. Now sucking the same pages in 1/5 the time. bye bye pandas