search

billray0259 / stockbot_2020

A previous attempt at profitable algorithmic trading
0 stars 0 forks source link

When scraping finviz the last line in their table often gets duplicated in our dataframe #2

Closed billray0259 closed 4 years ago

billray0259 commented 4 years ago

When the scraper gets to the end of the data and attempts to load a Finviz page that is higher than what Finviz actually has, Finviz returns a page with just the last row. The scraper recognizes when it presented with a single row page, records it, and stops there. I didn't want to make the scraper not record this single row page because it's possible the last page actually does only have one row and recording one duplicate doesn't matter much. I think I can also fix it.

billray0259 commented 4 years ago

Fixed by adding this code at the bottom of the save_finviz method

if (df.values[-1] == df.values[-2]).all():
            df = df[:-1]