[Optional] Web Scraper bug?

[michac789]

When using web scraper, something strange happen... Now, we can set some limit on how many indexes we want to get. If we say 10, then we get 10 index. But for some reason, at large number, there is a discrepancy something like: we say to get 5000, but we only get 4993 (I forgot the exact number, in summary there is potentially something wrong with the web scraper; it might actually skip some index?)

[michac789]

made this optional -- to be investigated later (not so important now)

[michac789]

additional: information is duplicated in the database temporary solution is to update it via python shell (in production db, DONE), but need a way to fix this later