leftmove / pinestreetlocal

Free and open-source stock tracking website for America's biggest money managers. Democratizing SEC filings by making them more accessible and useful.
https://content.wallstreetlocal.com
MIT License
32 stars 6 forks source link

Fix time estimation #3

Open leftmove opened 6 months ago

leftmove commented 6 months ago

Time estimation has been broken for a while.

It is either too slow or not functional whatsoever. Additonally, the "algorithmn" is a simple divide by two function that gives an incredibly rough "estimation." Since wallstretlocal has gone public, there are now enough statistics in the database to create a reasonable average time required using log counts.

chrisgarcia4158 commented 6 months ago

Potential average aggregation query.

db.statistics.aggregate(
   [
     {
       $group:
         {
           _id: "$type",
           avg: { $avg: { $divide: [ { $size: "$log.logs" }, "$log.time.elapsed" ] } }
         }
     }
   ]
)
chrisgarcia4158 commented 6 months ago

Results for query.

{ "_id" : "create-historical", "avg" : 14.349005705254921 }
{ "_id" : "create-latest", "avg" : 17.308616566972628 }

Historical queries do about 14 per second and latest queries do about 17 per second. In order to create an accurate estimation, take stock count, multiply it by two, and add four. I've added some more accurate code to the add_query_log function, but until that collects enough statistics this will have to do.