run-llama / sec-insights

A real world full-stack application using LlamaIndex
https://www.secinsights.ai/
MIT License
2.32k stars 631 forks source link

frontend hotfix for duplicated sec docs #70

Closed sourabhdesai closed 10 months ago

sourabhdesai commented 10 months ago

Ran the cron job earlier today to pull some of the latest SEC filings. Seems like the newest version of the sec-edgar-downloader has a different metadata structure for the filings it downloads, which means the paths the filings get stored into S3 are different than before. Hence leading to some amount of duplication in terms of the dropdown form on the landing page as seen below:

Screenshot 2023-11-03 at 7 32 50 PM

This simple hotfix just does some de-duplication on the frontend to ensure we're only showing one filing from each filing period. It also does some extra filtering to ensure that we ignore any documents that aren't SEC filings. This would be useful for local development if you're trying to use the site with some custom documents already stored in the DB.

vercel[bot] commented 10 months ago

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
llama-app-frontend ✅ Ready (Inspect) Visit Preview 💬 Add feedback Nov 4, 2023 2:51am