search

piqoni / matcha

Daily Digest Reader
MIT License
454 stars 23 forks source link

Feature request: use go routines for fetching feeds concurrently #41

Closed barakplasma closed 3 weeks ago

barakplasma commented 2 months ago

I half implemented concurrent feed downloads, and it speeds up matcha significantly when there are a lot of feeds.

I implemented it, but I haven't figured out how to close the channel of items. I can open a PR later if you're interested in this contribution.

CalabashSquash commented 1 month ago

I personally would be interested in this, and can help you debug the problem you mentioned :100:

piqoni commented 1 month ago

Out of curiosity, may I know what is the use case, ie, the way I use the tool is to generate markdowns in background and I read the markdown files whenever? Or do you want to use the terminal mode and you are not happy with the sequential load of items?

piqoni commented 1 month ago

Oh and if its the terminal mode, I actually just pushed a new simple project that concentrates on speed (background pre-fetching of article content) https://github.com/piqoni/cast-text But currently its supports only one rss feed, so its more for news outlets, so not exactly matcha but similar in nature.

barakplasma commented 1 month ago

I have a lot of feeds (100s), and haven't yet added matcha to my task scheduler. So I felt the loading time for matcha to download each feed sequentially with blocking io.

This is the relevant code I changed: https://github.com/piqoni/matcha/blob/899b78251aea553c333afaa5d55e0fc69727edb0/main.go#L16-L27

I made a wait group and a go routine for parseFeeds which get sent to a channel. I collect the channel when the waitgroup is done I still writeFeed sequentially

I like the look of the terminal reader too, I'll check it out.