search

BeMoreHumanOrg / bemorehuman

The recommendation engine with particular focus on uniqueness of the person receiving the rec.
https://bemorehuman.org
MIT License
49 stars 0 forks source link

Make system automatically update valences #1

Closed unbrand closed 1 year ago

unbrand commented 2 years ago

Currently valences need to be generated manually. It would be great if the valences would be generated by a live bemorehuman system as needed.

unbrand commented 2 years ago

(comment copied from issue #4 which is a duplicate) Could try using malloc_trim(0) after beast export. If that doesn’t work, use a different malloc than glibc’s? The issue is that glibc's free doen't really free you might expect. So the beast export currently needs to be in a separate process.

unbrand commented 2 years ago

Yeah, considering how heavy the beast (valence cache) generation is in terms of cpu, probably better to not have recgen do it live. This means the beast gen can be done parallel to the live recgen then when ready, recgen can be bounced.

Wait a second -- that's older thinking where I needed every scrap of RAM. We should be able to rely on multi-threaded nature to keep serving while simultaneously building new cache. Plus, if we're writing to a file, then recgen has the handle open so... better to control from within recgen. Try that and malloc_trim(0).

unbrand commented 2 years ago

Hm. Recgen does have the handle open, but only for a short amount of time. Just long enough to dump the cache of new events. So, we have more flexibility with how to solve this. Current thinking is more closely aligned with "small pieces loosely joined" by using a long-running script.