codingteam / horta-hell

Configurable XMPP bot written in Scala based on the Akka framework.
MIT License
15 stars 8 forks source link

Decrease Markov network memory footprint #413

Open ForNeVeR opened 7 years ago

ForNeVeR commented 7 years ago

I believe that currently Markov network images are consuming more memory that it's needed for them. We need to profile Horta memory and decrease network memory footprint if that's possible.

ForNeVeR commented 7 years ago

Please first of all check #134 when implementing.

ForNeVeR commented 7 years ago

Some optimization points (things to consider):

  1. Intern strings (words) after parsing the file.
  2. Intern phrase vectors inside Platonus library.
  3. Make a single parser actor (so it won't eat too much memory while parsing multiple networks at once).
  4. Progressively load strings from the database (don't load them all at once).