search

StephanAkkerman / FluentAI

Automating language learning with the power of Artificial Intelligence. This repository presents FluentAI, a tool that combines Fluent Forever techniques with AI-driven automation. It streamlines the process of creating Anki flashcards, making language acquisition faster and more efficient.
https://akkerman.ai/FluentAI/
MIT License
9 stars 1 forks source link

MemoryError when Loading FastText Embeddings for ImageabilityPredictor Due to Large Model Size #43

Closed WinstonLam closed 3 weeks ago

WinstonLam commented 4 weeks ago

Describe the bug A MemoryError occurs when loading the FastText embeddings for the ImageabilityPredictor in imageability.py, due to the large size of the embedding model (cc.en.300.bin.gz). This suggests that the current system does not have enough memory to load the full model, limiting functionality for users with lower system resources.

To Reproduce Steps to reproduce the behavior:

  1. Load the FastText model (cc.en.300.bin.gz) in imageability.py.
  2. Attempt to initialize ImageabilityPredictor.
  3. Observe the MemoryError during the loading process. *this is tested on a setup with 16gb RAM

Expected behavior The FastText model should load without causing a memory error, allowing the ImageabilityPredictor to function properly even on systems with lower memory.

StephanAkkerman commented 4 weeks ago

We need 1 file that is responsible for all the FastText model stuff, now the model gets loaded for both semantic similarity and imageability leading to longer wait times and increased memory.

StephanAkkerman commented 4 weeks ago

@WinstonLam please test again, my max memory with word2mnemonic was around 9GB for python