facebookresearch / DrQA

Reading Wikipedia to Answer Open-Domain Questions
Other
4.48k stars 898 forks source link

How much RAM is needed to run the interactive script? #266

Open Martin36 opened 3 years ago

Martin36 commented 3 years ago

When running python scripts/pipeline/interactive.py, I get the following error:

Traceback (most recent call last):
  File "scripts/pipeline/interactive.py", line 70, in <module>
    tokenizer=args.tokenizer
  File "/mnt/d/Documents/NLP/DrQA/drqa/pipeline/drqa.py", line 146, in __init__
    initargs=(tok_class, tok_opts, db_class, db_opts, fixed_candidates)
  File "/home/martin/miniconda3/lib/python3.7/multiprocessing/context.py", line 119, in Pool
    context=self.get_context())
  File "/home/martin/miniconda3/lib/python3.7/multiprocessing/pool.py", line 176, in __init__
    self._repopulate_pool()
  File "/home/martin/miniconda3/lib/python3.7/multiprocessing/pool.py", line 241, in _repopulate_pool
    w.start()
  File "/home/martin/miniconda3/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/home/martin/miniconda3/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/home/martin/miniconda3/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/home/martin/miniconda3/lib/python3.7/multiprocessing/popen_fork.py", line 70, in _launch
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

I am running the script in Ubuntu for Windows 10, with a 16 GB RAM. When looking at one of the files that are imported docs-tfidf-ngram=2-hash=16777216-tokenizer=simple.npz I noticed that it was 12.2 GB in size. I figured that this would probably explain the error.

So my question is then, how much memory is needed to run the interactive script?

Thanks in advance 😄