RuntimeError: [enforce fail at CPUAllocator.cpp:65]

loretoparisi / wave2vec-recognize-docker

Wave2vec 2.0 Recognize pipeline

MIT License

33 stars 10 forks source link

RuntimeError: [enforce fail at CPUAllocator.cpp:65] #3

Open raja1196 opened 3 years ago

raja1196 commented 3 years ago

Followed the installation steps, built the Dockerfile (Which had its own hiccups, the fairseq repository does not have base_architecture definition in their models file, will raise a PR for it separately) and ran the code.

Running command python examples/wav2vec/recognize.py --wav_path /app/data/test.WAV --w2v_path /app/data/wav2vec_small_10m.pt --target_dict_path /app/data/dict.ltr.txt

Error: RuntimeError: [enforce fail at CPUAllocator.cpp:65] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 314663671488 bytes. Error code 12 (Cannot allocate memory)

Screenshot of the error:

I tried with different models of wave2vec with similar error. Do let me know if more information is needed. Running this on Azure DS VM.

loretoparisi commented 3 years ago

PR for base_architecture definition merged thanks.

raja1196 commented 3 years ago

I did a couple of things.

Changed the memory limit for the docker to 8GB
used a different audio file

I got the result: WE THE PEOPLE OF THE UNITED STATES IN ORDER TO FORM A MORE PERFECT UNION ESTABLISH JUSTICE INSURE DOMESTIC TRANQUILLITY PROVIDE FOR THE COMMON DEFENCE

I suspect that the length of the audio file could be a reason it is throwing the error (do let me know if it it is a more prevalent problem). The audio file i was trying to transcribe was around 1hr, compared to the 10 sec clip linked above.

raja1196 commented 3 years ago

https://github.com/pytorch/fairseq/issues/2651#issuecomment-719894563

I followed this comment for reference.

loretoparisi commented 3 years ago

@raja1196 thanks we will check this case.

iman3sh commented 3 years ago

I'm trying to run XLRS-53 after fine-tuning with this code but the following error happens. can anyone help me?

/usr/local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 Traceback (most recent call last): File "src/recognize.py", line 170, in model = Wav2VecPredictor(args.w2v_path, args.target_dict_path) File "src/recognize.py", line 122, in init self._model = self._load_model(w2v_path, self._target_dict) File "src/recognize.py", line 145, in _load_model args = base_architecture(w2v["args"]) File "src/recognize.py", line 24, in base_architecture args.no_pretrained_weights = getattr(args, "no_pretrained_weights", False)