Python.exe has stopped working

[krongch2]

I ran the command

python -m aeneas.tools.execute_task input.mp3 input.txt "task_language=eng|os_task_file_format=srt|is_text_type=plain" output.srt And Python crashed. I was wondering what might possibly cause this problem. Here's my input.mp3 and input.txt

https://app.box.com/s/51ekd2pug9q7ar37xm81wheh8alg7i5h

[readbeyond]

Hi,

thank you for providing input files and configuration.

On my Linux box it runs just fine, I get the desired output.srt.

A few questions to understand the issue:

1. What is your environment?

Version of Windows (10, 8.1, 7, etc.) 32 or 64 bit? Python version (2.7.x or 3.4/3.5/3.6)?

Are you sure you do NOT have e.g. a 32-bit built of Python on a 64-bit Windows?

2. How did you install aeneas? (all-in-one installer, pip, manually via github, etc.)

3. run with the -v flag to see some debug information:

python -m aeneas.tools.execute_task input.mp3 ... output.srt -v

and copy-and-paste the output in a reply here or use the --log:

python -m aeneas.tools.execute_task input.mp3 ... output.srt --log=debug.log

and attach the generated debug.log

=== === ===

Also note that your input text file has blank lines, which will result in blank/empty/zero-length fragments in the SRT output. Did you mean "mplain" instead of "plain"? Or maybe you do not want the blank lines in your input file. Example output, note the empty fragments #4 and #6:

1 00:00:00,000 --> 00:00:03,840 the villagers he had become acquainted with at his destinations.

2 00:00:03,840 --> 00:00:10,160 The maidservant he'd fallen for during a long stay at an inn, waiting for a blizzard to pass.

3 00:00:10,160 --> 00:00:12,240 And on and on.

4 00:00:12,240 --> 00:00:12,240

5 00:00:12,240 --> 00:00:17,280 In short, he longed for company more and more frequently.

6 00:00:17,280 --> 00:00:17,280

7 00:00:17,280 --> 00:00:27,840 Such longing was an occupational hazard for merchants who spent the better part of a year alone in a wagon, but Lawrence had only recently begun to feel it.

=== === ===

HTH,

AP

On 05/23/2017 10:58 AM, Mick Krongchon wrote:

I ran the command

python -m aeneas.tools.execute_task input.mp3 input.txt "task_language=eng|os_task_file_format=srt|is_text_type=plain" output.srt

And Python crashed. I was wondering what might possibly cause this problem. Here's my input.mp3 and input.txt

https://app.box.com/s/51ekd2pug9q7ar37xm81wheh8alg7i5h

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/readbeyond/aeneas/issues/180, or mute the thread https://github.com/notifications/unsubscribe-auth/AFEodjSOO0VqqvvYaSofUpKD1bjDhQVmks5r8p-tgaJpZM4NjYaE.

[krongch2]

I use 64-bit Windows 10 and 32-bit Python 2.7.13. I used the all-in-one installer. I don't know if that would be the issue though because I run that command with several other inputs, and it works just fine on this environment.

Here is the output of the run with -v flag: debug.txt

Also, thank you for pointing out about the blank lines.

[danielbair]

I can confirm python crashing with the source files the OP shared on my Windows 10 x64 virtual machine using the latest all-in-one installer. I've included the last part of the verbose debug output.

[DEBU] ESPEAKTTSWrapper: Language to voice code: 'eng' => 'en' [DEBU] ESPEAKTTSWrapper: Preparing u_text... done [DEBU] ESPEAKTTSWrapper: Preparing c_text... [DEBU] ESPEAKTTSWrapper: Preparing c_text... done [DEBU] ESPEAKTTSWrapper: Calling aeneas.cew directly [DEBU] ESPEAKTTSWrapper: Importing aeneas.cew... [DEBU] ESPEAKTTSWrapper: Importing aeneas.cew... done [DEBU] ESPEAKTTSWrapper: Calling aeneas.cew... [DEBU] ESPEAKTTSWrapper: Calling aeneas.cew... done [DEBU] ESPEAKTTSWrapper: sr: 22050 [DEBU] ESPEAKTTSWrapper: sf: 760 [DEBU] ESPEAKTTSWrapper: Returning 760 time anchors [DEBU] ESPEAKTTSWrapper: Current time 1884.786 [DEBU] ESPEAKTTSWrapper: Synthesized 32110 characters [DEBU] ESPEAKTTSWrapper: Synthesizing using C extension... done [DEBU] Synthesizer: Synthesizing text... done [DEBU] ExecuteTask: STEP 3 END (synthesize text) [DEBU] ExecuteTask: STEP 3 DURATION 19.799 (synthesize text) [DEBU] ExecuteTask: STEP 4 BEGIN (extract MFCC synt wave) [DEBU] AudioFileMFCC: Initializing MFCCs... [DEBU] AudioFile: Loading audio data... [DEBU] AudioFile: self.file_format is None or not good => converting self.file_path [DEBU] AudioFile: Temporary PCM16 mono WAVE file: 'c:\users\ieuser\appdata\local\temp\tmpnqw_hx.wav' [DEBU] AudioFile: Converting audio file to mono... [DEBU] FFMPEGWrapper: Calling with arguments '['ffmpeg', '-i', u'c:\users\ieuser\appdata\local\temp\tmpy23gk1.wav', '-ac', '1', '-ar', '16000', '-y', '-map_metadata', '-1', '-flags', '+bitexact', '-f', 'wav', u'c:\users\ieuser\appdata\local\temp\tmpnqw_hx.wav']' [DEBU] FFMPEGWrapper: Call completed [DEBU] FFMPEGWrapper: Returning output file path 'c:\users\ieuser\appdata\local\temp\tmpnqw_hx.wav' [DEBU] AudioFile: Converting audio file to mono... done [DEBU] AudioFile: Deleted temporary audio file: 'c:\users\ieuser\appdata\local\temp\tmpnqw_hx.wav' [DEBU] AudioFile: Sample length: 1885.297 [DEBU] AudioFile: Sample rate: 16000 [DEBU] AudioFile: Audio format: pcm16 [DEBU] AudioFile: Audio channels: 1 [DEBU] AudioFile: Loading audio data... done [DEBU] AudioFileMFCC: C extension 'cmfcc' enabled [DEBU] AudioFileMFCC: C extension 'cmfcc' enabled and it can be loaded [DEBU] AudioFileMFCC: Computing MFCCs using C extension... [DEBU] AudioFileMFCC: Importing cmfcc... [DEBU] AudioFileMFCC: Importing cmfcc... done [DEBU] AudioFileMFCC: Computing MFCCs using C extension... done [DEBU] AudioFileMFCC: Clearing the audio data... [DEBU] AudioFile: Clear audio_data [DEBU] AudioFileMFCC: Clearing the audio data... done [DEBU] AudioFileMFCC: Initializing MFCCs... done [DEBU] ExecuteTask: STEP 4 END (extract MFCC synt wave) [DEBU] ExecuteTask: STEP 4 DURATION 2.969 (extract MFCC synt wave) [DEBU] ExecuteTask: STEP 5 BEGIN (align waves) [DEBU] ExecuteTask: Creating DTWAligner... [DEBU] ExecuteTask: Creating DTWAligner... done [DEBU] ExecuteTask: Computing boundary indices... [DEBU] DTWAligner: Requested algorithm: 'stripe' [DEBU] DTWAligner: delta = 3000 [DEBU] DTWAligner: m = 47132 [DEBU] DTWAligner: Using unmasked MFCC [DEBU] DTWAligner: Number of MFCC frames in real wave: 60001 [DEBU] DTWAligner: Number of MFCC frames in synt wave: 47132 [DEBU] DTWAligner: Computing with STRIPE algo [DEBU] DTWAligner: Computing path... [DEBU] DTWAligner: Computing path... [DEBU] DTWStripe: C extension 'cdtw' enabled [DEBU] DTWStripe: C extension 'cdtw' enabled and it can be loaded [DEBU] DTWStripe: Computing path using C extension... [DEBU] DTWStripe: Importing cdtw... [DEBU] DTWStripe: Importing cdtw... done [DEBU] DTWStripe: n m delta: 60001 47132 3000

[readbeyond]

Dear Mick, Daniel,

thank you for the information.

I strongly suspect that this problem is related to the other issue:

https://github.com/readbeyond/aeneas/issues/172

On 32-bit builds there is a theoretical limit of 2 GB per single contiguous memory allocation. For these particular input files, the DTW matrix allocated by the aeneas cdtw extension requires:

n delta 8 bytes/element = 60001 3000 8 = 1.44 GB

Since there are other objects in RAM, I suspect that the abrupt termination is due to the impossibility for the OS to satisfy the request of the cdtw extension to get the ~1.44 GB of CONTIGUOUS memory.

This hypothesis is corroborated by the fact that on shorter audio file --- if I understood correctly --- Mick said that aeneas was working correctly. A quick search seems to confirm the issue is being experienced by other users as well: https://stackoverflow.com/questions/4441947/why-python-memory-error-with-list-append-lots-of-ram-left

=== === ===

Mick, while Daniel and I investigate the problem, you have the following options:

1. exclude the cdtw extension with -r="cdtw=False", but aeneas will be very slow
2. install Python 64-bit, then install numpy and aeneas via pip:

pip install numpy pip install aeneas

(but right now I am not sure if the C extensions in aeneas are compiled correctly)

3. use the aeneas-vagrant image: https://github.com/readbeyond/aeneas-vagrant

4. if you have audio files with length < 1h30m, you can freely use the aeneas Web app: http://aeneasweb.org/

HTH,

AP

[danielbair]

Installing Python x64 will mean that espeak cew will not be available as I haven't found a way to compile espeak on x64. The rest of the aeneas application might work better but there will be a performance loss as aeneas cew will be disabled.

I wonder if festival or other TTS engines are available for Win x64 machines.

[krongch2]

Hi ReadBeyond and Daniel,

Thanks so much for looking at the issue. I have been using the Web app in the meantime, and I wish I had known about it earlier.

[readbeyond]

Sure, you are welcome. Unfortunately I am not a Windows user, so I am not sure everything is smooth there, plus Python C extensions have always been difficult to work with on Windows.

In general, I would say that having the cdtw extension is more important than having the cew extension:

run_time(aeneas + cmfcc + cdtw - cew) << run_time(aeneas + cmfcc - cdtw

• cew)

Also, from a user perspective, a segfault is not really nice. I can mitigate this by catching a MemoryException in Python-land. Still, in the current configuration (64-bit OS + 32-bit Python), if the user feeds a "too long" audio, she will get the segfault from the cdtw extension, and the only way to circumvent it consists in disabling the cdtw via -r="cdtw=False", which results in extremely slow overall runtime.

Daniel, I could not look into this issue myself for at least the next 2 weeks, as I am moving. As always, I am open to suggestions about how to handle this issue.

I also feel there is no need for immediate action, as workarounds are available (vagrant VM, aeneasweb.org), so no need to rush either. If other users experience the problem, for the moment we can direct them to this GH issue. If their number keeps growing, I will add a note in the documentation.

Thank you for your time,

AP

[danielbair]

I have overloaded the vagrant before. I was just playing around with it and gave it the whole KJV (all 66 books) in a zip job. So some information in the documentation on suggested max file sizes might be good for 32 and 64 bit. It might also be nice if aeneas could warn the user that it will probably exhaust the system memory with the submitted task or job, and maybe even ask if they want to continue.

[pettarin]

Hi @danielbair , ages ago I looked into the possibility of finding out the amount of free RAM on the system. It turns out that it is not that simple, if you want to support Windows (macOS and linux are different, but relatively "easy"). Plus, adding this check adds dependencies.

I agree that a minimum can be be adding in the documentation some rule-of-thumb relationship between audio length of a task and quantity of free RAM required by aeneas. I can do this some time soon.

With regard to the suggestion of warning the user: given a Task (i.e., after knowing the length of the audio file), we can print out an estimate of the RAM required, but I am not seeing favorably the "user confirmation" thing: to work, it should be the default mode (with an optional "--i-asked-dad-please-go-ahead" switch to avoid it), it adds complexity in the CLI tools, and it makes the CLI tools interactive instead of non-interactive, which I do not see as an improvement. Conversely, a "--check-ram-with-user" (or something similar) switch, to turn on the check (disabled by default), might be considered for v2.

[pettarin]

@danielbair a question: in theory, when processing a Job, each Task is executed sequentially, and its data discarded before computing the next one. Did your KJV (King James V Bible?) Job failed because one of the 66 books had a very long audio file? How long was the longest one? (Otherwise, there might be a forgotten "flush" somewhere...)

[shaund]

I'm still having this happen with a 27.5MB file, but not with a file that's the first 4MB of the same file. I see the issue is closed, has it been fixed somewhere? I installed everything with the installer (1.7.3) from https://github.com/sillsdev/aeneas-installer/releases

[readbeyond]

The issue has been closed because, without further information, I assumed we determined there was a mismatch/issue with 32 vs 64 bit on Windows OR the fact that the file was too large for the amount of RAM available on that particular machine.

In both cases, there is little to nothing I can do about it --- maybe except try to output a descriptive error, rather than crashing.

Assuming "27.5MB" is the size of the (MP3?) audio file, it should be processable by any modern machine with 4 GB of RAM (maybe 2 GB suffice already) --- if not, there is definitely an issue on windows, which might or might not be related to issue #190.

Unfortunately now I maxed out my time dedicated to open source for this week, and the next weekend I will have some personal appointments, which means I will try to look into sort of issues the following weekend (mid May).

Meanwhile, you can use https://aeneasweb.org and, if you really really want this issue solved, it would be helpful if you can send to aeneas@readbeyond.it a Dropbox/GoogleDoc/Drive link from which I can get the audio and text files you are trying to process that cause the crash, along with the exact command line and which version of Windows you are on.

[swarup-panda]

Hi Guys, Please help I got an issue while installing any package on Anaconda Environment need your help. Have a look at the attached screenshot. Uninstalling and reinstalling didn't seems to work for me kindly provide me a solution for this.

[shaund]

@readbeyond sent an email

[readbeyond]

I will not reply to emails privately. That is called "consulting" and comes with a compensation in exchange for dedicated time and confidentiality.

Public venues, like GitHub Issues or the aeneas mailing lists are better, because other people might benefit from your question and my answer.

Back to your issue, please note that Windows is not officially supported, as I do not own a personal Windows machine.

This weekend is busy for me, I will try to have a look at the problem next week(end).

Are there any clues if you specify the verbose (-v) switch in your command line?

On 05/11/2018 10:51 PM, shaund wrote:

@readbeyond https://github.com/readbeyond sent an email

[readbeyond]

@shaund Disregard my last message, I mixed you with "swarup-panda" .

Now I see you meant to send me the audio and text files, thank you for those, they will help investigating the problem.

On 05/11/2018 10:51 PM, shaund wrote:

@readbeyond https://github.com/readbeyond sent an email

[readbeyond]

@swarup-panda As stated above, Windows is not officially supported and there is no solution (yet).

If you need a workaround, you can either use the aeneas Web application, or try using the aeneas-vagrant approach, i.e. installing a Linux VM inside VirtualBox using the provided Vagrant config file.

Hopefully next week(end) I will have time to look into the issue.

On 05/11/2018 10:51 PM, shaund wrote:

@readbeyond https://github.com/readbeyond sent an email