Upload wheel to PyPI

[readbeyond]

lxml, bs, numpy all have wheels on PyPI now.

Note that some of the extra deps (pafy, tgt) do not have wheels, the other (boto3, requests, youtube-dl, Pillow) have them.

bdist_wheel is easy and works right now. Since I compile it on my Debian testing laptop, my worry is that the wheel only works on extremely updated linux boxes. If installing via the wheel fails (or, worse, the installation succeeds but it is broken --- e.g., because there is an issue with the .so linked by the C extensions), what happens? Is the source tar.gz fetched instead and the installation restart from it?

The suggested "workaround" consists in creating a manylinux1 wheel, but it requires a complex setup, and I still need to figure out how to run it locally without Travis. See https://github.com/pypa/manylinux and https://github.com/pypa/python-manylinux-demo

[mgaitan]

Hello, I would like to help with this. I'm using aeneas in https://github.com/mgaitan/miau and it's marvelous

What's the problem on setup travis in the repo to build the manylinux based package? why you need to build it locally?

btw, an analog setup could be done for windows using appveyor service.

[readbeyond]

Hi Martín,

thanks for this suggestion as well.

My comment about building locally should be regarded as a "desideratum", rather than a hard requirement: I always prefer having a "plan B" while depending from a third-party, in case the latter becomes unavailable.

Honestly, I just have not had time to look into this issue much to comment competently --- but if you are willing to put together the travis (and appveyor?) build configuration file(s), I will be more than happy to merge them in.

One difficulty I see --- but again, I might be wrong on this --- is that we need eSpeak installed in the build environment, since the cew C extension links against the eSpeak C API header, and the wheel must ship that too. (Ideally, the same holds for Festival, and its C++ API header.)

Thanks,

Alberto Pettarin

On 05/08/2017 06:01 AM, Martín Gaitán wrote:

Hello, I would like to help with this. I'm using aeneas in https://github.com/mgaitan/miau and it's marvelous

What's the problem on setup travis in the repo to build the manylinux based package? why you need to build it locally?

btw, an analog setup could be done for windows using appveyor service.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/readbeyond/aeneas/issues/157#issuecomment-299767895, or mute the thread https://github.com/notifications/unsubscribe-auth/AFEodp6ja1fCvGaXS2Vo64XhqTLbjWBhks5r3pOrgaJpZM4L3cg6.

[pettarin]

I did a quick test downloading the Docker CentOS images that are used by the manylinux project, and indeed getting eSpeak to install there seems... complicated:

• no yum package on CentOS 5
• using eSpeak RPMs for CentOS does not work
• trying to compile from source raises a lot of errors

I also tried to use the new eSpeak-ng project, since its C API is compatible with the old eSpeak C API, and it does not compile either.

(When I say "it does not compile" I mean that it doesn't out-of-the-box or by installing dependencies via yum. I am sure that, investing some time, one can at least patch the source code of eSpeak/eSpeak-ng to make it compile on CentOS 5.)

So, it seems that this route is a bit of a time sink. Maybe using a slightly less ancient Linux distribution might help, but then we need to find out which one to use and/or if anyone else already investigated a similar approach. For now, I am timed out with this issue.

[mgaitan]

I've been experiment on this yesterday and seems it's possible to compile the wheel using precompiled espeask rpm packages.

Check https://travis-ci.org/mgaitan/aeneas/jobs/232012747

As a test, I've just installed each generated wheel and import it (because I assume ffmpeg isn't mandatory to build the package but it is to run tesst, right? I need to figure out how to install ffmpeg)

Also this works for x64 architecture, for 32 bits there is a little problem with lxml due there is no official wheel and trying to compile from source requires newers libraries thant available oin centros 5.

anyway, I guess it's a valid proof of concept that it is a good way to go ahead. What do you think?

[readbeyond]

2017-05-19: CORRECTION: the last official version of eSpeak is 1.48.04, not 1.48.03 as I wrote originally.

Hi,

thank you for looking into it. Today will be busy for me, I plan to have a look at it tomorrow (Tuesday 16).

For now just two comments:

1. the last eSpeak available is 1.48.03 (after a hiatus, the project was forked and become eSpeak-ng) --- if there are no RPMs for it (I see your current script pulls eSpeak 1.45, and I quick search found no RPMs for 1.48.03 for CentOS 5), I would build from source, if possible.

2. I am fine not compiling a wheel for 32 bit: they can still install from the source tar.gz.

Thanks,

AP

On 05/14/2017 07:36 PM, Martín Gaitán wrote:

I've been experiment on this yesterday and seems it's possible to compile the wheel using precompiled espeask rpm packages.

Check https://travis-ci.org/mgaitan/aeneas/jobs/232012747

As a test, I've just installed each generated wheel and import it (because I assume ffmpeg isn't mandatory to build the package but it is to run tesst, right? I need to figure out how to install ffmpeg)

Also this works for x64 architecture, for 32 bits there is a little problem with lxml due there is no official wheel and trying to compile from source requires newers libraries thant available oin centros 5.

anyway, I guess it's a valid proof of concept that it is a good way to go ahead. What do you think?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/readbeyond/aeneas/issues/157#issuecomment-301327563, or mute the thread https://github.com/notifications/unsubscribe-auth/AFEodiSiyOptQ2CvJ4U620MZf0YrYhpeks5r5zuRgaJpZM4L3cg6.

[readbeyond]

@mgaitan I did not forget about this issue, I am just a bit busy since I am moving at the end of the month.

Since no RPM for CentOS 5 of eSpeak 1.48.04 seems to be available, and I am a bit wary of using a previous version to compile the Python wheel, I have explored the possibility of compiling portaudio 1.19 and eSpeak 1.48.04 from source. make && make install portaudio works fine inside the Docker container. make eSpeak completes, but make install --- while apparently completing without errors --- does not actually install the library and executable system-wise (still inside the Docker container).

I think that, once I figure this out, we should be all set to proceed generating a wheel for aeneas.

Note: I am installing portaudio from source because if I yum install portaudio, then eSpeak does not compile.

[readbeyond]

@mgaitan OK, I was able to successfully compile both portaudio and eSpeak 1.48.04 from source in the manylinux CentOS 5 Docker VM, and from there creating the Python wheels. I confirmed that they work, by copying the .whl files (aeneas, beautifulsoup4, lxml, numpy) to another Debian machine, and installing them in a virtualenv.

Also, installing ffmpeg is not required to compile the wheels, but it is needed in the target environment to run aeneas once installed.

[readbeyond]

I forgot: I will add a script for creating the wheels soon.

[readbeyond]

@mgaitan I hit an unexpected and serious problem.

I took your script and modified it so that it builds portaudio and eSpeak from source. Then it creates the wheels. Apparently, it works fine, the .whl files are created and they can be installed.

In particular, I can copy them and pip install them in a virtualenv on a different Debian machine, and the aeneas package is installed correctly there.

However, the cew extension does not work:

$ python -m aeneas.diagnostics
[INFO] ffprobe        OK
[INFO] ffmpeg         OK
Can't read data file: '/usr/share/espeak-data/phontab'
Failed to load espeak-data

This is because eSpeak 1.48.04 hardcodes /usr/share/espeak-data as the directory where the phone data should exist, while e.g. on Debian those files are put at /usr/lib/x86_64-linux-gnu/espeak-data if the user installs eSpeak via apt-get. Unfortunately, eSpeak 1.48.04 does not have an autoconf script --- so, when the cew extension is statically linked, that path gets hardcoded in the wheel too.

Since in general Linux users will install eSpeak via their package managers, and it is likely that the eSpeak data directory will NOT end be /usr/share/espeak-data, these wheels are not good. Of course, if I install the wheel in a virtualenv inside the CentOS image, it works as expected, because there the eSpeak data directory IS /usr/share/espeak-data.

(I also tried using eSpeak-ng 1.49.1, which has autoconf config files, but the autogen.sh fails --- even if I yum install libtool first.)

More worryingly, I am not sure if this problem can be fixed at all (without embedding the entire eSpeak code inside aeneas), unless there is a way to "fix" the path in the compiled artifact using the manylinux tools.

Note that if I disable the cew extension, everything else works --- but of course the TTS step is slower:

$ # inside the virtualenv where I installed aeneas using aeneas*.whl
$ python -m aeneas.tools.execute_task --example-srt -r="cew=False"
... WORKS FINE ...

but shipping the wheel without the cew extension is not an option. I prefer users getting the src tarball and compiling on their machines, rather than shipping a wheel without cew.

I need to give up for today.

[readbeyond]

Update: I managed to compile eSpeak-ng (which is API back-compatible with eSpeak) inside the CentOS image, and from there to create the wheels. However, the problem with the path is still present. For example, the default configuration installs the eSpeak(-ng) data files into /usr/local/share/espeak-ng-data/, while e.g. Debian package place them into /usr/lib/x86_64-linux-gnu/espeak-data --- so that:

$ python -m aeneas.diagnostics
[INFO] ffprobe        OK
[INFO] ffmpeg         OK
Error processing file '/usr/local/share/espeak-ng-data/phontab': No such file or directory.

However, if I create a symlink, it works:

$ ln -s /usr/lib/x86_64-linux-gnu/espeak-ng-data /usr/local/share/espeak-ng-data
$ python -m aeneas.diagnostics
[INFO] ffprobe        OK
[INFO] ffmpeg         OK
[INFO] espeak         OK
[INFO] aeneas.tools   OK
[INFO] shell encoding OK
[INFO] aeneas.cdtw    AVAILABLE
[INFO] aeneas.cmfcc   AVAILABLE
[INFO] aeneas.cew     AVAILABLE
[INFO] All required dependencies are met and all available Python C extensions are working

[mgaitan]

that's truly amazing!

On Sat, May 20, 2017 at 6:36 PM, ReadBeyond notifications@github.com wrote:

Update: I managed to compile eSpeak-ng (which is API back-compatible with eSpeak) inside the CentOS image, and from there to create the wheels. However, the problem with the path is still present. For example, the default configuration installs the eSpeak(-ng) data files into /usr/local/share/espeak-ng-data/, while e.g. Debian package place them into /usr/lib/x86_64-linux-gnu/espeak-data --- so that:

$ python -m aeneas.diagnostics [INFO] ffprobe OK [INFO] ffmpeg OK Error processing file '/usr/local/share/espeak-ng-data/phontab': No such file or directory.

However, if I create a symlink, it works:

$ ln -s /usr/lib/x86_64-linux-gnu/espeak-ng-data /usr/local/share/espeak-ng-data $ python -m aeneas.diagnostics [INFO] ffprobe OK [INFO] ffmpeg OK [INFO] espeak OK [INFO] aeneas.tools OK [INFO] shell encoding OK [INFO] aeneas.cdtw AVAILABLE [INFO] aeneas.cmfcc AVAILABLE [INFO] aeneas.cew AVAILABLE [INFO] All required dependencies are met and all available Python C extensions are working

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/readbeyond/aeneas/issues/157#issuecomment-302900183, or mute the thread https://github.com/notifications/unsubscribe-auth/ACPyB4Jetk-_QlcXw0lX2ugQVM1J5cn5ks5r71zygaJpZM4L3cg6 .

-- mgaitan.github.io textosypretextos.com.ar

[pettarin]

It works, but it is far from ideal. I am not really favorable about distributing wheels that then require the user to create a symbolic link somewhere, especially since /usr/... usually requires root privileges.

Right now I believe that the eSpeak data that gets "hardcoded" in the wheel is a no-go. In fact, at pip-install time, on different linux distros eSpeak (and the eSpeak data directory) is installed at different paths. Potentially, it can be installed in user space (e.g. /home/username/espeak )... how does pip know where to find the "correct" path of the eSpeak data directory, if the wheel is pre-compiled?

Also note that the same problem applies even if there exists a way to "modify" the path in the wheel. (I guess this is what auditwheel does.)

Another solution would be creating a wheel for each linux distribution... but that is a maintenance nightmare.

So, due to the lack of viable ways to have wheels that work on any linux, I am OK with people downloading the source and compiling the C extensions at install time. If a user really needs a wheel for their system, they can get the source on a devel machine, create the wheel there, and deploy it as necessary.

[pettarin]

At this point, I am more interested in fixing the numpy requirement and/or having aeneas on conda-forge (which seems becoming more popular as python "distribution").

[readbeyond]

I am closing this for now.

[tslater]

@readbeyond Just reading through this old thread. If I am using AWS polly, would I need espeak? Is there a way to disable the dep of speak for a simpler build?