search

morevnaproject-org / papagayo-ng

Papagayo is a lip-syncing program designed to help you line up phonemes (mouth shapes) with the actual recorded sound of actors speaking. Papagayo makes it easy to lip sync animated characters by making the process very simple - just type in the words being spoken (or copy/paste them from the animation's script), then drag the words on top of the sound's waveform until they line up with the proper sounds.
233 stars 51 forks source link

Script for generating Linux AppImage package #99

Closed morevnaproject closed 3 years ago

morevnaproject commented 3 years ago

This is a script to generate Linux AppImage package. Just execute build/linux-appimage/build.sh script and it will generate package for you.

Here is a sample package generated by this script - Papagayo_NG-d91a1b7-x86_64.AppImage

At the moment the package doesn't includes Allosaurus, as it makes very large package - 1,2 Gb (comparing to ~300 Mb without Allosaurus). You can enable Allosaurus by replacing following line

./squashfs-root/AppRun -m pip install $(grep -ivE "allosaurus" "${SOURCES_DIR}requirements.txt")

to this one

./squashfs-root/AppRun -m pip install -r "${SOURCES_DIR}requirements.txt"

Have fun! ^__^

steveway commented 3 years ago

That's really nice! I'm currently on vacation. I'll try to test this as soon as possible. I'll be home during the weekend so I might look into that if I find the time, otherwise sometime the week after that or so. Can you distribute binaries frozen with pyinstaller using appimage? Even with allosaurus the windows binaries are "only" around 300MB, so I would think that the result for Linux should be similar.

morevnaproject commented 3 years ago

Can you distribute binaries frozen with pyinstaller using appimage?

I haven't worked with pyinstaller, but I quickly looking at its docs I've got an impresson that it is kind of replacing concept for AppImage. It would be nice to investigate it as alternative method of distribution for Linux, but I have not time for that right now, unfortunately.

Other possibility is to look for redundant files in the build and cleanup them before packing into AppImage, as it is done here - https://github.com/AppImage/AppImageKit/wiki/Bundling-Python-apps (see code after "# Remove unneeded parts." line)

morevnaproject commented 3 years ago

Quick investigation gives me that the package size is so big because of two files:

opt/python3.7/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so (1,2 Gb)
opt/python3.7/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so (372 Mb)

Discussion on this topic - https://discuss.pytorch.org/t/libtorch-cuda-so-is-too-large-2gb/103155/25

morevnaproject commented 3 years ago

I have forced installation of Torch package without CUDA and enabled Allosaurus. This results in package at 538 Mb which is much better.

I believe there are more a room for optimization by removing redundant files from PySide and maybe other things like torch/tests. screenshot_001

steveway commented 3 years ago

Hi, about the size. I just tested building this on a Ubuntu VM and the script seems to work nicely. One of the bigger files inside is libQT5WebEngineCore.so.5 which I believe we don't need if I'm not mistaken. I'm not exactly sure how the Appimage file gets created exactly with the script and where we can manually remove files using the script inside it. Oh, I think I see. Yes that works, I pushed a small update which removes that, this shrinks the Appimage a little bit. I also fixed a warning from QT. https://github.com/steveway/papagayo-ng/commit/0825db3c8059fc607b1c8e3714a0d61ecf24bd49 Why is it set to Python 3.7? I saw that there are also prebuild images up to 3.10, for Windows I'm testing this with 3.9 should we use that too for Linux?

morevnaproject commented 3 years ago

Why is it set to Python 3.7? I saw that there are also prebuild images up to 3.10, for Windows I'm testing this with 3.9 should we use that too for Linux?

I remember my first attempt was to make it with Python 3.10 and the resulting AppImage was throwing an error (probably issued by PySide, but not remember exactly). Then I have googled for that error and found some page which was referencing this error as coming starting from Python 3.8. So I have downgraded to Python 3.7 and it worked. Sorry I don't remember exact details. In any case it is simple to change for other Python version and test. ^__^

steveway commented 3 years ago

Ok, I tested this with the 3.9 manylinux2014 package and that seems to work correctly too. I am now removing the tests from Torch too to save some space. And I created a small script which copies the needed project files for Papagayo-NG to a separate folder. https://github.com/steveway/papagayo-ng/commit/4efe61e1f36c88189192bf62f378e4df6a5207a7 With these changes I was able to get the size of the AppImage down to 352.1MiB

morevnaproject commented 3 years ago

Awesome! I suggest to use this script to build version 1.4.2 - https://github.com/morevnaproject-org/papagayo-ng/releases/tag/v1.4.2 I can look at OSX build then.