AppImage is missing a library

probonopd commented 4 years ago

On Ubuntu 18.04: python3: error while loading shared libraries: libpython3.8.so.1.0: cannot open shared object file: No such file or directory. It needs to be bundled.

Reference: https://github.com/AppImage/appimage.github.io/pull/1903

probonopd commented 4 years ago

Maybe this would be a good candidate for https://github.com/niess/python-appimage#for-applications-developers @niess?

niess commented 4 years ago

The user install with a Python AppImage seems to work:

wget https://github.com/niess/python-appimage/releases/download/python3.8/python3.8.2-cp38-cp38-manylinux1_x86_64.AppImage
chmod u+x  python3.8.2-cp38-cp38-manylinux1_x86_64.AppImage
./python3.8.2-cp38-cp38-manylinux1_x86_64.AppImage -m pip install --upgrade guiscrcpy --user
guiscrcpy

It starts the GUI and complains that adb and scrcpy are not found (I didn't install them indeed). Note however that this is not an isolated install. For example, the exe are installed in .local/bin whatever the Python version which can lead to troubles when using several versions of Python.

Installing to the extracted image seems to work at first. E.g as:

./python3.8.2-cp38-cp38-manylinux1_x86_64.AppImage --appimage-extract
mv squashfs-root python3.8
./python3.8/AppRun -m pip install guiscrcpy

However when running the guiscrcpy script it does not start the GUI as previously. Instead I get the following error related to git:

./python3.8/usr/bin/guiscrcpy
ERROR:root:adb could not be found in any of the paths ['bin', '/usr/bin', '~/.local/bin', '~/bin', '/usr/local/bin']
ERROR:root:scrcpy could not be found in any of the paths ['bin', '/usr/bin', '~/.local/bin', '~/bin', '/usr/local/bin']
Traceback (most recent call last):
  File "./python3.8/usr/bin/guiscrcpy", line 6, in <module>
    from guiscrcpy.launcher import bootstrap
  File "guiscrcpy/launcher.py", line 78, in <module>
    v = version()
  File "guiscrcpy/lib/ver.py", line 35, in __init__
    if self.get_repo():
  File "guiscrcpy/lib/ver.py", line 68, in get_repo
    if not repo.git.describe("--tags").startswith('0.'):
  File "git/cmd.py", line 542, in <lambda>
    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
  File "git/cmd.py", line 1005, in _call_process
    return self.execute(call, **exec_kwargs)
  File "git/cmd.py", line 822, in execute
    raise GitCommandError(command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git describe --tags
  stderr: 'fatal: No names found, cannot describe anything.'

Maybe system install is not supported?

Also, it could make sense to package scrcpy and its deps as well in the AppImage?

srevinsaju commented 4 years ago

Thanks @niess for the quick response.

git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git describe --tags
  stderr: 'fatal: No names found, cannot describe anythin

This is because guiscrcpy gets the latest tags from .git folder, as the folder is absent, it can be just well to remove GitPython package.

I am not very sure how AppImage works in reality, for me its just a concept (still learning). Is appimage isolated, like snaps and flatpak, from the rest of the system. If yes, I could ask users to install scrcpy from their packagers.

guiscrcpy's UI will start, if adb is found. It is not compulsory to have scrcpy to run guiscrcpy.

The link to adb by Google, is this, but I am not sure if I am allowed to package this with the AppImage. Is it?

niess commented 4 years ago

@srevinsaju Indeed after uninstalling GitPython as:

./python3.8/AppRun -m pip uninstall GitPython

it works :) At least the GUI starts and prompts for scrcpy location.

Yet, there is this warning message: WARNING:root:GitPython not installed. Fallback to pip3 version reading which might be a bit confusing for end users. In addition, users might already have GitPython installed in there user space (--user) in which case the startup will still fail.

Surprisingly I don't get this issue with the --user install. Maybe you bypass git in this case? Possibly there could be a command line option e.g. --with-git in order to run with GitPython and use the fallback otherwise. Or, if git is only used in order to get the version, then in your setup.py script you could generate a guiscrcpy/version.py file with the git version information when building your package. For example, I am doing this here.

The Python AppImages that I am producing can be considered as relocatable Python installs. There are standard except for this that there can be moved. So there are not strictly isolated. For example, they see the packages in the --user space. Basically, end users could simply get one of those AppImages and then pip install guiscrcpy with it. It should be equivalent to manually installing Python to a custom location and then pip installing your package.

Concerning the legalities related to packaging adb and scrcpy in your AppImage, I have to say that I am not quite sure as well. @probonopd ? Is it enough to add their specific licences in the AppImage?

srevinsaju commented 4 years ago

@niess, Thanks for confirming this.

./python3.8/AppRun -m pip uninstall GitPython

I have pushed a commit to master regarding this on a8da37197361f7d2ac07e2470c1c67a8c08b96ca .

Surprisingly I don't get this issue with the --user install. Maybe you bypass git in this case? Possibly there could be a command line option e.g. --with-git in order to run with GitPython and use the fallback otherwise.

Interesting. Thanks. I have not yet tried a system-wide installation of guiscrcpy, as all of my mentors warn me not to use pip with sudo. Maybe, it might be fixed by a8da37197361f7d2ac07e2470c1c67a8c08b96ca.

The Python AppImages that I am producing can be considered as relocatable Python installs

I did not understand what relocatable is. Does that mean, thst I will not be able to access /usr/bin. As scrcpy binaries are not yet released, should we actually build it from source?

niess commented 4 years ago

@srevinsaju @probonopd I put a preliminary version of the guiscrcpy AppImage over there. Is this OK for you?

It was simply generated by extracting an existing Python 3.8 AppImage and pip installing to it. Note that I did not update the meta data for now, e.g. the desktop file etc.

srevinsaju commented 4 years ago

Thanks @niess. I will check after 8hrs; its quite late in my part of the world 😇. I will let you know when I fix the rest of the GitPython bugs, permanently.

srevinsaju commented 4 years ago

@niess I have tested the AppImage. The GUI starts, but I am not able to see my devices connected on adb. And also, adb is found by guiscrcpy. Is it that, the adb AppImage will not be able to access the devices connected to usb ports / LAN.?

niess commented 4 years ago

@srevinsaju I did not understand. Is adb found or not? Note also that adb is not bundled in this AppImage. It only contains Python 3.8 and guiscrcpy with its Python deps, namely PyQt5.

There is no isolation in this AppImage. It mounts to /tmp at startup. So it sees your native system and reciprocally the native system sees it. In a temporary location though.

srevinsaju commented 4 years ago

yea, ADB is found in the AppImage. but my devices cannot be detected by the adb. I am not sure why. If I extract the AppImage, I am able to detect the adb. @niess, are you able to mirror your mobile screen using guiscrcpy?

srevinsaju commented 4 years ago

@niess , I got a new error, I were trying to debug why the AppImage fails to do screen mirroring

opt/python3.8/bin/guiscrcpy: bad interpreter: /home/niess/Soft/grand/python-appimage/python3.8/usr/bin/../../opt/python3.8/bin/python3.8: no such file or directory

niess commented 4 years ago

@srevinsaju There was a bug in python-appimage resulting in the shebang scripts of upgraded packages not being properly pruned. It should be solved now. I updated the guiscrcpy AppImage as well, still over there.

Yet this will not solve your issues related to adb. Does adb work with the extracted image?

niess commented 4 years ago

@srevinsaju I finally managed to install adb a scrcpy. I had to enable testing repositories that for. I am using Debian 9. On my side the updated AppImage of guiscrcpy seems to work fine. I can control my Android tablet from the PC. Its great by the way :)

Does it still fail for you? What Linux do you use?

srevinsaju commented 4 years ago

Thanks @niess. I had tested the first link to the AppImage and not the second link. My linux distribution is Arch Linux. I will check out the second AppImage once again tomorrow, (after 8hrs). I tested the first AppImage twice, and it did not work as expected, meaning, the GUI starts, but not the scrcpy's window.

niess commented 4 years ago

Fine. I didn't change anything except the shebang of guiscrcpy. So you will likely have the same problems :(

Good Night! :)

srevinsaju commented 4 years ago

@niess , I have found the error. The AppImage works under a certain condition. This might be a bug in your AppImage making script, but I am not sure why exactly it happens.

If the version of guiscrcpy is not installed / is of different version, the AppImage crashes. I am working on 3.7 right now. The AppImage you have built is of 3.5. As I test 3.7, I have an installation of 3.7 on by ~/.local/ level folder. In that case, guiscrcpy GUI works, but I cannot work with adb, scrcpy or anything related to hardware.
If the version of guiscrcpy corresponds to the current guiscrcpy version, the adb connections work, and I am able to do screen mirroring. Perfectly.

I am still debugging on what actually is happening, maybe it is the adb which causes the problem. guiscrcpy searches the /usr/bin, /usr/local/bin for adb. In the case its not found, it asks for the adb. In this case, in the AppImage, should we add the adb to it, because, that might be the cause of the error.

Thanks for the AppImage and your work for scrcpy. If you are interested, you might like to join https://discord.gg/cc6exT

niess commented 4 years ago

@srevinsaju Python packages installed with --user (i.e. under ~/.local on LInux) have higher priority over system ones. This is the usual Python behaviour. It implies that when your guiscrcpy script imports the guiscrcpy package it loads the user one, not the AppImage one. In the case of a Python based app this migt be problematic indeed. I didn't realize this so far because I have nothing installed under --user for Python 3.8.

According to the CLI help you can prevent this by adding the -s option when calling Python. For example, if you extract the guiscrcpy AppImage you can edit the AppRun file and change:

"${APPDIR}/usr/bin/python3.8" "${APPDIR}/opt/python3.8/bin/guiscrcpy" "$@"

to

"${APPDIR}/usr/bin/python3.8" -s "${APPDIR}/opt/python3.8/bin/guiscrcpy" "$@"

Then you can rebuild the image with appimagetool. I updated the guiscrcpy AppImage with this fix if you'd rather download it.

On my side its working, but I have nothings installed under --user.

P.S: Sorry for the inconvenience and thank your for beta testing :P

srevinsaju commented 4 years ago

Thanks @niess . Another user came in trying to run guiscrcpy on Linux Mint, with no supported packages for their system, and they were benefited by this AppImage, Another user have tested it on Linux Mint and it works out of the box. Thanks a lot.

What next can I do for this? like testing or something.

P.S: Sorry for the inconvenience and thank your for beta testing :P

Its absolutely fine. It was my error, which caused the problem

srevinsaju commented 4 years ago

@niess Thanks a lot. I have added the -s flag, and now it works perfectly.

@probonopd @niess how to proceed, like updating metadata and related items? @niess, will you be maintaining guiscrcpy.AppImage?

niess commented 4 years ago

Great :)

I think its better if you do it yourself, if that's OK with you? Concerning the metadata, the logo you'd like to use etc. And so you can publish it directly in the releases of your GitHub repo.

The linux-appimage package allows to automate the generation of the AppImage. You can get it from PyPI for example or from GitHub. You just need to create a directory with a requirements file, an entrypoint script and the app metadata. You can copy one of the examples over there. Maybe the tasmotizer one?

For the requirements.txt you only need to put guiscrcpy. In the entrypoint.sh you need to add the -s option. I will update the examples with this extra flag. The other files are the metadata. When you are done you can run:

python-appimage build app path/to/your/app/meta/folder

You could add those meta to your GitHub repo for example and add a dedicated workflow. You can find example of workflows for building & releasing the AppImages here, e.g. applications.yml and appimage.yml.

srevinsaju commented 4 years ago

Ok, thanks a lot. I appreciate your work on guiscrcpy.AppImage to let me get started on it. Also glad to know that you liked guiscrcpy.

srevinsaju commented 4 years ago

@niess, thanks a lot for the python-appimage tool. I have packaged guiscrcpy adding adb and scrcpy pre-packaged.

The AppImage has been created. It works on my system. I will keep testing on other VMs too.

Thanks a lot again.

@probonopd should I update the PR on @appimagehub?

niess commented 4 years ago

@srevinsaju Great that you managed to set this AppImage :)

I just tried the AppImage without adb and scrcpy installed on my system. It doesn't find adb because it expects it under /usr/bin whereas it is located under $APPDIR/usr/bin where $APPDIR is a temporary location set at runtime by the AppImage. Is there a way to tell guiscrcpy where to look for binaries? E.g. an env var? Or CLI option? Then you could adapt the AppRun (entrypoint.sh) exporting this env var before calling Python or adding the CLI option to the call.

Also I had a look at the deps of adb and guiscrcpy. While adb is pretty standalone, scrcpy depends on libSDL2 among others, which is not packaged in the AppImage. Packaging binaries requires a bit more work than Python packages because you need to package most shared libs as well and to modify the binaries & libs $RUNPATH. This can be done with linuxeploy. For example:

./guiscrcpy-x86_64.AppImage --appimage-extract
./linuxdeploy-x86_64.AppImage --appdir=squashfs-root

Note that you need to have all shared libs available on your system when running linuxdeploy. In addition, for best compatibility this should be run on an old enough Linux (e.g. CentOS 6). I am using the manylinux docker images for this. For example as here.

This is a bit technical. You might also not package adb and scrcpy in your AppImage for now? Then the end user would need to install them himself as when doing pip install guiscrcpy.

srevinsaju commented 4 years ago

Sure, I am willing to do the hard work once, before the upcoming release of 3.7 (#76). Thanks for the information on adb and scrcpy. I had not known about that, as I tested it on my own system. My Arch Linux distro have frequent crashes when running python_appimage tool. So I am now depending on ubuntu:18.04 docker images to build AppImages, as I feel those are pretty old and can fix the deps.

Regarding scrcpy, I might try to package scrcpy with its binaries, I will try, If I succeed with packaging scrcpy using linuxdeploy, I might release 3.7 with the perfect appimage. I wanted to package scrcpy too, because installing scrcpy on systems without a prebuilt package is a mess, and people just don't want to waste their time building new stuff. I will try to do this as what you have told. Thanks a lot

srevinsaju commented 4 years ago

@niess, I have attempted to package binaries of scrcpy for the first time. I am not sure if it would work, I would be grateful to you, if you test this AppImage, for the last time, I hope it works

guiscrcpy-x86-64.AppImage

niess commented 4 years ago

@srevinsaju Its getting closer :P

Youd packaged scrcpy almost runs on my Debian. I had 3 issues, two of which I could solve.

The RPATH is not set on scrcpy. Therefore the packaged libraries are not found by default. You can solve this with patchelf, e.g. as:
```
wget https://github.com/niess/patchelf.appimage/releases/download/rolling/patchelf-x86_64.AppImage
chmod +x patchelf-x86_64.AppImage
./patchelf-x86_64.AppImage --set-rpath '$ORIGIN/../lib' ./squashfs-root/usr/bin/scrcpy
```
This sets a search path relative to the binary location ($ORIGIN). Alternatively you could instead add $APPDIR/usr/lib to LD_LIBRARY_PATH in the entrypoint.sh script, before calling python (e.g. 1st line).
libbluray.so.2 is required but not packaged. That's strange because it is not in the AppImage exclude list. Did you use linuxdeploy for the packaging?

As a temporary fix I added it manually from my package manager.
Once the two previous issues have been patched scrcpy loads but then I get the following error:
```
ERROR: '/usr/local/share/scrcpy/scrcpy-server' does not exist or is not a regular file
```
Indeed this is now located under $APPDIR/usr/share/scrcpy. Is there a way to tell scrcpy to look for scrscpy-server under another location than the default /usr/local/share/scrcpy?

srevinsaju commented 4 years ago

thanks for the test, I did a quick test in Ubuntu 20.04, and it didn't work due to the same reason.

i didn't use linuxdeploy. instead, I manually ran ldd and did a trial and error, to select the libraries and reduce the size of the AppImage by not including the unnecessary ones, in my test, scrcpy worked without libbluray*.so.*. I have the binary with me, and I will push to bintray and fix that packaging error.

I did this, because I didn't understand what Linux deploy is or how to use it properly

I added the precedence of finding adb and scrcpy on 3.7 on 3af3956, but the link I provided to you was 3.5's, which was my mistake. I had used GNU sed to edit the AppRun file created by python-appimage
I had built scrcpy on /usr/local level, and hence it searches for it in /usr/local/ for scrcpy-server, but however I actually placed it in /usr/share. What I would prefer to do it, is to actually create a /usr/local/share/scrcpy in the AppImage and then paste the scrcpy-server in there, which would be cool and I wouldn't put any more effort on it. I would also consider asking @rom1v to specify the scrcpy-server path. it respects some ENV var as I concur.

I apologize, If I am disturbing you at any costs, your testing has helped me find these errors. Thanks a lot again

niess commented 4 years ago

@srevinsaju No problem. Don't hesitate if you need more tests from me.

For me it would be helpful to have scrcpy bundled inside the AppImage. Because installing it with the package manager requires enabling testing which has side effects :( So I'd rather run it from an AppImage if available.

niess commented 4 years ago

@srevinsaju Concerning scrcpy-server not being found apparently one indicate its location with the following env var: SCRCPY_SERVER_PATH. So for example in the entrypoint.sh or AppRun you can add:

export SCRCPY_SERVER_PATH="${APPDIR}/usr/share/scrcpy/scrcpy-server"

niess commented 4 years ago

In addition one needs to add adb to the PATH, e.g. exporting $APPDIR/usr/bin to the PATH in the entrypoint.

With these fixes I could run the packaged scrcpy.

srevinsaju commented 4 years ago

Yes, I am downloading the CentOS 6 Live to test if any more dependency errors are there. On Ubuntu 20.04, along with libbluray, I had to package some more libs. Maybe , I can finally test it on CentOS and confirm that scrcpy doesn't depend on any more libs.

srevinsaju commented 4 years ago

@niess This is my latest AppImage, I have considered the fixes you have mentioned above + some new libraries to run scrcpy

https://github.com/srevinsaju/guiscrcpy/suites/609844413/artifacts/4641630

This release should have 3.7 (beta) and should automatically detect adb using the environment variable GUISCRCPY_ADB and GUISCRCPY_SCRCPY. And also, the ADB variable for scrcpy to scan for devices. It also includes the support for multiple devices. I had a hard time to get dynamic builds (The requirements,txt took guiscrcpy from PYPI. I actually wanted to build the current branch, so it took a bit of time). It havent tested it yet, my bandwidth is too low, I will let you know if it still fails, (hope it doesn't).

srevinsaju commented 4 years ago

@niess , I tested on CentOS 6, but it fails due to a GLIB_C error cause due to the import of PyQt5. This might be because, I ran the python_appimage initially on Ubuntu 18.04 and used Ubuntu 18.04 as my test image. I guess, compiling on older version might help, but I am not sure. Can you let me know if the above AppImage works on your system?

srevinsaju commented 4 years ago

@niess , here is an appimage compiled on ubuntu-16.04 for your reference, https://github.com/srevinsaju/guiscrcpy/suites/610201559/artifacts/4649369

I am downloading this, I am not sure if it would work.

niess commented 4 years ago

@srevinsaju The PyQt5 wheels require CentOS7 or later (manylinux2014 tag). You can see this on PyPI by checking the download files. You'll see that the LInux wheel is labelled manylinux2014 which is built on CentOS7.

niess commented 4 years ago

P.S: therefore you might consider using manylinux2014 as base image when building your app.

srevinsaju commented 4 years ago

ok thanks @niess , I will update the manylinux tag, but to confirm, will an app with base image manylinux2014 work on CentOS6 and other old oses?

niess commented 4 years ago

@srevinsaju I just tested the new AppImage and I confirm that the extracted image works for me (Debian 9) without adb and scrcpy installed :)

However I had the following issue which took me quite some time to debug. At first it was crashing because I had an old config file from a previous run where adb was installed under /usr/bin. Therefore guiscrcpy was trying to run adb using the config file location instead of the AppImage one which was failing. Erasing the old config file solved this issue. Note however that when running from an AppImage (not extracted) the path of adb changes at each run because the image is mounted to a temporary name. Since the config file is re-generated as so it would fail when running the AppImage a second time.

This could be solved by having the ADB env var (or related) having precedence over the config file location.

When running from the live image it currently fails to find adb. I could solved this by simply adding $APPDIR/usr/bin to the PATH in AppRun. E.g. as:

export PATH="${APPDIR}/usr/bin:${PATH}"

niess commented 4 years ago

Concerning manylinux2014 I think that it would fail on CentOS6 though I never tried. But then PyQt5 would fail as well in principle. So except if you can run without PyQt5 there is no benefit for you to use an older image than manylinux2014.

niess commented 4 years ago

I might be wrong :P I am interested in the result if you test. Can PyQt5 wheel run on CentOS6?

Also, just wanted to say that its amlost working now and that's great :)

srevinsaju commented 4 years ago

@niess, No, pip3 install PyQt5 downloads the tar.gz pip3 install PyQt5 --use-wheels fails with unsupported OS, OS not supported error.

PyQt5-5.14.2-5.14.2-cp35.cp36.cp37.cp38-abi3-manylinux2014_x86_64.whl is not a supported wheel on this platform.

Also, just wanted to say that its amlost working now and that's great :)

Thanks a lot. I had attempted to package AppImage 3 months ago, for the same guiscrcpy, and reached no where, and that caused @probonopd to create this issue here, because of the bad AppImage i had submitted to the AppImage hub. If python-appimage wasn't there, I could not have done this, so far.

niess commented 4 years ago

OK this makes sense. Then there is no gain for you to use an older image than manylinux2014 for Python if you use the PyQt5 wheel.

If you want to support older OS (e.g. CentOS 6) you'll need to install PyQt5 from the source distribution using an old distrib (manylinux1, or manylinux2010). But that's probably a tough task on old OS. Otherwise I guess that the PyQt5 wheel would be provided for those.

By the way, Python wheels and AppImages are actually very similar. Both package binary deps by building them on an old distrib (CentOS variations). So once you have a working AppImage you could build a Python wheel from it as well, e.g. that would contain adb and scrcpy in your case. Conversely, if a Python package has a proper wheel, i.e. including all of its binary deps, then its trivial to install it to a Python AppImage. The extra difficulty that we have is related to bundling adb and scrcpy which are non Python binary deps of guiscrcpy.

srevinsaju commented 4 years ago

@niess apparently, my CI is failing ~after your latest commit~, I am debugging it right now I checked all previous versions, some error is causing appimage CI to fail, for a reason which I was not able to understand. My PR which succeeded yesterday, failed today. I am not sure

niess commented 4 years ago

@srevinsaju Indeed, I have similar errors. Sorry for this. I am having a look.

srevinsaju commented 4 years ago

It seems like when we specify a path in the requirements.txt, it crashes, but when we provide a PYPI package, it doesnt

As an aside, is it possible to cache the downloaded AppImage python3.8.2-cp38-cp38-manylinux1_x86_64.AppImage in my system, so I can reuse it without downloading a new version. Thanks

niess commented 4 years ago

There is a typo apparently: No such file or directory: '.../squashfs-root/opt/python3.8/python3.8'. There is a bin missing.

niess commented 4 years ago

I found it. I am testing and patching. Sorry, this was affecting the extracted image which I didn't test before committing. Well this shows that I need some CI test before releasing ...

niess commented 4 years ago

@srevinsaju Concerning the caching, I don't know on GitHub workflows. I never used it. What would be needed as well Is an option to provide a base image on disk when running python-appimage, instead of downloading. I'll add it to the issues.

srevinsaju commented 4 years ago

There is a typo apparently: No such file or directory: '.../squashfs-root/opt/python3.8/python3.8'. There is a bin missing.

Ok, thanks a lot; I took a bit of time, testing it to see something went wrong on my side

@srevinsaju Concerning the caching, I don't know on GitHub workflows. I never used it. What would be needed as well Is an option to provide a base image on disk when running python-appimage, instead of downloading. I'll add it to the issues.

Not Github workflows, on my local machine, I wanted to cache a copy of python, so I don't need to download it every time I build. Thanks, it was maybe a Feature Request

srevinsaju commented 4 years ago

@niess, now the CI builds correctly, Thanks a lot for the quick fix

srevinsaju / guiscrcpy

AppImage is missing a library #56