Inconsistent APPDIR_LIBRARY_PATH generation

[mschuckmann]

I've got a problem where the path is APPDIR_LIBRARY_PATH (and by extension LD_LIBRARY_PATH) are not being generated consistently the same from build to build, is there (or has there ever been any known issue with the _get_appdir_library_paths() function?

The specific problem that keeps coming up has to do with the libblas shared library which is being included in my AppImge at $APPDIR/usr/lib/x86_64-linux-gui/libblas. The basic problem is that sometimes when I build my AppImage this directory is added to APPDIR_LIBRARY_PATH, but sometimes the path is not added and when it's not in the path my AppImage fails to run because it can't find the libblas shared library. I've verified that in either case the actual shared library is always included in the AppImage.

I've also reviewed the __get_appdir_library_paths() function and I can't see any reason why it would sometimes fail, unless there was some sort of race condition going on.

Any help would be very much appreciated.

Matt S.

P.S. I should note that I'm using a version of AppImage that was installed to my docker image around April 10, 2022. I'm currently stuck at this version because of various other problems I've had with with newer versions of AppImage-builder.

[azubieta]

Yes there are race conditions on v1.0.2 which should be addressed now. They were produced by the usage of the system shell to run binaries inside the bundle (which was causing a silent failure).

The changes were merged to main and are available in the latest continuous build. Could you please give it a try. It would be great if we could isolate the faulty parts from your recipe into a test case so I could make sure I will not broke it again.

[mschuckmann]

So I am unable to test the current continuous build with my yaml recipes because the shbang in many of the python scripts in the image are now relative paths instead of absolute with respect to the APPDIR

For example my recipe bundles parts of ROS in the AppImage, this includes the /opt/ros/galactic/lib/rqt_reconfigure/rqt_reconfigure script which looks like this

#!/usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'rqt-reconfigure==1.0.8','console_scripts','rqt_reconfigure'
__requires__ = 'rqt-reconfigure==1.0.8'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('rqt-reconfigure==1.0.8', 'console_scripts', 'rqt_reconfigure')()
    )

With the current continuous build of AppImage-builder the bundled file will not run and looks like this:

#! usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'rqt-reconfigure==1.0.8','console_scripts','rqt_reconfigure'
__requires__ = 'rqt-reconfigure==1.0.8'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('rqt-reconfigure==1.0.8', 'console_scripts', 'rqt_reconfigure')()
    )

With an older build of AppImage-builder (prior to v1.0.0) the bundled script does run and looks like this:

#!/tmp/appimage-qfgEJoQ-env python3
# EASY-INSTALL-ENTRY-SCRIPT: 'rqt-reconfigure==1.0.8','console_scripts','rqt_reconfigure'
__requires__ = 'rqt-reconfigure==1.0.8'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('rqt-reconfigure==1.0.8', 'console_scripts', 'rqt_reconfigure')()
    )

It is unclear to me if I'm doing something wrong or if there is a bug in AppImage-builder or ?? Would you like me to create different issue for this or?

[azubieta]

The change on the shebang is intentional, this is used to make sure that the scripts get executed using the python version embed in the AppImage.

If you those scripts to be executed using the system python you need to list them at AppDir > runtime > preserve it expects a list of paths.

[mschuckmann]

I believe that I want them to be run using the Python version that is included in the AppImage, how do I make that happen?

Also is there a place where this new functionality is documented?

[azubieta]

You can find some information about why this is done here: https://github.com/AppImageCrafters/AppRun/blob/master/docs/USAGE.md#patch-shebang-on-scripts

To try your script directly you can call it from the AppRun. Edit the exec and exec_args entries at AppRun.env to point your script.

[mschuckmann]

So here's the deal I'm not running just one script, I'm running several scripts and executables, it's kind of how ROS works. My main AppRun entry is a bash script that runs a ROS launch file, that ROS launch file runs multiple other processes, some are Python applications, some are compiled programs.

[azubieta]

I get an idea. First you have to bundle bash and make that your main entry point with your start-up script as argument. From that script you should be able to call your other python binaries without issues.

[mschuckmann]

@azubieta so your last idea is exactly what I've been doing and has stopped working with recent versions of appimage-builder. The problem is the new shbang's use a relative path and thus assume the working directory is $APPDIR and this can't always be controlled as is the case with the ROS launch file mechanism.

Is there a reason why the modified shbang can't use the $APPDIR environment variable to locate the interpreter?

[azubieta]

Is there a reason why the modified shbang can't use the $APPDIR environment variable to locate the interpreter?

Sadly we cannot have environment variables in the binary path of the shebang. So it must be a fixed path or a relative path. Maybe you could share your broken AppDir or a part of it for me to check it.

[mschuckmann]

I'll put together a quick example for you to try.

[mschuckmann]

@azubieta The attached zip file has a .yaml, bash script and ros launch file that demonstrates the problem I'm having.

With an install of appimage-builder from April 10, 2022 the AppImage built from this recipe works correctly in that the ROS2 applications rviz2 and rqt are launched.

With a current install of appimage-builder (version 1.0.3.dev14+g7b1fe9b) the built AppImage fails to run properly in that the ROS2 applications rvis2 and rqt are not launched. The errors are below, the basic problem seems to be that the shbang in the files static_transform_publisher, rvis2, rqt_reconfigure have been modified to relative paths and that simply does not work with the ROS launch mechanism.

I realize that ROS2 is a unique beast that you might not be all that familiar with but I can't be the only person that needs to be able to launch python applications via a secondary application where the current working directory can't be assumed.

appimage-ros2-test.zip

$ ./appimage-ros2-0.0.0-x86_64.AppImage 
[INFO] [launch]: All log files can be found below /home/mschuckmann/.ros/log/2022-06-15-19-14-55-781647-aether-3059
[INFO] [launch]: Default logging verbosity is set to INFO
[ERROR] [static_transform_publisher-1]: exception occurred while executing process:
Traceback (most recent call last):
  File "/tmp/.mount_appimauCtdv8/opt/ros/galactic/lib/python3.8/site-packages/launch/actions/execute_process.py", line 737, in __execute_process
    transport, self._subprocess_protocol = await async_execute_process(
  File "/tmp/.mount_appimauCtdv8/opt/ros/galactic/lib/python3.8/site-packages/osrf_pycommon/process_utils/async_execute_process_asyncio/impl.py", line 139, in async_execute_process
    transport, protocol = await _async_execute_process_nopty(
  File "/tmp/.mount_appimauCtdv8/opt/ros/galactic/lib/python3.8/site-packages/osrf_pycommon/process_utils/async_execute_process_asyncio/impl.py", line 45, in _async_execute_process_nopty
    transport, protocol = await loop.subprocess_exec(
  File "/tmp/.mount_appimauCtdv8/usr/lib/python3.8/asyncio/base_events.py", line 1630, in subprocess_exec
    transport = await self._make_subprocess_transport(
  File "/tmp/.mount_appimauCtdv8/usr/lib/python3.8/asyncio/unix_events.py", line 197, in _make_subprocess_transport
    transp = _UnixSubprocessTransport(self, protocol, args, shell,
  File "/tmp/.mount_appimauCtdv8/usr/lib/python3.8/asyncio/base_subprocess.py", line 36, in __init__
    self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
  File "/tmp/.mount_appimauCtdv8/usr/lib/python3.8/asyncio/unix_events.py", line 789, in _start
    self._proc = subprocess.Popen(
  File "/tmp/.mount_appimauCtdv8/usr/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/tmp/.mount_appimauCtdv8/usr/lib/python3.8/subprocess.py", line 1592, in _execute_child
    self._posix_spawn(args, executable, env, restore_signals,
  File "/tmp/.mount_appimauCtdv8/usr/lib/python3.8/subprocess.py", line 1543, in _posix_spawn
    self.pid = os.posix_spawn(executable, args, env, **kwargs)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/.mount_appimauCtdv8/opt/ros/galactic/lib/tf2_ros/static_transform_publisher'

[ERROR] [rviz2-2]: exception occurred while executing process:
Traceback (most recent call last):
  File "/tmp/.mount_appimauCtdv8/opt/ros/galactic/lib/python3.8/site-packages/launch/actions/execute_process.py", line 737, in __execute_process
    transport, self._subprocess_protocol = await async_execute_process(
  File "/tmp/.mount_appimauCtdv8/opt/ros/galactic/lib/python3.8/site-packages/osrf_pycommon/process_utils/async_execute_process_asyncio/impl.py", line 139, in async_execute_process
    transport, protocol = await _async_execute_process_nopty(
  File "/tmp/.mount_appimauCtdv8/opt/ros/galactic/lib/python3.8/site-packages/osrf_pycommon/process_utils/async_execute_process_asyncio/impl.py", line 45, in _async_execute_process_nopty
    transport, protocol = await loop.subprocess_exec(
  File "/tmp/.mount_appimauCtdv8/usr/lib/python3.8/asyncio/base_events.py", line 1630, in subprocess_exec
    transport = await self._make_subprocess_transport(
  File "/tmp/.mount_appimauCtdv8/usr/lib/python3.8/asyncio/unix_events.py", line 197, in _make_subprocess_transport
    transp = _UnixSubprocessTransport(self, protocol, args, shell,
  File "/tmp/.mount_appimauCtdv8/usr/lib/python3.8/asyncio/base_subprocess.py", line 36, in __init__
    self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
  File "/tmp/.mount_appimauCtdv8/usr/lib/python3.8/asyncio/unix_events.py", line 789, in _start
    self._proc = subprocess.Popen(
  File "/tmp/.mount_appimauCtdv8/usr/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/tmp/.mount_appimauCtdv8/usr/lib/python3.8/subprocess.py", line 1592, in _execute_child
    self._posix_spawn(args, executable, env, restore_signals,
  File "/tmp/.mount_appimauCtdv8/usr/lib/python3.8/subprocess.py", line 1543, in _posix_spawn
    self.pid = os.posix_spawn(executable, args, env, **kwargs)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/.mount_appimauCtdv8/opt/ros/galactic/lib/rviz2/rviz2'

[ERROR] [rqt_reconfigure-3]: exception occurred while executing process:
Traceback (most recent call last):
  File "/tmp/.mount_appimauCtdv8/opt/ros/galactic/lib/python3.8/site-packages/launch/actions/execute_process.py", line 737, in __execute_process
    transport, self._subprocess_protocol = await async_execute_process(
  File "/tmp/.mount_appimauCtdv8/opt/ros/galactic/lib/python3.8/site-packages/osrf_pycommon/process_utils/async_execute_process_asyncio/impl.py", line 139, in async_execute_process
    transport, protocol = await _async_execute_process_nopty(
  File "/tmp/.mount_appimauCtdv8/opt/ros/galactic/lib/python3.8/site-packages/osrf_pycommon/process_utils/async_execute_process_asyncio/impl.py", line 45, in _async_execute_process_nopty
    transport, protocol = await loop.subprocess_exec(
  File "/tmp/.mount_appimauCtdv8/usr/lib/python3.8/asyncio/base_events.py", line 1630, in subprocess_exec
    transport = await self._make_subprocess_transport(
  File "/tmp/.mount_appimauCtdv8/usr/lib/python3.8/asyncio/unix_events.py", line 197, in _make_subprocess_transport
    transp = _UnixSubprocessTransport(self, protocol, args, shell,
  File "/tmp/.mount_appimauCtdv8/usr/lib/python3.8/asyncio/base_subprocess.py", line 36, in __init__
    self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
  File "/tmp/.mount_appimauCtdv8/usr/lib/python3.8/asyncio/unix_events.py", line 789, in _start
    self._proc = subprocess.Popen(
  File "/tmp/.mount_appimauCtdv8/usr/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/tmp/.mount_appimauCtdv8/usr/lib/python3.8/subprocess.py", line 1592, in _execute_child
    self._posix_spawn(args, executable, env, restore_signals,
  File "/tmp/.mount_appimauCtdv8/usr/lib/python3.8/subprocess.py", line 1543, in _posix_spawn
    self.pid = os.posix_spawn(executable, args, env, **kwargs)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/.mount_appimauCtdv8/opt/ros/galactic/lib/rqt_reconfigure/rqt_reconfigure'

[azubieta]

A question, are you calling those apps using subprocess.run(shell=True, ...) ? If that's is the case I would recommend bundling a copy of bash and calling that bash with a full path i.e.: subprocess.run(["/path/to/bundled/bash", "-s", "other argrs"].

Why? appimage-builder uses the AppRun framework to maintain a private environment which should be used only by the bundled binaries. This private environment is removed in case an external binary is executed, /bin/sh is an external binary and it used by popen to run other application if it doesn't receive a fixed path. In conclusion the private environment is being removed before reaching your rqt_reconfigure script.

If your application will NEVER require to use the system shell you can create a path mapping like this /bin/sh:$APPDIR/bin/sh or something similar for bash. Again, be aware that all scripts will be executed using the AppImage private environment.

[mschuckmann]

So I’m not calling those apps directly the ROS infrastructure is and I have no control over how ROS has implemented launching applications and or scripts. I can try to look into how they do it, but making a change to the usage of subprocess.popen () like you suggest is probably not in the cards.

I can try creating a path mapping for /bin/sh like you suggest but it’s not clear to me how that’s going to solve the problem of relative paths in the shbang.

Matt S.

On Thu, Jun 16, 2022 at 9:25 AM Alexis López Zubieta < @.***> wrote:

A question, are you calling those apps using subprocess.run(shell=True, ...) ? If that's is the case I would recommend bundling a copy of bash and calling that bash with a full path i.e.: subprocess.run(["/path/to/bundled/bash", "-s", "other argrs"].

Why? appimage-builder uses the AppRun framework https://github.com/AppImageCrafters/AppRun to maintain a private environment which should be used only by the bundled binaries. This private environment is removed in case an external binary is executed, /bin/sh is an external binary and it used by popen to run other application if it doesn't receive a fixed path. In conclusion the private environment is being removed before reaching your rqt_reconfigure script.

If your application will NEVER require to use the system shell you can create a path mapping like this /bin/sh:$APPDIR/bin/sh or something similar for bash. Again, be aware that all scripts will be executed using the AppImage private environment.

— Reply to this email directly, view it on GitHub https://github.com/AppImageCrafters/appimage-builder/issues/239#issuecomment-1157875266, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADYXOSVWLDXM3253UI7HP4DVPNIPNANCNFSM5YLQYXIA . You are receiving this because you authored the thread.Message ID: @.***>

-- Matt S

[mschuckmann]

So unfortunately that does not work with the current implementation of AppImage-builder (it used to work prior to v1.0.0). I think it has to do with how ROS launches subprocesses, it must assume that the shebang is valid and an absolue path as opposed to the relative path you have now.

Is there any reason why the shbangs can't be modified to use the $APPDIR environment variable? Something like this:

! ${APPDIR}/usr/bin/python3

I may try doing this manually for the scripts I care about.

Matt S.

On Sat, Jun 11, 2022 at 9:13 AM Alexis López Zubieta < @.***> wrote:

I get an idea. First you have to bundle bash and make that your main entry point with your start-up script as argument. From that script you should be able to call your other python binaries without issues.

— Reply to this email directly, view it on GitHub https://github.com/AppImageCrafters/appimage-builder/issues/239#issuecomment-1152957992, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADYXOSWIHETQUOQKAXVUFUTVOS3KPANCNFSM5YLQYXIA . You are receiving this because you authored the thread.Message ID: @.***>