Building from source: DESTDIR doesn't produce portable installation

kierdavis commented 10 months ago

Describe the bug

In regard to building & installing the v2 CLI from source, the recommended behaviour of the DESTDIR option isn't honored in several places:

The $prefix/bin/aws -> $prefix/lib/aws-cli/bin/aws symlink (and likewise for aws_completer)
The script interpreter of $prefix/lib/aws-cli/bin/aws (and likewise for aws_completer)

Expected Behavior

The symlink created at $DESTDIR/$prefix/bin/aws should point to $prefix/lib/aws-cli/bin/aws.
The first line of $DESTDIR/$prefix/lib/aws-cli/bin/aws should be #!$prefix/lib/aws-cli/bin/python.

More generally, the contents of $DESTDIR should not themselves contain any references to $DESTDIR, so that it's possible to copy the contents of this directory to another machine's root filesystem (typically via a RPM/DEB/etc) and still end up with a working installation.

Current Behavior

$DESTDIR/$prefix/bin/aws points to $DESTDIR/$prefix/lib/aws-cli/bin/aws, so this will be a broken symlink if $DESTDIR contents are packaged up and installed onto another machine.
$DESTDIR/$prefix/lib/aws-cli/bin/aws's first line is #!$DESTDIR/$prefix/..., which won't exist if $DESTDIR contents are packaged up and installed onto another machine.

Reproduction Steps

$ git checkout 2.15.6
$ ./configure --prefix=/usr/local --with-download-deps
...
$ make
...
$ make DESTDIR=$PWD/install_staging install
...
$ readlink ./install_staging/usr/local/bin/aws
/home/kdavis/checkouts/github/aws/aws-cli/install_staging/usr/local/lib/aws-cli/bin/aws
$ head -1 ./install_staging/usr/local/lib/aws-cli/bin/aws
#!/home/kdavis/checkouts/github/aws/aws-cli/install_staging/usr/local/lib/aws-cli/bin/python

Possible Solution

Symlinks can be trivially fixed by making them relative.

The script interpreter must be an absolute path though, so this might require introducing more awareness of MOCKDEST into the build scripts.

Additional Information/Context

No response

CLI version used

2.15.6

Environment details (OS name and version, etc.)

Linux - RHEL 9

tuananh commented 10 months ago

I also faced this issue when trying to build from src.

amberkushwaha commented 10 months ago

The $prefix/bin/aws -> $prefix/lib/aws-cli/bin/aws symlink (and likewise for aws_completer)

The script interpreter of $prefix/lib/aws-cli/bin/aws (and likewise for aws_completer) the script is being prepared in the same obligations and marks as used before formats.

Expected Behavior The symlink created at $DESTDIR/$prefix/bin/aws should point to $prefix/lib/aws-cli/bin/aws.

The first line of $DESTDIR/$prefix/lib/aws-cli/bin/aws should be #!$prefix/lib/aws-cli/bin/python.

More generally, the contents of $DESTDIR should not themselves contain any references to $DESTDIR, so that it's possible to copy the contents of this directory to another machine's root filesystem (typically via a RPM/DEB/etc) and still end up with a working installation.

This code of conduct is still the in the initial stage so please do the needful for the coding ground formats and tempers.

Stealthii commented 7 months ago

For anyone looking towards a workaround for this issue until AWS fix it, you only need to redirect the symlinks and fix the shebangs to address the reference issue.

# Building package
build_dir="$(mktemp -d)"
target_dir="$(mktemp -d)"
tarball_uri="https://github.com/aws/aws-cli/archive/refs/tags/${awscli_version}.tar.gz"
# Download and extract tarball
curl -sL "${tarball_uri}" | tar -C "${build_dir}" -xz
# Configure and make install to target directory
(
        set -e
        cd "${build_dir}/aws-cli-${awscli_version}"
        export PYTHON="${python}"
        ./configure --prefix "/usr" --with-download-deps
        make
        make install DESTDIR="${target_dir}"
)

# Fix broken DESTDIR behaviour in aws-cli
for f in $(find "${target_dir}" -type l); do
        # Fix symlinks
        ln -sfn "$(readlink -f "${f}" | sed "s#${target_dir}##")" "${f}"
done
for f in $(grep -rl "${target_dir}" ${target_dir}); do
        # Fix shebangs
        sed -i "s|${target_dir}||g" $f
done

tim-finnigan commented 5 months ago

Thanks for reporting this issue and your patience here, will plan to bring this up for discussion with the team.

tim-finnigan commented 5 months ago

After bringing this up with the team, the consensus was that more research is required here. Specifically in terms of highlighting the causes and clarifying the scenarios in which it occurs. Will keep this open for further tracking and review, please feel free to share more details in the comments. Edit: Looks like this overlaps with installation symlink issue reported in https://github.com/aws/aws-cli/issues/6852, we may want to consolidate these for tracking.

Stealthii commented 5 months ago

See GNU's DESTDIR documentation for reference on this coding standard, also CMake's DESTDIR documentation for comparison on similar behavior in other build systems.

The expectation is that you can define an installation prefix, such as /usr:

./configure --prefix "/usr"

But run the make install against a DESTDIR as a buildroot target for packaging (such as /tmp/awscli):

make install DESTDIR="/tmp/awscli"

With regards to re-producing the issue, the above comment sets up an example build & target install using this standard pattern.

The issue stems from parts of your build process adhering to DESTDIR (for example the make install correctly places files under the DESTDIR tree), however your current configuration of Make (and the build_system arguments) does not adhere to the prefix when setting up symlinks or shebangs.

TL;DR: DESTDIR should be used to define a staging relocation of the given installation destination directory, not to be confused with prefix which defines the target installation base path.

aws / aws-cli