justincolangelo commented 2 years ago

Command that is running: hab pkg build . Setting pkg_source in a plan.sh file to either a git URL or a zipped file pkg_source="https://location/of/file.tar.bz2" and having do_download() overridden seems to not use the do_download() nor does it try to download the zipped file. The output is always:

Cloning into 'repo'...
Username for '[https://repo-location.com'](https://repo-location.com%27/): fatal: could not read Username for '[https://repo-location.com'](https://repo-location.com%27/): Success

The OS (including version) where you are running any of the Habitat commands.

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

Debug/backtrace of the command you are trying to run. You can set the following environment variables before running the hab command to generate a trace:

Running the hab command with the RUST env vars didn't product a stacktrace. I think maybe because it is just a failure to download from a git remote.

$ hab --version
hab 1.6.521/20220603154827

[ ] Current Habitat environment variables where the hab command or supervisor is running.

   hab-studio: Exported: HAB_AUTH_TOKEN=[redacted]
   hab-studio: Exported: HAB_ORIGIN=[redacted]
   hab-studio: Exported: HAB_BLDR_URL=https://bldr.biome.sh/
   hab-studio: Exported: AWS_DEFAULT_REGION=[redacted]
   hab-studio: Exported: AWS_SECRET_ACCESS_KEY=[redacted]
   hab-studio: Exported: HAB_STUDIO_HOST_ARCH=[redacted]
   hab-studio: Exported: AWS_ACCESS_KEY_ID=[redacted]
   hab-studio: Exported: GITLAB_USER=[redacted]
   hab-studio: Exported: GITLAB_TOKEN=[redacted]

My goal was to have the hab pkg build . command pull from a remote private repo (using either the username:token option in the URL or the SSH clone option. It doesn't seem to matter what I put into the do_download() method or the pkg_source variable.

justincolangelo commented 2 years ago

I wanted to update this issue. This is running as part of a CI/CD process. I had to rm -rf /hab/cache/artifacts/ for it to respect the new values in those plan vars. If that is a known thing then let's close this.

vkurniavka commented 2 years ago

Any news? has the same issue in Docker container, locally works fine but when I'm tying to use linux_x64 Docker image got the same error

atrniv commented 2 years ago

@justincolangelo / @vkurniavka , would you be able to share the plan.sh file you are using so that I could attempt to reproduce this issue ?

vkurniavka commented 2 years ago

@atrniv this is my plan.sh it works locally but fails in docker

pkg_name=$package_name pkg_origin=$origin pkg_version="1.5.1" pkg_major="1"; pkg_minor="5"; pkg_maintainer="maitainer" pkg_source=$source_git_url pkg_license=('MPL-2.0') pkg_filename="$package_name-1.5.1.tar.bz2" pkg_branch=$branch commit_hash=$commit

pkg_deps=( core/coreutils/8.30/20190115012313 core/bash/4.4.19/20190115012619 core/which/2.21/20190430084037 $origin/erlang/22.0 )

pkg_build_deps=( core/coreutils/8.30/20190115012313 core/git/2.26.2/20200601121014 $origin/erlang/22.0 $origin/elixir/1.8.0 core/aws-cli/1.21.11/20220311142250 )

pkg_exports=( [port]=phx.port )

pkg_description="Project $pkg_branch branch commit $commit_hash for $origin"

do_download() { export GIT_SSL_NO_VERIFY=true export GITLAB_TOKEN=$GITLAB_TOKEN export GITLAB_USER=$GITLAB_USER rm -rf ./project* echo "Clone the repo $pkg_source ..." git clone --branch $pkg_branch $pkg_source mv ./project ./project-$pkg_version tar -cjvf $HAB_CACHE_SRC_PATH/$pkg_filename ./project-$pkg_version --exclude project-$pkg_version/.git --exclude project-$pkg_version/spec }

do_verify() { return 0 }

do_prepare() { export LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 export MIX_ENV=prod export RELEASE_VERSION="$pkg_major.$pkg_minor.$(echo $pkg_prefix | cut -d '/' -f 7)"

# Rebar3 will hate us otherwise because it looks for
# /usr/bin/env when it does some of its compiling
[[ ! -f /usr/bin/env ]] && ln -s "$(pkg_path_for coreutils)/bin/env" /usr/bin/env

return 0

}

do_build() { echo "Building project..." mix local.hex --force mix local.rebar --force mix deps.get --only prod mix deps.clean mime --build rm -rf _build mix compile }

do_install() { echo "Installing the build..." rm -rf _build/prod/rel/proj/releases MIX_ENV=prod mix distillery.release

TODO 1.9 releases chmod 0655 _build/prod/rel/proj/bin/*

cp -a _build/prod/rel/proj/* ${pkg_prefix}

for f in $(find ${pkg_prefix} -name '*.sh')
do
    fix_interpreter "$f" core/bash bin/bash
    fix_interpreter "$f" core/coreutils bin/env
    # TODO 1.9 releases chmod 0655 "$f"
done

# TODO 1.9 releases chmod 0655 elixir, bin/erl

}

do_strip() { return 0 }

do_end() { return 0 }

vkurniavka commented 2 years ago

@atrniv any updates? Now I'm facing the same issue on local machine and have no chance to build project

justincolangelo commented 2 years ago

We are seeing issues downloading hab studio now.

‘’’ ✗✗✗ Failed to install package core/hab-studio/1.6.521 - When applicable, we try once, then re-attempt 5 times to [381]✗✗✗ download a package. Unfortunately, we failed to download [382](✗✗✗ core/hab-studio/1.6.521/20220603160937 for x86_64-linux. Last error: [404 Not Found] ‘’’

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

habitat-sh / habitat

pkg_source not respected and do_download() overwrite not seen #8561

TODO 1.9 releases chmod 0655 _build/prod/rel/proj/bin/*