Fetching dependency by git ref fails when upgrading to Erlang 26

der-ambi commented 1 month ago

Pre-Check

[x] If you are filing for a bug, please do a quick search in current issues first
For bugs, mention if you are willing or interested in helping fix the issue:
- Would like to, but don't have Erlang experience
For questions or support, it helps to include context around your project or problem
- After upgrading from Erlang 14 to Erlang 26, our Buildroot build pipeline has stopped working.
Think of a descriptive title (more descriptive than 'feature X is broken' unless it is fully broken)

Environment

Rebar version: 3.22.1 git version: 2.40.1

rebar.config:

[...]
{deps, [
    {gun, "1.3.3"},
    {certifi, "2.9.0"},
    {grisp_updater, {git, "https://github.com/grisp/grisp_updater.git", {ref, "9b47ea1"}}}
]}.
[...]

Current behaviour

Describe the current behaviour. In case of a failure, crash, or exception, please include the result of running the command with debug information:

DEBUG=1 DIAGNOSTIC=1 rebar3 as prod release
===> Expanded command sequence to be run: [as]
===> Running provider: as
===> Expanded command sequence to be run: [app_discovery,install_deps,lock,compile,release]
===> Running provider: app_discovery
===> Found top-level apps: [dsaplus_updater]
    using config: [{src_dirs,["src"]},{lib_dirs,["apps/*","lib/*","."]}]
===> Running provider: install_deps
===> Verifying dependencies...
===> Fetching certifi v2.9.0
===> Making request to get package certifi from repo hexpm (#{name => <<"hexpm">>,repo_verify_origin => true,
         repo_name => <<"hexpm">>,repo_url => <<"https://repo.hex.pm">>,
         repo_organization => undefined,api_url => <<"https://hex.pm/api">>,
         repo_verify => true})
===> Version cached at /home/gitlab-runner/.cache/rebar3/hex/hexpm/packages/certifi-2.9.0.tar is up to date, reusing it
===> sh info:
    cwd: "/home/gitlab-runner/build/HJSQmqA-/0/xxx/yyy/buildroot/output/build/xxx-updater"
    cmd: rm -rf /home/gitlab-runner/build/HJSQmqA-/0/xxx/yyy/buildroot/output/build/xxx-updater/_build/default/lib/certifi

===>    opts: [{use_stdout,false},abort_on_error]

===> Port Cmd: rm -rf /home/gitlab-runner/build/HJSQmqA-/0/xxx/yyy/buildroot/output/build/xxx-updater/_build/default/lib/certifi
Port Opts: [exit_status,
            {line,16384},
            use_stdio,stderr_to_stdout,hide,eof,binary]

===> Moving checkout "/tmp/.tmp_dir411272103885" to "/home/gitlab-runner/build/HJSQmqA-/0/xxx/yyy/buildroot/output/build/xxx-updater/_build/default/lib/certifi"
===> sh info:
    cwd: "/home/gitlab-runner/build/HJSQmqA-/0/xxx/yyy/buildroot/output/build/xxx-updater"
    cmd: mv /tmp/.tmp_dir411272103885 /home/gitlab-runner/build/HJSQmqA-/0/xxx/yyy/buildroot/output/build/xxx-updater/_build/default/lib/certifi

===>    opts: [{use_stdout,false},abort_on_error]

===> Port Cmd: mv /tmp/.tmp_dir411272103885 /home/gitlab-runner/build/HJSQmqA-/0/xxx/yyy/buildroot/output/build/xxx-updater/_build/default/lib/certifi
Port Opts: [exit_status,
            {line,16384},
            use_stdio,stderr_to_stdout,hide,eof,binary]

===> Linking _build/default/lib/certifi to _build/prod/lib/certifi
===> Fetching grisp_updater (from {git,"https://github.com/grisp/grisp_updater.git",
                         {ref,"9b47ea1"}})
===> sh info:
    cwd: "/home/gitlab-runner/build/HJSQmqA-/0/xxx/yyy/buildroot/output/build/xxx-updater"
    cmd: git --version

===>    opts: []

===> Port Cmd: git --version
Port Opts: [exit_status,
            {line,16384},
            use_stdio,stderr_to_stdout,hide,eof,binary]

===> sh info:
    cwd: "/home/gitlab-runner/build/HJSQmqA-/0/xxx/yyy/buildroot/output/build/xxx-updater"
    cmd: git clone  -n https://github.com/grisp/grisp_updater.git .tmp_dir380264277771

===>    opts: [{cd,"/tmp"}]

===> Port Cmd: git clone  -n https://github.com/grisp/grisp_updater.git .tmp_dir380264277771
Port Opts: [{cd,"/tmp"},
            exit_status,
            {line,16384},
            use_stdio,stderr_to_stdout,hide,eof,binary]

===> sh info:
    cwd: "/home/gitlab-runner/build/HJSQmqA-/0/xxx/yyy/buildroot/output/build/xxx-updater"
    cmd: git checkout -q 9b47ea1

===>    opts: [{cd,"/tmp/.tmp_dir380264277771"}]

===> Port Cmd: git checkout -q 9b47ea1
Port Opts: [{cd,"/tmp/.tmp_dir380264277771"},
            exit_status,
            {line,16384},
            use_stdio,stderr_to_stdout,hide,eof,binary]

===> sh(git checkout -q 9b47ea1)
failed with return code 128 and the following output:
fatal: not a git repository: '.'

It looks to me as if the git checkout is attempted in the wrong directory as the cd /tmp/ specified in the opts does not happen (cwd does not change across command outputs)

Expected behaviour

The command works as it did with Erlang 24.3.2

ferd commented 1 month ago

I could not replicate the issue locally (git version 2.39.3):

→ rebar3 compile
===> Verifying dependencies...
===> Fetching certifi v2.9.0
===> Fetching grisp_updater (from {git,"https://github.com/grisp/grisp_updater.git",
                         {ref,"9b47ea1"}})
...
===> Compiling grisp_updater
===> Analyzing applications...
===> Compiling chk
λ [vps] chk → rebar3 version
rebar 3.23.0+build.5377.refbfc6839d on Erlang/OTP 26 Erts 14.1.1

Specifically the diagnostic view of checking out grisp_updater:

===> Fetching grisp_updater (from {git,"https://github.com/grisp/grisp_updater.git",
                         {ref,"9b47ea1"}})
===> sh info:
        cwd: "/tmp/chk"
        cmd: git --version

===>    opts: []

===> Port Cmd: git --version
Port Opts: [exit_status,
            {line,16384},
            use_stdio,stderr_to_stdout,hide,eof,binary]

===> sh info:
        cwd: "/tmp/chk"
        cmd: git clone  -n https://github.com/grisp/grisp_updater.git .tmp_dir25254007423

===>    opts: [{cd,"/tmp"}]

===> Port Cmd: git clone  -n https://github.com/grisp/grisp_updater.git .tmp_dir25254007423
Port Opts: [{cd,"/tmp"},
            exit_status,
            {line,16384},
            use_stdio,stderr_to_stdout,hide,eof,binary]

===> sh info:
        cwd: "/tmp/chk"
        cmd: git checkout -q 9b47ea1

===>    opts: [{cd,"/tmp/.tmp_dir25254007423"}]

===> Port Cmd: git checkout -q 9b47ea1
Port Opts: [{cd,"/tmp/.tmp_dir25254007423"},
            exit_status,
            {line,16384},
            use_stdio,stderr_to_stdout,hide,eof,binary]

===> sh info:
        cwd: "/tmp/chk"
        cmd: rm -rf /tmp/chk/_build/default/lib/grisp_updater

===>    opts: [{use_stdout,false},abort_on_error]

===> Port Cmd: rm -rf /tmp/chk/_build/default/lib/grisp_updater
Port Opts: [exit_status,
            {line,16384},
            use_stdio,stderr_to_stdout,hide,eof,binary]

===> Moving checkout "/tmp/.tmp_dir25254007423" to "/tmp/chk/_build/default/lib/grisp_updater"
===> sh info:
        cwd: "/tmp/chk"
        cmd: mv /tmp/.tmp_dir25254007423 /tmp/chk/_build/default/lib/grisp_updater

===>    opts: [{use_stdout,false},abort_on_error]

===> Port Cmd: mv /tmp/.tmp_dir25254007423 /tmp/chk/_build/default/lib/grisp_updater
Port Opts: [exit_status,
            {line,16384},
            use_stdio,stderr_to_stdout,hide,eof,binary]

I'm not sure if there's anything weird about the runners in gitlab or their filesystem that would make this happen? Are you able to replicate in other environments?

der-ambi commented 1 month ago

Found the culprit: The new buildroot release sets GIT_DIR=. (https://github.com/buildroot/buildroot/commit/c07aafa087927ac6f10591bc13e30b7ecd5c7b7f)

This results in git checkout looking for the .git directory in the directory where the pipeline checked the tool that contains the rebar config files instead of the checked out dependency.

Not sure if you want to guard rebar against this, otherwise this issue can be closed.

erlang / rebar3