Transient crash when upgrading a custom compiler plugin

ElectronicRU commented 3 years ago

The issue

When upgrading a custom compiler plugin (it compiles JSON Schema v4 to Erlang), a transient crash (undef) happens. It happens consistently when an upgrade actually happens, but does fine (and re-compiles it fine), when you re-run.

I'll try and compile a minimal test case, if necessary.

Environment

Unfortunately, I'm not able to open-source the project yet, but I'm working on it. The plugin is your run of the mill rebar_compiler behaviour, does nothing weird, and calls any specific code only in the compile callback -- otherwise pretty similar to e.g. rebar_compiler_yrl.

Rebar3 report
 version 3.15.1
 generated at 2021-04-20T07:58:25+00:00
=================
Please submit this along with your issue at https://github.com/erlang/rebar3/issues (and feel free to edit out private information, if any)
-----------------
Task: plugins
Entered as:
  plugins upgrade jsstyped_rebar
-----------------
Operating System: win32
ERTS: Erlang/OTP 23 [erts-11.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1]
Root Directory: c:/Program Files/erl-23.3
Library directory: c:/Program Files/erl-23.3/lib
-----------------
Loaded Applications:
bbmustache: 1.10.0
certifi: 2.5.3
cf: 0.3.1
common_test: 1.20
compiler: 7.6.7
crypto: 4.9
cth_readable: 1.5.1
dialyzer: 4.3.1
edoc: 0.12
erlware_commons: 1.4.0
eunit: 2.6
eunit_formatters: 0.5.0
getopt: 1.0.1
hipe: 4.0.1
inets: 7.3.2
kernel: 7.3
providers: 1.8.1
public_key: 1.10
relx: 4.4.0
sasl: 4.0.2
snmp: 5.8
ssl_verify_fun: 1.1.6
stdlib: 3.14.1
syntax_tools: 2.5
tools: 3.4.4

-----------------
Escript path: c:/Users/alex0/Documents/Sources/InnoChain/innobpapi/../rebar3
Providers:
  app_discovery as clean compile compile cover ct deps dialyzer do edoc escriptize eunit get-deps help install install_deps list lock new path pkgs release relup report repos shell state tar tree unlock update upgrade upgrade upgrade version xref

Current behaviour

Running the command after the ref has updated causes a crash.

DIAGNOSTIC=1 rebar3 plugins upgrade jsstyped_rebar
===> Load global config file c:/Users/alex0/.config/rebar3/rebar.config
===> Setting paths to [deps]
===> Compile (apps)
===> Setting paths to [plugins]
===> Setting paths to [deps]
===> Setting paths to [plugins]
===> Setting paths to [plugins]
===> Expanded command sequence to be run: []
===> Running provider: do
===> Expanded command sequence to be run: [{plugins,upgrade}]
===> Running provider: {plugins,upgrade}
===> Getting definition for package root from repo hexpm (#{api_url => <<"https://hex.pm/api">>,name => <<"hexpm">>,
         repo_name => <<"hexpm">>,repo_organization => undefined,
         repo_url => <<"https://repo.hex.pm">>,repo_verify => true,
         repo_verify_origin => true})
===> Hex get_package request failed: {ok,
                                             {403,
                                              #{<<"accept-ranges">> =>
                                                 <<"bytes">>,
                                                <<"age">> => <<"0">>,
                                                <<"connection">> =>
                                                 <<"keep-alive">>,
                                                <<"content-length">> =>
                                                 <<"243">>,
                                                <<"content-type">> =>
                                                 <<"application/xml">>,
                                                <<"date">> =>
                                                 <<"Tue, 20 Apr 2021 07:56:09 GMT">>,
                                                <<"server">> => <<"AmazonS3">>,
                                                <<"via">> => <<"1.1 varnish">>,
                                                <<"x-amz-id-2">> =>
                                                 <<"YoKrIbQztO4SKzdpN+lSLB0SleMjweFmTEbixfgsvtGD4xVIBAfNcMPFAmjub0VYrHt3yp3iqRg=">>,
                                                <<"x-amz-request-id">> =>
                                                 <<"TEWM59EZ2W52YHES">>,
                                                <<"x-cache">> =>
                                                 <<"MISS, MISS">>,
                                                <<"x-cache-hits">> =>
                                                 <<"0, 0">>,
                                                <<"x-served-by">> =>
                                                 <<"cache-dca17765-DCA, cache-fra19177-FRA">>,
                                                <<"x-timer">> =>
                                                 <<"S1618905370.711057,VS0,VE125">>},
                                              <<"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>TEWM59EZ2W52YHES</RequestId><HostId>YoKrIbQztO4SKzdpN+lSLB0SleMjweFmTEbixfgsvtGD4xVIBAfNcMPFAmjub0VYrHt3yp3iqRg=</HostId></Error>">>}}
===> Failed to update package root from repo hexpm
===> Failed to fetch updates for package root from repo hexpm
===> sh info:
        cwd: "c:/Users/alex0/Documents/Sources/InnoChain/innobpapi"
        cmd: git --version

===>    opts: []

===> Port Cmd: cmd /q /c git --version
Port Opts: [exit_status,
            {line,16384},
            use_stdio,stderr_to_stdout,hide,eof,binary]

===> sh info:
        cwd: "c:/Users/alex0/Documents/Sources/InnoChain/innobpapi"
        cmd: git fetch origin v011_support

===>    opts: [{cd,"c:/Users/alex0/Documents/Sources/InnoChain/innobpapi/_build/default/plugins/jsstyped_rebar"}]

===> Port Cmd: cmd /q /c git fetch origin v011_support
Port Opts: [{cd,"c:/Users/alex0/Documents/Sources/InnoChain/innobpapi/_build/default/plugins/jsstyped_rebar"},
            exit_status,
            {line,16384},
            use_stdio,stderr_to_stdout,hide,eof,binary]

===> sh info:
        cwd: "c:/Users/alex0/Documents/Sources/InnoChain/innobpapi"
        cmd: git log HEAD..origin/v011_support --oneline

===>    opts: [{cd,"c:/Users/alex0/Documents/Sources/InnoChain/innobpapi/_build/default/plugins/jsstyped_rebar"}]

===> Port Cmd: cmd /q /c git log HEAD..origin/v011_support --oneline
Port Opts: [{cd,"c:/Users/alex0/Documents/Sources/InnoChain/innobpapi/_build/default/plugins/jsstyped_rebar"},
            exit_status,
            {line,16384},
            use_stdio,stderr_to_stdout,hide,eof,binary]

===> Checking git branch v011_support for updates
===> Upgrading jsstyped_rebar (from {git,"git@gitlab.com:InnoChain/jsstyped_rebar.git",
                          {branch,"v011_support"}})
===> sh info:
        cwd: "c:/Users/alex0/Documents/Sources/InnoChain/innobpapi"
        cmd: git clone  git@gitlab.com:InnoChain/jsstyped_rebar.git .tmp_dir248202452179 -b v011_support --single-branch

===>    opts: [{cd,"c:/Users/alex0/AppData/Local/Temp"}]

===> Port Cmd: cmd /q /c git clone  git@gitlab.com:InnoChain/jsstyped_rebar.git .tmp_dir248202452179 -b v011_support --single-branch
Port Opts: [{cd,"c:/Users/alex0/AppData/Local/Temp"},
            exit_status,
            {line,16384},
            use_stdio,stderr_to_stdout,hide,eof,binary]

===> sh info:
        cwd: "c:/Users/alex0/Documents/Sources/InnoChain/innobpapi"
        cmd: rd /q /s "c:\\Users\\alex0\\Documents\\Sources\\InnoChain\\innobpapi\\_build\\default\\plugins\\jsstyped_rebar"

===>    opts: [{use_stdout,false},return_on_error]

===> Port Cmd: cmd /q /c rd /q /s "c:\\Users\\alex0\\Documents\\Sources\\InnoChain\\innobpapi\\_build\\default\\plugins\\jsstyped_rebar"
Port Opts: [exit_status,
            {line,16384},
            use_stdio,stderr_to_stdout,hide,eof,binary]

===> Moving checkout "c:/Users/alex0/AppData/Local/Temp/.tmp_dir248202452179" to "c:/Users/alex0/Documents/Sources/InnoChain/innobpapi/_build/default/plugins/jsstyped_rebar"
===> sh info:
        cwd: "c:/Users/alex0/Documents/Sources/InnoChain/innobpapi"
        cmd: robocopy /move /e "c:\\Users\\alex0\\AppData\\Local\\Temp\\.tmp_dir248202452179" "c:\\Users\\alex0\\Documents\\Sources\\InnoChain\\innobpapi\\_build\\default\\plugins\\jsstyped_rebar" 1> nul

===>    opts: [{use_stdout,false},return_on_error]

===> Port Cmd: cmd /q /c robocopy /move /e "c:\\Users\\alex0\\AppData\\Local\\Temp\\.tmp_dir248202452179" "c:\\Users\\alex0\\Documents\\Sources\\InnoChain\\innobpapi\\_build\\default\\plugins\\jsstyped_rebar" 1> nul
Port Opts: [exit_status,
            {line,16384},
            use_stdio,stderr_to_stdout,hide,eof,binary]

===> No upgrade needed for jsx
===> Compile (plugins)
===> Running hooks for compile in app jsstyped_rebar (c:/Users/alex0/Documents/Sources/InnoChain/innobpapi/_build/default/plugins/jsstyped_rebar) with configuration:
===>    {pre_hooks, []}.
===> run_hooks("c:/Users/alex0/Documents/Sources/InnoChain/innobpapi/_build/default/plugins/jsstyped_rebar", pre_hooks, compile) -> no hooks defined

===> Running hooks for erlc_compile in app jsstyped_rebar (c:/Users/alex0/Documents/Sources/InnoChain/innobpapi/_build/default/plugins/jsstyped_rebar) with configuration:
===>    {pre_hooks, []}.
===> run_hooks("c:/Users/alex0/Documents/Sources/InnoChain/innobpapi/_build/default/plugins/jsstyped_rebar", pre_hooks, erlc_compile) -> no hooks defined

===> Setting paths to [deps]
===> Analyzing applications...
===> Uncaught error in rebar_core. Run with DIAGNOSTIC=1 to see stacktrace or consult rebar3.crashdump
===> Uncaught error: undef
===> When submitting a bug report, please include the output of `rebar3 report "your command"`

Expected behaviour

Not crashing, probably -- this is such a weird issue.

paulo-ferraz-oliveira commented 3 years ago

@ElectronicRU, could you try and use https://github.com/erlang/rebar3/pull/2542 to see if it solves your issue? Thanks.

ferd commented 3 years ago

The other common source of issues around these things ("needing a second compile and then it works") is that the dep name as declared in {deps, [{DepName, ...}]} does not match the actual OTP application name.

ElectronicRU commented 3 years ago

Huh - but in this case, it's a plugin with empty deps. Could the issue be that the plugin app doesn't depend on rebar, somehow? I'll check.

ferd commented 3 years ago

The plugin app should not depend on rebar. Do check if the plugin itself is misnamed.

ElectronicRU commented 3 years ago

Doesn't seem to be, no.

ferd commented 3 years ago

Is there any way to reproduce this? There seems to be reliance on a private plugin, I'm not sure there's much we can do at this point.

ElectronicRU commented 3 years ago

Unfortunately, yeah :( I think this can be closed for now, I wasn't able to find a smaller breaking example. Once I negotiate releasing the plugin code to open source land, I'll reopen this.

erlang / rebar3