Backend in Docker crash after build on latest patch

walterzhu29 commented 1 month ago

Description

backend           | ERROR! Config provider Config.Reader failed with:
backend           | ** (Code.LoadError) could not load /app/releases/6.2.0/config_helper.exs. Reason: enoent
backend           |     (elixir 1.14.5) lib/code.ex:1826: Code.find_file!/2
backend           |     (elixir 1.14.5) lib/code.ex:1199: Code.eval_file/2
backend           |     (stdlib 3.17) erl_eval.erl:685: :erl_eval.do_apply/6
backend           |     (stdlib 3.17) erl_eval.erl:123: :erl_eval.exprs/5
backend           |     (elixir 1.14.5) src/elixir.erl:297: :elixir.eval_forms/4
backend           |     (elixir 1.14.5) lib/module/parallel_checker.ex:110: Module.ParallelChecker.verify/1
backend           |     (elixir 1.14.5) lib/code.ex:425: Code.validated_eval_string/3
backend           |     (elixir 1.14.5) lib/config.ex:288: Config.__eval__!/3
backend           | 
backend           | {"init terminating in do_boot",{#{'__exception__'=>true,'__struct__'=>'Elixir.Code.LoadError',file=><<"/app/releases/6.2.0/config_helper.exs">>,message=><<"could not load /app/releases/6.2.0/config_helper.exs. Reason: enoent">>,reason=>enoent},[{'Elixir.Code','find_file!',2,[{file,"lib/code.ex"},{line,1826}]},{'Elixir.Code',eval_file,2,[{file,"lib/code.ex"},{line,1199}]},{erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,685}]},{erl_eval,exprs,5,[{file,"erl_eval.erl"},{line,123}]},{elixir,eval_forms,4,[{file,"src/elixir.erl"},{line,297}]},{'Elixir.Module.ParallelChecker',verify,1,[{file,"lib/module/parallel_checker.ex"},{line,110}]},{'Elixir.Code',validated_eval_string,3,[{file,"lib/code.ex"},{line,425}]},{'Elixir.Config','__eval__!',3,[{file,"lib/config.ex"},{line,288}]}]}}
backend           | init terminating in do_boot ({,[{Elixir.Code,find_file!,2,[{_},{_}]},{Elixir.Code,eval_file,2,[{_},{_}]},{erl_eval,do_apply,6,[{_},{_}]},{erl_eval,exprs,5,[{_},{_}]},{elixir,eval_forms,4,[{_},{_}]},{Elixir.Module.ParallelChecker,verify,1,[{_},{_}]},{Elixir.Code,validated_eval_string,3,[{_},{_}]},{Elixir.Config,__eval__!,3,[{_},{_}]}]})
backend           | 
backend           | Crash dump is being written to: erl_crash.dump...done

Backend was working good, recently rebuild all images and keep getting backend crash.

Type of the installation

Docker-compose

Type of the JSON RPC archive node

Geth

Type of the chain

L2

Link to the page

No response

Steps to reproduce

No response

Backend version

v6.5.0-beta-59-g88394ff6fb

Frontend version

No response

Elixir & Erlang/OTP versions

Elixir 1.14.5

Operating system

No response

Additional information

No response

Whickey344 commented 1 month ago

Why did this happen

walterzhu29 commented 1 month ago

More info for the backend image [ { "Id": "sha256:f744f060dac3186eb28dc4f6af96638dd68854ef132f03977abdb21480eeeb4f", "RepoTags": [ "blockscout/blockscout-optimism:latest" ], "RepoDigests": [ "blockscout/blockscout-optimism@sha256:418acf43922eb32c6c8fc5f10219362587f9970a1e9cc60fdeefd11159a31e30" ], "Parent": "", "Comment": "buildkit.dockerfile.v0", "Created": "2024-05-30T11:50:03.623762285Z",

walterzhu29 commented 1 month ago

The issue happens on image blockscout/blockscout-optimism:latest that released 3 days ago, https://hub.docker.com/layers/blockscout/blockscout-optimism/latest/images/sha256-1b48cce27e9deba5f65a07c49df85ecc12f5b14d2ac9bf3fc41c0b701d7100c4?context=explore

After I downgrade the version back to 6.5.0, the crash issue was gone.

avenbreaks commented 1 month ago

The issue happens on image blockscout/blockscout-optimism:latest that released 3 days ago, https://hub.docker.com/layers/blockscout/blockscout-optimism/latest/images/sha256-1b48cce27e9deba5f65a07c49df85ecc12f5b14d2ac9bf3fc41c0b701d7100c4?context=explore

After I downgrade the version back to 6.5.0, the crash issue was gone.

solve this issue? i got same issue. use production-optimism branch, so just change branch v6.5.0 ?

avenbreaks commented 1 month ago

The issue happens on image blockscout/blockscout-optimism:latest that released 3 days ago, https://hub.docker.com/layers/blockscout/blockscout-optimism/latest/images/sha256-1b48cce27e9deba5f65a07c49df85ecc12f5b14d2ac9bf3fc41c0b701d7100c4?context=explore After I downgrade the version back to 6.5.0, the crash issue was gone.

solve this issue? i got same issue. use production-optimism branch, so just change branch v6.5.0 ?

v6.5.0 i got logs error but explorer running

** (Postgrex.Error) ERROR 42703 (undefined_column) column s0.implementation_name does not exist

    query: SELECT s0."id", s0."name", s0."compiler_version", s0."optimization", s0."contract_source_code", s0."constructor_arguments", s0."evm_version", s0."optimization_runs", s0."external_libraries", s0."abi", s0."verified_via_sourcify", s0."verified_via_eth_bytecode_db", s0."verified_via_verifier_alliance", s0."partially_verified", s0."file_path", s0."is_vyper_contract", s0."is_changed_bytecode", s0."bytecode_checked_at", s0."contract_code_md5", s0."implementation_name", s0."compiler_settings", s0."implementation_fetched_at", s0."implementation_address_hash", s0."license_type", s0."certified", s0."address_hash", s0."inserted_at", s0."updated_at", s0."address_hash" FROM "smart_contracts" AS s0 WHERE (s0."address_hash" = $1)
    (ecto_sql 3.11.1) lib/ecto/adapters/sql.ex:1054: Ecto.Adapters.SQL.raise_sql_call_error/1
    (ecto_sql 3.11.1) lib/ecto/adapters/sql.ex:952: Ecto.Adapters.SQL.execute/6
    (ecto 3.11.2) lib/ecto/repo/queryable.ex:232: Ecto.Repo.Queryable.execute/4
    (ecto 3.11.2) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
    (ecto 3.11.2) lib/ecto/repo/preloader.ex:284: Ecto.Repo.Preloader.fetch_query/8
    (elixir 1.14.5) lib/task/supervised.ex:89: Task.Supervised.invoke_mfa/2
    (elixir 1.14.5) lib/task/supervised.ex:34: Task.Supervised.reply/4
    (stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Function: &:erlang.apply/2
    Args: [#Function<10.14054386/1 in Ecto.Repo.Preloader.maybe_pmap/3>, [#Function<21.14054386/1 in Ecto.Repo.Preloader.prepare_queries/6>]]
2024-06-03T05:33:38.299 [error] Ranch protocol #PID<0.23376.0> of listener BlockScoutWeb.Endpoint.HTTP (connection #PID<0.23375.0>, stream id 1) terminated

walterzhu29 commented 1 month ago

The issue happens on image blockscout/blockscout-optimism:latest that released 3 days ago, https://hub.docker.com/layers/blockscout/blockscout-optimism/latest/images/sha256-1b48cce27e9deba5f65a07c49df85ecc12f5b14d2ac9bf3fc41c0b701d7100c4?context=explore After I downgrade the version back to 6.5.0, the crash issue was gone.

solve this issue? i got same issue. use production-optimism branch, so just change branch v6.5.0 ?

v6.5.0 i got logs error but explorer running

** (Postgrex.Error) ERROR 42703 (undefined_column) column s0.implementation_name does not exist

    query: SELECT s0."id", s0."name", s0."compiler_version", s0."optimization", s0."contract_source_code", s0."constructor_arguments", s0."evm_version", s0."optimization_runs", s0."external_libraries", s0."abi", s0."verified_via_sourcify", s0."verified_via_eth_bytecode_db", s0."verified_via_verifier_alliance", s0."partially_verified", s0."file_path", s0."is_vyper_contract", s0."is_changed_bytecode", s0."bytecode_checked_at", s0."contract_code_md5", s0."implementation_name", s0."compiler_settings", s0."implementation_fetched_at", s0."implementation_address_hash", s0."license_type", s0."certified", s0."address_hash", s0."inserted_at", s0."updated_at", s0."address_hash" FROM "smart_contracts" AS s0 WHERE (s0."address_hash" = $1)
    (ecto_sql 3.11.1) lib/ecto/adapters/sql.ex:1054: Ecto.Adapters.SQL.raise_sql_call_error/1
    (ecto_sql 3.11.1) lib/ecto/adapters/sql.ex:952: Ecto.Adapters.SQL.execute/6
    (ecto 3.11.2) lib/ecto/repo/queryable.ex:232: Ecto.Repo.Queryable.execute/4
    (ecto 3.11.2) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
    (ecto 3.11.2) lib/ecto/repo/preloader.ex:284: Ecto.Repo.Preloader.fetch_query/8
    (elixir 1.14.5) lib/task/supervised.ex:89: Task.Supervised.invoke_mfa/2
    (elixir 1.14.5) lib/task/supervised.ex:34: Task.Supervised.reply/4
    (stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Function: &:erlang.apply/2
    Args: [#Function<10.14054386/1 in Ecto.Repo.Preloader.maybe_pmap/3>, [#Function<21.14054386/1 in Ecto.Repo.Preloader.prepare_queries/6>]]
2024-06-03T05:33:38.299 [error] Ranch protocol #PID<0.23376.0> of listener BlockScoutWeb.Endpoint.HTTP (connection #PID<0.23375.0>, stream id 1) terminated

Gotta check you docker image version instead of branch tag

walterzhu29 commented 1 month ago

The issue happens on image blockscout/blockscout-optimism:latest that released 3 days ago, https://hub.docker.com/layers/blockscout/blockscout-optimism/latest/images/sha256-1b48cce27e9deba5f65a07c49df85ecc12f5b14d2ac9bf3fc41c0b701d7100c4?context=explore After I downgrade the version back to 6.5.0, the crash issue was gone.

solve this issue? i got same issue. use production-optimism branch, so just change branch v6.5.0 ?

yes manually downgrade the image back to 6.5.0 solve the issue.

jeremyccrane commented 1 month ago

I'm having the same issue, but with 6.5.0 and 6.6.0.

ERROR! Config provider Config.Reader failed with:
** (Code.LoadError) could not load /app/releases/6.6.0/config_helper.exs. Reason: enoent
    (elixir 1.14.5) lib/code.ex:1826: Code.find_file!/2
    (elixir 1.14.5) lib/code.ex:1199: Code.eval_file/2
    (stdlib 3.17) erl_eval.erl:685: :erl_eval.do_apply/6
    (stdlib 3.17) erl_eval.erl:123: :erl_eval.exprs/5
    (elixir 1.14.5) src/elixir.erl:297: :elixir.eval_forms/4
    (elixir 1.14.5) lib/module/parallel_checker.ex:110: Module.ParallelChecker.verify/1
    (elixir 1.14.5) lib/code.ex:425: Code.validated_eval_string/3
    (elixir 1.14.5) lib/config.ex:288: Config.__eval__!/3

Here is the line from my Dockerfile copying the file over:

COPY --from=builder /app/config/config_helper.exs /app/releases/${RELEASE_VERSION}/config_helper.exs

How do we downgrade the docker image to tag or branch that works?

GimluCom commented 1 month ago

@jeremyccrane Could you share the output prior to the error? Is this also optimism or another chain type?

jeremyccrane commented 1 month ago

Yes, here it is:

➜  blockscout git:(v6.5.0-branch) docker run -it blockscout:v6.5.0              
ERROR! Config provider Config.Reader failed with:
** (Code.LoadError) could not load /app/releases/6.5.0/config_helper.exs. Reason: enoent
    (elixir 1.14.5) lib/code.ex:1826: Code.find_file!/2
    (elixir 1.14.5) lib/code.ex:1199: Code.eval_file/2
    (stdlib 3.17) erl_eval.erl:685: :erl_eval.do_apply/6
    (stdlib 3.17) erl_eval.erl:123: :erl_eval.exprs/5
    (elixir 1.14.5) src/elixir.erl:297: :elixir.eval_forms/4
    (elixir 1.14.5) lib/module/parallel_checker.ex:110: Module.ParallelChecker.verify/1
    (elixir 1.14.5) lib/code.ex:425: Code.validated_eval_string/3
    (elixir 1.14.5) lib/config.ex:288: Config.__eval__!/3

{"init terminating in do_boot",{#{'__exception__'=>true,'__struct__'=>'Elixir.Code.LoadError',file=><<"/app/releases/6.5.0/config_helper.exs">>,message=><<"could not load /app/releases/6.5.0/config_helper.exs. Reason: enoent">>,reason=>enoent},[{'Elixir.Code','find_file!',2,[{file,"lib/code.ex"},{line,1826}]},{'Elixir.Code',eval_file,2,[{file,"lib/code.ex"},{line,1199}]},{erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,685}]},{erl_eval,exprs,5,[{file,"erl_eval.erl"},{line,123}]},{elixir,eval_forms,4,[{file,"src/elixir.erl"},{line,297}]},{'Elixir.Module.ParallelChecker',verify,1,[{file,"lib/module/parallel_checker.ex"},{line,110}]},{'Elixir.Code',validated_eval_str=ERROR REPORT==== 8-Jun-2024::16:43:56.585486 ===
Error in process <0.9.0> on node blockscout@773b614605c9 with exit value:
{#{'__exception__' => true,'__struct__' => 'Elixir.Code.LoadError',
   file => <<"/app/releases/6.5.0/config_helper.exs">>,
   message =>
       <<"could not load /app/releases/6.5.0/config_helper.exs. Reason: enoent">>,
   reason => enoent},
 [{'Elixir.Code','find_file!',2,[{file,"lib/code.ex"},{line,1826}]},
  {'Elixir.Code',eval_file,2,[{file,"lib/code.ex"},{line,1199}]},
  {erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,685}]},
  {erl_eval,exprs,5,[{file,"erl_eval.erl"},{line,123}]},
  {elixir,eval_forms,4,[{file,"src/elixir.erl"},{line,297}]},
  {'Elixir.Module.ParallelChecker',verify,1,
                                   [{file,"lib/module/parallel_checker.ex"},
                                    {line,110}]},
  {'Elixir.Code',validated_eval_string,3,[{file,"lib/code.ex"},{line,425}]},
  {'Elixir.Config','__eval__!',3,[{file,"lib/config.ex"},{line,288}]}]}

ing,3,[{file,"lib/code.ex"},{line,425}]},{'Elixir.Config','__eval__!',3,[{file,"lib/config.ex"},{line,288}]}]}}
init terminating in do_boot ({,[{Elixir.Code,find_file!,2,[{_},{_}]},{Elixir.Code,eval_file,2,[{_},{_}]},{erl_eval,do_apply,6,[{_},{_}]},{erl_eval,exprs,5,[{_},{_}]},{elixir,eval_forms,4,[{_},{_}]},{Elixir.Module.ParallelChecker,verify,1,[{_},{_}]},{Elixir.Code,validated_eval_string,3,[{_},{_}]},{Elixir.Config,__eval__!,3,[{_},{_}]}]})

Crash dump is being written to: erl_crash.dump...done

Type of the installation

Dockerfile

Type of the JSON RPC archive node

Geth

Type of the chain

L1 Ethereum

GimluCom commented 1 month ago

@jeremyccrane Please look at https://docs.blockscout.com on how to run Blockscout with docker.

If you want to run a specific version, add this line in your envs/common-blockscout.env file:

DOCKER_TAG=6.6.0

To run the Blockscout backend, you can try this command:

docker compose up -d backend

blockscout / blockscout