operating behind a user auth proxy server

[tobiasdroessler]

I have problems running a phoenix liveview demo behind a proxy server which needs username and Password for Auth.

1) I export the proxy server env export https_proxy=http://{USER}:{PASSW}@{PROXYSERVER}:{PORT}

2) Then I get:

~/phoenixHi/my_app$ mix phx.server
warning: the :gettext compiler is no longer required in your mix.exs.

Please find the following line in your mix.exs and remove the :gettext entry:

    compilers: [..., :gettext, ...] ++ Mix.compilers(),

  (gettext 0.20.0) lib/mix/tasks/compile.gettext.ex:5: Mix.Tasks.Compile.Gettext.run/1
  (mix 1.13.4) lib/mix/task.ex:397: anonymous fn/3 in Mix.Task.run_task/3
  (mix 1.13.4) lib/mix/tasks/compile.all.ex:92: Mix.Tasks.Compile.All.run_compiler/2
  (mix 1.13.4) lib/mix/tasks/compile.all.ex:72: Mix.Tasks.Compile.All.compile/4
  (mix 1.13.4) lib/mix/tasks/compile.all.ex:59: Mix.Tasks.Compile.All.with_logger_app/2
  (mix 1.13.4) lib/mix/tasks/compile.all.ex:36: Mix.Tasks.Compile.All.run/1

[info] Running MyAppWeb.Endpoint with cowboy 2.9.0 at 127.0.0.1:4000 (http)
[debug] Downloading esbuild from https://registry.npmjs.org/esbuild-linux-64/-/esbuild-linux-64-0.14.29.tgz
[debug] Using HTTPS_PROXY: http://<USER>:<PASSWORD>@<PROXYSERVER>:8080
[info] Access MyAppWeb.Endpoint at http://localhost:4000
[error] Task #PID<0.542.0> started from MyAppWeb.Endpoint terminating
** (RuntimeError) couldn't fetch https://registry.npmjs.org/esbuild-linux-64/-/esbuild-linux-64-0.14.29.tgz: {:error, {:could_not_establish_ssl_tunnel, {'HTTP/1.1', 407, 'Proxy Authentication Required'}}}
    (esbuild 0.5.0) lib/esbuild.ex:302: Esbuild.fetch_body!/1
    (esbuild 0.5.0) lib/esbuild.ex:209: Esbuild.install/0
    (esbuild 0.5.0) lib/esbuild.ex:189: Esbuild.install_and_run/2
    (phoenix 1.6.11) lib/phoenix/endpoint/watcher.ex:19: Phoenix.Endpoint.Watcher.watch/2
    (elixir 1.13.4) lib/task/supervised.ex:89: Task.Supervised.invoke_mfa/2
    (stdlib 4.0.1) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Function: &Phoenix.Endpoint.Watcher.watch/2
    Args: ["esbuild", {Esbuild, :install_and_run, [:default, ["--sourcemap=inline", "--watch"]]}] 

The out log HTTPS_PROXY Value is right (but also with user and password included).

I discussed this on the libera IRC channel ... but find no solution. They pointed me here to create an issue.

By the way: Downloading the requested file/package with curl / wget works fine.

[josevalim]

What is your Erlang/OTP version? Can you try Erlang/OTP 25 just in case?

In any case, you can always check the docs at https://hexdocs.pm/esbuild/Esbuild.html for instructions to install it manually in cases like these.

[tobiasdroessler]

thank you for hint ... i will got there later and follow instructions for manual installation.

Erlang Version:

erl --version
Erlang/OTP 25 [erts-13.0.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]

[Nicd]

Seems to me that the issue is that in https://github.com/phoenixframework/esbuild/blob/749fbe6c0d5ded2cfbaa6af65e71a6e2664cc85c/lib/esbuild.ex#L269-L279 only the host and port are picked up from the proxy env var. Userinfo is lost and not passed to the proxy.

[Nicd]

Looking at this answer on SO and its comments, it seems authentication credentials should be parsed from the URL and passed to :httpc.request, they sadly cannot be given to :httpc.set_options/1.

[josevalim]

There is no path in the above but we are discarding the user info. I have pushed a fix to main. @tobiasdroessler, can you please give main a try and let us know if it works?

[tobiasdroessler]

i am new to elixir and phoenix .. This was my first try with a hello world example. so I have no idea how to do this (compile and invoke in my sample project).... sorry. i have interest, but no experience. I ask at the IRC channel... but there was nothing useful.

[josevalim]

No problem! We are glad to help!

@tobiasdroessler open your mix.exs and find :esbuild in there and replace it by this:

{:esbuild, github: "phoenixframework/esbuild"},

And then in your terminal:

mix deps.get
mix escript.install

(remember to remove the :path configuration from your config/config.exs before running the above). It should either download or fail as before!

[tobiasdroessler]

@josevalim I checked your fix in the morning ... but it seems to be not working jet.

i get the following:

<USERNAME>@DR6-C-000AE:~/hellophoenix$ mix phx.server
warning: the :gettext compiler is no longer required in your mix.exs.

Please find the following line in your mix.exs and remove the :gettext entry:

    compilers: [..., :gettext, ...] ++ Mix.compilers(),

  (gettext 0.20.0) lib/mix/tasks/compile.gettext.ex:5: Mix.Tasks.Compile.Gettext.run/1
  (mix 1.13.4) lib/mix/task.ex:397: anonymous fn/3 in Mix.Task.run_task/3
  (mix 1.13.4) lib/mix/tasks/compile.all.ex:92: Mix.Tasks.Compile.All.run_compiler/2
  (mix 1.13.4) lib/mix/tasks/compile.all.ex:72: Mix.Tasks.Compile.All.compile/4
  (mix 1.13.4) lib/mix/tasks/compile.all.ex:59: Mix.Tasks.Compile.All.with_logger_app/2
  (mix 1.13.4) lib/mix/tasks/compile.all.ex:36: Mix.Tasks.Compile.All.run/1

[info] Running HellophoenixWeb.Endpoint with cowboy 2.9.0 at 127.0.0.1:4000 (http)
[debug] Downloading esbuild from https://registry.npmjs.org/esbuild-linux-64/-/esbuild-linux-64-0.14.29.tgz
[debug] Using HTTP_PROXY: http://<USERNAME>:<PASSWORD>@<PROXYSERVER>:8080
[debug] Using HTTPS_PROXY: http://<USERNAME>:<PASSWORD>@<PROXYSERVER>:8080
[info] Access HellophoenixWeb.Endpoint at http://localhost:4000
[error] Task #PID<0.547.0> started from HellophoenixWeb.Endpoint terminating
** (RuntimeError) couldn't fetch https://registry.npmjs.org/esbuild-linux-64/-/esbuild-linux-64-0.14.29.tgz: {:error, {:failed_connect, [{:to_address, {'<USERNAME>:<PASSWORD>@<PROXYSERVER>', 8080}}, {:inet, [:inet], :nxdomain}]}}

You may also install the "esbuild" executable manually, see the docs: https://hexdocs.pm/esbuild

    (esbuild 0.5.0) lib/esbuild.ex:300: Esbuild.fetch_body!/1
    (esbuild 0.5.0) lib/esbuild.ex:209: Esbuild.install/0
    (esbuild 0.5.0) lib/esbuild.ex:189: Esbuild.install_and_run/2
    (phoenix 1.6.12) lib/phoenix/endpoint/watcher.ex:19: Phoenix.Endpoint.Watcher.watch/2
    (elixir 1.13.4) lib/task/supervised.ex:89: Task.Supervised.invoke_mfa/2
    (stdlib 4.0.1) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Function: &Phoenix.Endpoint.Watcher.watch/2
    Args: ["esbuild", {Esbuild, :install_and_run, [:default, ["--sourcemap=inline", "--watch"]]}]
^C
BREAK: (a)bort (A)bort with dump (c)ontinue (p)roc info (i)nfo
       (l)oaded (v)ersion (k)ill (D)b-tables (d)istribution
a

==> the wget download of the file to check credentials was working.

[tobiasdroessler]

is here a update available etc. or could i do something to get it running ?

This week I get a request for a small tool at work .. this would be perfect situation to try the elixir environment.

[tobiasdroessler]

No problem! We are glad to help!

@tobiasdroessler open your mix.exs and find :esbuild in there and replace it by this:

{:esbuild, github: "phoenixframework/esbuild"},

And then in your terminal:

mix deps.get
mix escript.install

(remember to remove the :path configuration from your config/config.exs before running the above). It should either download or fail as before!

I tried one more... but no, the integration at least is not working :(

drtdr@DR6-C-000AE:/mnt/c/Users/drtdr/helloPhx/foo$ mix deps.get
mix escript.install
* Getting esbuild (https://github.com/phoenixframework/esbuild.git)
remote: Enumerating objects: 473, done.
remote: Counting objects: 100% (101/101), done.
remote: Compressing objects: 100% (59/59), done.
remote: Total 473 (delta 54), reused 71 (delta 37), pack-reused 372
origin/HEAD set to main
Resolving Hex dependencies...
Resolution completed in 0.145s
Unchanged:
  castore 1.0.0
  connection 1.1.0
  cowboy 2.9.0
  cowboy_telemetry 0.4.0
  cowlib 2.11.0
  db_connection 2.4.3
  decimal 2.0.0
  ecto 3.9.4
  ecto_sql 3.9.2
  expo 0.3.0
  file_system 0.2.10
  floki 0.34.0
  gettext 0.22.0
  jason 1.4.0
  mime 2.0.3
  phoenix 1.6.15
  phoenix_ecto 4.4.0
  phoenix_html 3.2.0
  phoenix_live_dashboard 0.6.5
  phoenix_live_reload 1.4.1
  phoenix_live_view 0.17.12
  phoenix_pubsub 2.1.1
  phoenix_template 1.0.0
  phoenix_view 2.0.2
  plug 1.14.0
  plug_cowboy 2.6.0
  plug_crypto 1.2.3
  postgrex 0.16.5
  ranch 1.8.0
  swoosh 1.9.1
  telemetry 1.2.1
  telemetry_metrics 0.6.1
  telemetry_poller 1.0.0

drtdr@DR6-C-000AE:/mnt/c/Users/drtdr/helloPhx/foo$ mix escript.install
==> esbuild
Compiling 3 files (.ex)
Generated esbuild app
warning: the :gettext compiler is no longer required in your mix.exs.

Please find the following line in your mix.exs and remove the :gettext entry:

    compilers: [..., :gettext, ...] ++ Mix.compilers(),

  (gettext 0.22.0) lib/mix/tasks/compile.gettext.ex:5: Mix.Tasks.Compile.Gettext.run/1
  (mix 1.14.1) lib/mix/task.ex:421: anonymous fn/3 in Mix.Task.run_task/4
  (mix 1.14.1) lib/mix/tasks/compile.all.ex:92: Mix.Tasks.Compile.All.run_compiler/2
  (mix 1.14.1) lib/mix/tasks/compile.all.ex:72: Mix.Tasks.Compile.All.compile/4
  (mix 1.14.1) lib/mix/tasks/compile.all.ex:59: Mix.Tasks.Compile.All.with_logger_app/2
  (mix 1.14.1) lib/mix/tasks/compile.all.ex:33: Mix.Tasks.Compile.All.run/1

==> foo
Generated foo app

** (Mix) Could not generate escript, please set :main_module in your project configuration (under :escript option) to a module that implements main/1

[josevalim]

If for some reason it still does not work, please follow these steps to have esbuild installed and running from npm: https://hexdocs.pm/esbuild/Esbuild.html#module-esbuild-configuration

[marcotaubmann]

There is no path in the above but we are discarding the user info. I have pushed a fix to main. @tobiasdroessler, can you please give main a try and let us know if it works?

Hi there,

I tried that fix but it didn't work for me. It seems that :httpc wants its proxy authentication credentials as a HttpOption via :httpc.request().

A little rewrite of the fetch_body() function worked for my setup. It also includes a change so that only the one environment variable for the scheme of the requested url is used (either http_proxy or https_proxy).

mix test with no https_proxy, https_proxy without credentials and https_proxy with credentials all passed. Would be happy to make changes if required.

@tobiasdroessler maybe you can try and confirm my version: {:esbuild, "~> 0.6", runtime: Mix.env() == :dev, git: "https://github.com/marcotaubmann/esbuild.git"},

@josevalim may I create a merge request for these changes? https://github.com/phoenixframework/esbuild/compare/main...marcotaubmann:esbuild:main

I could prepare similar changes and a merge request for phoenixframework/tailwind if this one is fine.

[josevalim]

PRs welcome!

[josevalim]

One for tailwind is also welcome, please CC me and remember to run mix format before. Thanks!~ :)

[marcotaubmann]

One for tailwind is also welcome, please CC me and remember to run mix format before. Thanks!~ :)

Comming up. Sorry I missed the formatting. I hope its okay to ask here: When I clone the tailwind repo and run mix format I get many newline changes in 4 different files. I guess these aren't supposed to come up, maybe i should make the changes, run mix format and then discard everything except my changes?

[josevalim]

Formatting everything is also fine!