andersju / webbkoll

An online tool that checks how a website is doing with regards to privacy
MIT License
266 stars 27 forks source link

Issue when using https #20

Closed laxmucas closed 4 years ago

laxmucas commented 4 years ago

When I use https, a "Server internal error" occurs after the domain entry. I'm running in Production Mode. I get the following error message in the console:

17:50:14.145 [error] Ranch listener WebbkollWeb.Endpoint.HTTPS, connection process #PID<0.636.0>, stream 15 had its request process #PID<0.644.0> exit with reason {{%ArgumentError{message: "argument error"}, [{:re, :run, ["http://searchdomain.com", "^(?:(?:https?|ftp)://)(?:\\S+(?::\\S*)?@)?(?:(?!(?:10|127)(?:\\.\\d{1,3}){3})(?!(?:169\\.254|192\\.168)(?:\\.\\d{1,3}){2})(?!172\\.(?:1[6-9]|2\\d|3[0-1])(?:\\.\\d{1,3}){2})(?:[1-9]\\d?|1\\d\\d|2[01]\\d|22[0-3])(?:\\.(?:1?\\d{1,2}|2[0-4]\\d|25[0-5])){2}(?:\\.(?:[1-9]\\d?|1\\d\\d|2[0-4]\\d|25[0-4]))|(?:(?:[a-z\\x{00a1}-\\x{ffff}0-9]-*)*[a-z\\x{00a1}-\\x{ffff}0-9]+)(?:\\.(?:[a-z\\x{00a1}-\\x{ffff}0-9]-*)*[a-z\\x{00a1}-\\x{ffff}0-9]+)*(?:\\.(?:[a-z\\x{00a1}-\\x{ffff}]{2,}))\\.?)(?::\\d{2,5})?(?:[/?#]\\S*)?$", [capture: :none]], []}, {Regex, :safe_run, 3, [file: 'lib/regex.ex', line: 470]}, {Regex, :match?, 2, [file: 'lib/regex.ex', line: 269]}, {WebbkollWeb.SiteController, :validate_url, 2, [file: 'lib/webbkoll_web/controllers/site_controller.ex', line: 161]}, {WebbkollWeb.SiteController, :phoenix_controller_pipeline, 2, [file: 'lib/webbkoll_web/controllers/site_controller.ex', line: 1]}, {WebbkollWeb.Endpoint, :instrument, 4, [file: 'arams: %Plug.Conn.Unfetched{aspect: :body_params}, cookies: %Plug.Conn.Unfetched{aspect: :cookies}, halted: false, host: "mydomain.com", method: "GET", owner: #PID<0.644.0>, params: %Plug.Conn.Unfetched{aspect: :params}, path_info: ["de", "check"], path_params: %{}, port: 443, private: %{}, query_params: %Plug.Conn.Unfetched{aspect: :query_params}, query_string: "url=searchdomain.com", remote_ip: {46, 59, 209, 220}, req_cookies: %Plug.Conn.Unfetched{aspect: :cookies}, req_headers: [{"accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"}, {"accept-encoding", "gzip, deflate, br"}, {"accept-language", "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"}, {"sec-fetch-mode", "navigate"}, {"sec-fetch-site", "same-origin"}, {"sec-fetch-user", "?1"}, {"upgrade-insecure-requests", "1"}, {"user-agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36"}], request_path: "/de/check", resp_body: nil, resp_cookies: %{}, resp_headers: [{"cache-control", "max-age=0, private, must-revalidate"}], scheme: :https, script_name: [], secret_key_base: nil, state: :unset, status: nil}, []]}} and stacktrace [{Phoenix.Endpoint.Cowboy2Handler, :init, 2, [file: 'lib/phoenix/endpoint/cowboy2_handler.ex', line: 43]}, {:cowboy_handler, :execute, 2, [file: '/home/max/webbkoll/webbkoll-frontend/deps/cowboy/src/cowboy_handler.erl', line: 41]}, {:cowboy_stream_h, :execute, 3, [file: '/home/max/webbkoll/webbkoll-frontend/deps/cowboy/src/cowboy_stream_h.erl', line: 296]}, {:cowboy_stream_h, :request_process, 3, [file: '/home/max/webbkoll/webbkoll-frontend/deps/cowbonfetched{aspect: :body_params}, cookies: %Plug.Conn.Unfetched{aspect: :cookies}, halted: false, host: "mydomain.com", method: "GET", owner: #PID<0.644.0>, params: %Plug.Conn.Unfetched{aspect: :params}, path_info: ["de", "check"], path_params: %{}, port: 443, private: %{}, query_params: %Plug.Conn.Unfetched{aspect: :query_params}, query_string: "url=searchdomain.com", remote_ip: {46, 59, 209, 220}, req_cookies: %Plug.Conn.Unfetched{aspect: :cookies}, req_headers: [{"accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"}, {"accept-encoding", "gzip, deflate, br"}, {"accept-language", "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"}, {"sec-fetch-mode", "navigate"}, {"sec-fetch-site", "same-origin"}, {"sec-fetch-user", "?1"}, {"upgrade-insecure-requests", "1"}, {"user-agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36"}], request_path: "/de/check", resp_body: nil, resp_cookies: %{}, resp_headers: [{"cache-control", "max-age=0, private, must-revalidate"}], scheme: :https, script_name: [], secret_key_base: nil, state: :unset, status: nil}, []]}} and stacktrace [{Phoenix.Endpoint.Cowboy2Handler, :init, 2, [file: 'lib/phoenix/endpoint/cowboy2_handler.ex', line: 43]}, {:cowboy_handler, :execute, 2, [file: '/home/max/webbkoll/webbkoll-frontend/deps/cowboy/src/cowboy_handler.erl', line: 41]}, {:cowboy_stream_h, :execute, 3, [file: '/home/max/webbkoll/webbkoll-frontend/deps/cowboy/src/cowboy_stream_h.erl', line: 296]}, {:cowboy_stream_h, :request_process, 3, [file: '/home/max/webbkoll/webbkoll-frontend/deps/cowboy/src/cowboy_stream_h.erl', line: 274]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}] file: 'lib/webbkoll_web/endpoint.ex', line: 1]}, {Phoenix.Router, :__call__, 1, [file: 'lib/phoenix/router.ex', line: 275]}, {WebbkollWeb.Endpoint, :plug_builder_call, 2, [file: 'lib/webbkoll_web/endpoint.ex', line: 1]}]}, {WebbkollWeb.Endpoint, :call, [%Plug.Conn{adapter: {Plug.Cowboy.Conn, :...}, assigns: %{}, before_send: [], body_params: %Plug.Conn.Unfetched{aspect: :body_params}, cookies: %Plug.Conn.Unfetched{aspect: :cookies}, halted: false, host: "mydomain.com", method: "GET", owner: #PID<0.644.0>, params: %Plug.Conn.Unfetched{aspect: :params}, path_info: ["de", "check"], path_params: %{}, port: 443, private: %{}, query_params: %Plug.Conn.Unfetched{aspect: :query_params}, query_string: "url=searchdomain.com", remote_ip: {46, 59, 209, 220}, req_cookies: %Plug.Conn.Unfetched{aspect: :cookies}, req_headers: [{"accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"}, {"accept-encoding", "gzip, deflate, br"}, {"accept-language", "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"}, {"sec-fetch-mode", "navigate"}, {"sec-fetch-site", "same-origin"}, {"sec-fetch-user", "?1"}, {"upgrade-insecure-requests", "1"}, {"user-agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36"}], request_path: "/de/check", resp_body: nil, resp_cookies: %{}, resp_headers: [{"cache-control", "max-age=0, private, must-revalidate"}], scheme: :https, script_name: [], secret_key_base: nil, state: :unset, status: nil}, []]}} and stacktrace [{Phoenix.Endpoint.Cowboy2Handler, :init, 2, [file: 'lib/phoenix/endpoint/cowboy2_handler.ex', line: 43]}, {:cowboy_handler, :execute, 2, [file: '/home/max/webbkoll/webbkoll-frontend/deps/cowboy/src/cowboy_handler.erl', line: 41]}, {:cowboy_stream_h, :execute, 3, [file: '/home/max/webbkoll/webbkoll-frontend/deps/cowboy/src/cowboy_stream_h.erl', line: 296]}, {:cowboy_stream_h, :request_process, 3, [file: '/home/max/webbkoll/webbkoll-frontend/deps/cowboy/src/cowboy_stream_h.erl', line: 274]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}] ine: 161]}, {WebbkollWeb.SiteController, :phoenix_controller_pipeline, 2, [file: 'lib/webbkoll_web/controllers/site_controller.ex', line: 1]}, {WebbkollWeb.Endpoint, :instrument, 4, [file: 'lib/webbkoll_web/endpoint.ex', line: 1]}, {Phoenix.Router, :__call__, 1, [file: 'lib/phoenix/router.ex', line: 275]}, {WebbkollWeb.Endpoint, :plug_builder_call, 2, [file: 'lib/webbkoll_web/endpoint.ex', line: 1]}]}, {WebbkollWeb.Endpoint, :call, [%Plug.Conn{adapter: {Plug.Cowboy.Conn, :...}, assigns: %{}, before_send: [], body_params: %Plug.Conn.Unfetched{aspect: :body_params}, cookies: %Plug.Conn.Unfetched{aspect: :cookies}, halted: false, host: "mydomain.com", method: "GET", owner: #PID<0.644.0>, params: %Plug.Conn.Unfetched{aspect: :params}, path_info: ["de", "check"], path_params: %{}, port: 443, private: %{}, query_params: %Plug.Conn.Unfetched{aspect: :query_params}, query_string: "url=searchdomain.com", remote_ip: {46, 59, 209, 220}, req_cookies: %Plug.Conn.Unfetched{aspect: :cookies}, req_headers: [{"accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"}, {"accept-encoding", "gzip, deflate, br"}, {"accept-language", "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"}, {"sec-fetch-mode", "navigate"}, {"sec-fetch-site", "same-origin"}, {"sec-fetch-user", "?1"}, {"upgrade-insecure-requests", "1"}, {"user-agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36"}], request_path: "/de/check", resp_body: nil, resp_cookies: %{}, resp_headers: [{"cache-control", "max-age=0, private, must-revalidate"}], scheme: :https, script_name: [], secret_key_base: nil, state: :unset, status: nil}, []]}} and stacktrace [{Phoenix.Endpoint.Cowboy2Handler, :init, 2, [file: 'lib/phoenix/endpoint/cowboy2_handler.ex', line: 43]}, {:cowboy_handler, :execute, 2, [file: '/home/max/webbkoll/webbkoll-frontend/deps/cowboy/src/cowboy_handler.erl', line: 41]}, {:cowboy_stream_h, :execute, 3, [file: '/home/max/webbkoll/webbkoll-frontend/deps/cowboy/src/cowboy_stream_h.erl', line: 296]}, {:cowboy_stream_h, :request_process, 3, [file: '/home/max/webbkoll/webbkoll-frontend/deps/cowboy/src/cowboy_stream_h.erl', line: 274]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}] ler, :validate_url, 2, [file: 'lib/webbkoll_web/controllers/site_controller.ex', line: 161]}, {WebbkollWeb.SiteController, :phoenix_controller_pipeline, 2, [file: 'lib/webbkoll_web/controllers/site_controller.ex', line: 1]}, {WebbkollWeb.Endpoint, :instrument, 4, [file: 'lib/webbkoll_web/endpoint.ex', line: 1]}, {Phoenix.Router, :__call__, 1, [file: 'lib/phoenix/router.ex', line: 275]}, {WebbkollWeb.Endpoint, :plug_builder_call, 2, [file: 'lib/webbkoll_web/endpoint.ex', line: 1]}]}, {WebbkollWeb.Endpoint, :call, [%Plug.Conn{adapter: {Plug.Cowboy.Conn, :...}, assigns: %{}, before_send: [], body_params: %Plug.Conn.Unfetched{aspect: :body_params}, cookies: %Plug.Conn.Unfetched{aspect: :cookies}, halted: false, host: "mydomain.com", method: "GET", owner: #PID<0.644.0>, params: %Plug.Conn.Unfetched{aspect: :params}, path_info: ["de", "check"], path_params: %{}, port: 443, private: %{}, query_params: %Plug.Conn.Unfetched{aspect: :query_params}, query_string: "url=searchdomain.com", remote_ip: {46, 59, 209, 220}, req_cookies: %Plug.Conn.Unfetched{aspect: :cookies}, req_headers: [{"accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"}, {"accept-encoding", "gzip, deflate, br"}, {"accept-language", "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"}, {"sec-fetch-mode", "navigate"}, {"sec-fetch-site", "same-origin"}, {"sec-fetch-user", "?1"}, {"upgrade-insecure-requests", "1"}, {"user-agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36"}], request_path: "/de/check", resp_body: nil, resp_cookies: %{}, resp_headers: [{"cache-control", "max-age=0, private, must-revalidate"}], scheme: :https, script_name: [], secret_key_base: nil, state: :unset, status: nil}, []]}} and stacktrace [{Phoenix.Endpoint.Cowboy2Handler, :init, 2, [file: 'lib/phoenix/endpoint/cowboy2_handler.ex', line: 43]}, {:cowboy_handler, :execute, 2, [file: '/home/max/webbkoll/webbkoll-frontend/deps/cowboy/src/cowboy_handler.erl', line: 41]}, {:cowboy_stream_h, :execute, 3, [file: '/home/max/webbkoll/webbkoll-frontend/deps/cowboy/src/cowboy_stream_h.erl', line: 296]}, {:cowboy_stream_h, :request_process, 3, [file: '/home/max/webbkoll/webbkoll-frontend/deps/cowboy/src/cowboy_stream_h.erl', line: 274]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}] 0-9]-*)*[a-z\\x{00a1}-\\x{ffff}0-9]+)*(?:\\.(?:[a-z\\x{00a1}-\\x{ffff}]{2,}))\\.?)(?::\\d{2,5})?(?:[/?#]\\S*)?$", [capture: :none]], []}, {Regex, :safe_run, 3, [file: 'lib/regex.ex', line: 470]}, {Regex, :match?, 2, [file: 'lib/regex.ex', line: 269]}, {WebbkollWeb.SiteController, :validate_url, 2, [file: 'lib/webbkoll_web/controllers/site_controller.ex', line: 161]}, {WebbkollWeb.SiteController, :phoenix_controller_pipeline, 2, [file: 'lib/webbkoll_web/controllers/site_controller.ex', line: 1]}, {WebbkollWeb.Endpoint, :instrument, 4, [file: 'lib/webbkoll_web/endpoint.ex', line: 1]}, {Phoenix.Router, :__call__, 1, [file: 'lib/phoenix/router.ex', line: 275]}, {WebbkollWeb.Endpoint, :plug_builder_call, 2, [file: 'lib/webbkoll_web/endpoint.ex', line: 1]}]}, {WebbkollWeb.Endpoint, :call, [%Plug.Conn{adapter: {Plug.Cowboy.Conn, :...}, assigns: %{}, before_send: [], body_params: %Plug.Conn.Unfetched{aspect: :body_params}, cookies: %Plug.Conn.Unfetched{aspect: :cookies}, halted: false, host: "mydomain.com", method: "GET", owner: #PID<0.644.0>, params: %Plug.Conn.Unfetched{aspect: :params}, path_info: ["de", "check"], path_params: %{}, port: 443, private: %{}, query_params: %Plug.Conn.Unfetched{aspect: :query_params}, query_string: "url=searchdomain.com", remote_ip: {46, 59, 209, 220}, req_cookies: %Plug.Conn.Unfetched{aspect: :cookies}, req_headers: [{"accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"}, {"accept-encoding", "gzip, deflate, br"}, {"accept-language", "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"}, {"sec-fetch-mode", "navigate"}, {"sec-fetch-site", "same-origin"}, {"sec-fetch-user", "?1"}, {"upgrade-insecure-requests", "1"}, {"user-agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36"}], request_path: "/de/check", resp_body: nil, resp_cookies: %{}, resp_headers: [{"cache-control", "max-age=0, private, must-revalidate"}], scheme: :https, script_name: [], secret_key_base: nil, state: :unset, status: nil}, []]}} and stacktrace [{Phoenix.Endpoint.Cowboy2Handler, :init, 2, [file: 'lib/phoenix/endpoint/cowboy2_handler.ex', line: 43]}, {:cowboy_handler, :execute, 2, [file: '/home/max/webbkoll/webbkoll-frontend/deps/cowboy/src/cowboy_handler.erl', line: 41]}, {:cowboy_stream_h, :execute, 3, [file: '/home/max/webbkoll/webbkoll-frontend/deps/cowboy/src/cowboy_stream_h.erl', line: 296]}, {:cowboy_stream_h, :request_process, 3, [file: '/home/max/webbkoll/webbkoll-frontend/deps/cowboy/src/cowboy_stream_h.erl', line: 274]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}]

Can you say what i did wrong? Everything works fine when im using http. The certificate also works without any problems.

andersju commented 4 years ago

Strange. I haven't used HTTPS directly in Webbkoll myself (I let nginx handle that). Just to be clear, you've configured Webbkoll (or Phoenix, rather) to use TLS/SSL, and it works until you submit the form on the front page? What version of Elixir and Erlang?

laxmucas commented 4 years ago

Thx for your response!

Elixir 1.9.1 and Erlang 22.1.1 (it says Elixir 1.9.1 compiled with Erlang/OTP 21).

How did you managed to let nginx handle the ssl? Is it possible to run Webbkoll as a nginx/apache webserver?

andersju commented 4 years ago

You can use nginx/Apache as a reverse proxy. Have nginx listen to port 80 and 443 (and take care of SSL) and let it forward things to Webbkoll (which might be running on port 4000 or something). Then Webbkoll doesn't need to know anything about SSL - you'd use the normal HTTP endpoint.

This is a common setup and nothing specific to Phoenix. Here are some guides: