Update runtime.exs To allow Plausible.Mailer tls false via environment variable

russellballestrini commented 3 months ago

Why

my self-hosted plausible was upgraded and stopped sending email and I tracked down that this is the setting that needs to change. I need to set tls to false since I'm sending email on port 25.

Changes

Proposed Change Description

Context

The Bamboo.SMTPAdapter configuration in the Plausible.Mailer setup currently has a hardcoded tls setting. This setting determines how the adapter handles TLS (Transport Layer Security) when connecting to the SMTP server. The possible values for the tls setting are:

:if_available: Attempt to use TLS if the SMTP server supports it.
:always: Always use TLS for the connection. If the server does not support TLS, the connection will fail.
false: Do not use TLS for the connection.

Problem

The current configuration does not allow for dynamic control over the tls setting based on environment variables. This can be problematic in environments where the TLS requirement may vary, such as development, testing, and production environments.

Solution

The proposed change introduces a new environment variable SMTP_HOST_TLS_ENABLED to dynamically control the tls setting. The tls setting will be determined based on the value of this environment variable, with a default value of :if_available if the variable is not set.

Code Change

The following code snippet shows the proposed change to the Plausible.Mailer configuration:

config :plausible, Plausible.Mailer,
  adapter: Bamboo.SMTPAdapter,
  server: get_var_from_path_or_env(config_dir, "SMTP_HOST_ADDR", "mail"),
  hostname: base_url.host,
  port: get_var_from_path_or_env(config_dir, "SMTP_HOST_PORT", "25"),
  username: get_var_from_path_or_env(config_dir, "SMTP_USER_NAME"),
  password: get_var_from_path_or_env(config_dir, "SMTP_USER_PWD"),
  tls: case get_var_from_path_or_env(config_dir, "SMTP_HOST_TLS_ENABLED", "if_available") do
    "true" -> :always
    "false" -> false
    _ -> :if_available
  end,
  allowed_tls_versions: [:tlsv1, :"tlsv1.1", :"tlsv1.2"],
  ssl: get_var_from_path_or_env(config_dir, "SMTP_HOST_SSL_ENABLED") || false,
  retries: get_var_from_path_or_env(config_dir, "SMTP_RETRIES") || 2,
  no_mx_lookups: get_var_from_path_or_env(config_dir, "SMTP_MX_LOOKUPS_ENABLED") || true,
  ssl_opts: [verify: :verify_none] # This will ignore SSL verification (not recommended for production)

Explanation

The tls setting is now determined by the SMTP_HOST_TLS_ENABLED environment variable.
If SMTP_HOST_TLS_ENABLED is set to "true", the tls setting will be :always.
If SMTP_HOST_TLS_ENABLED is set to "false", the tls setting will be false.
If SMTP_HOST_TLS_ENABLED is not set or has any other value, the tls setting will default to :if_available.

Pull Request Body

Title

Add dynamic control for TLS setting in Bamboo.SMTPAdapter configuration

Description

This pull request introduces a new environment variable SMTP_HOST_TLS_ENABLED to dynamically control the tls setting in the Bamboo.SMTPAdapter configuration for Plausible.Mailer. The tls setting determines how TLS (Transport Layer Security) is handled when connecting to the SMTP server.

Changes

Added a new environment variable SMTP_HOST_TLS_ENABLED.
Updated the Plausible.Mailer configuration to use the SMTP_HOST_TLS_ENABLED environment variable to determine the tls setting.
The tls setting will be:
- :always if SMTP_HOST_TLS_ENABLED is set to "true".
- false if SMTP_HOST_TLS_ENABLED is set to "false".
- :if_available if SMTP_HOST_TLS_ENABLED is not set or has any other value.

Rationale

This change allows for dynamic control over the tls setting based on environment variables, making it easier to configure the application for different environments (development, testing, production) without modifying the source code.

Example Usage

To disable TLS, set the following environment variable in your docker-compose.yml or environment configuration:


      SMTP_HOST_ADDR: "smtp.example.com"
      SMTP_HOST_PORT: 25
      SMTP_USER_NAME: "your_username"
      SMTP_USER_PWD: "your_password"
      SMTP_HOST_SSL_ENABLED: "false"
      SMTP_HOST_TLS_ENABLED: "false" # explicitly disable TLS
      SMTP_RETRIES: 2
      SMTP_MX_LOOKUPS_ENABLED: "true"

To always use TLS, set the following environment variable:


      SMTP_HOST_ADDR: "smtp.example.com"
      SMTP_HOST_PORT: 25
      SMTP_USER_NAME: "your_username"
      SMTP_USER_PWD: "your_password"
      SMTP_HOST_SSL_ENABLED: "true"
      SMTP_HOST_TLS_ENABLED: "true" # always use TLS
      SMTP_RETRIES: 2
      SMTP_MX_LOOKUPS_ENABLED: "true"

If SMTP_HOST_TLS_ENABLED is not set, the tls setting will default to :if_available.

Conclusion

This change enhances the flexibility and configurability of the Plausible.Mailer setup, allowing for better adaptation to different environments and requirements.

I need to disable this option for my email to go out for reset password links.

Tests

[ ] Automated tests have been added
[x] This PR does not require tests

Changelog

[ ] Entry has been added to changelog
[ ] This PR does not make a user-facing change

Documentation

[ ] Docs have been updated
[ ] This change does not need a documentation update

Dark mode

[ ] The UI has been tested both in dark and light mode
[ ] This PR does not change the UI

ruslandoga commented 3 months ago

👋 @russellballestrini

my self-hosted plausible was upgraded and stopped sending email

Quick check, have you tried using Bamboo.Mua?

I need to set tls to false since I'm sending email on port 25.

That's not how it works. SMTP on port 25 still supports and in many cases requires STARTTLS.

russellballestrini commented 3 months ago

👋 @russellballestrini

my self-hosted plausible was upgraded and stopped sending email

Quick check, have you tried using Bamboo.Mua?

I need to set tls to false since I'm sending email on port 25.

That's not how it works. SMTP on port 25 still supports and in many cases requires STARTTLS.

I tried it, also didn't send email so I switched back to default

russellballestrini commented 3 months ago

postfix is on 25 without tls and I route to it from docker using 173.17.0.1

ruslandoga commented 3 months ago

I tried it, also didn't send email so I switched back to default

Would you be able to try again? I would like to find out why it doesn't work. You can try and send the email manually like in the Verify section in https://gist.github.com/ruslandoga/c94ce526231fb77930132aaeda3fc3c9. If it doesn't work, it should return an error.

russellballestrini commented 3 months ago

I tried it, also didn't send email so I switched back to default

Would you be able to try again? I would like to find out why it doesn't work. You can try and send the email manually like in the Verify section in https://gist.github.com/ruslandoga/c94ce526231fb77930132aaeda3fc3c9. If it doesn't work, it should return an error.

sure but just to be clear you want me to test Bamboo.Mua with my patch?

# The email id to use for as from address of all communications from Plausible.
MAILER_ADAPTER=Bamboo.Mua
MAILER_EMAIL=noreply@analytics.example.com
SMTP_HOST_ADDR=172.17.0.1
SMTP_HOST_PORT=25
SMTP_HOST_SSL_ENABLED=false
SMTP_HOST_TLS_ENABLED=false
EXTRA_CONFIG_PATH="/app/my_config.exs"

# my_config.exs

import Config

config :plausible, Plausible.Mailer,
  tls: false

and the error

Interactive Elixir (1.16.0) - press Ctrl+C to exit (type h() ENTER for help)
iex(plausible@c4b28e92e971)1> Application.get_env :plausible, Plausible.Mailer
[
  adapter: Bamboo.Mua,
  auth: [username: nil, password: nil],
  relay: "172.17.0.1",
  port: 25,
  tls: false
]
iex(plausible@c4b28e92e971)2> {:ok, _delivered_email} =
...(plausible@c4b28e92e971)2>   Plausible.Mailer.deliver_now(
...(plausible@c4b28e92e971)2>     Bamboo.Email.new_email(
...(plausible@c4b28e92e971)2>       # pick one of your own mailboxes
...(plausible@c4b28e92e971)2>       to: "<removed>@gmail.com",
...(plausible@c4b28e92e971)2>       from: PlausibleWeb.Email.mailer_email_from(),
...(plausible@c4b28e92e971)2>       text_body: "test"
...(plausible@c4b28e92e971)2>     )
...(plausible@c4b28e92e971)2>   )
** (MatchError) no match of right hand side value: {:error, %Mua.TransportError{reason: {:tls_alert, {:bad_certificate, ~c"TLS client: In state wait_cert_cr at ssl_handshake.erl:2131 generated CLIENT ALERT: Fatal - Bad Certificate\n"}}}}
    (stdlib 5.2) erl_eval.erl:498: :erl_eval.expr/6
    iex:2: (file)

ruslandoga commented 3 months ago

sure but just to be clear you want me to test Bamboo.Mua with my patch?

No, no need for the patch. What you set in my_config.exs would be ignored.

The error means your relay presents an invalid certificate. You can try connecting to it manually with netcat or telnet or openssl to find out more.

Example with Gmail:

$ telnet gmail-smtp-in.l.google.com 25
Trying 64.233.170.27...
Connected to gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP 98e67ed59e1d1-2c939ccfb4dsi2526960a91.0 - gsmtp
EHLO localhost
250-mx.google.com at your service, [110.78.153.105]
250-SIZE 157286400
250-8BITMIME
250-STARTTLS
250-ENHANCEDSTATUSCODES
250 SMTPUTF8
Connection closed by foreign host.

What extensions does your relay return? If it returns STARTTLS, then you can try connecting to it with openssl.

$ openssl s_client -starttls smtp -crlf -connect gmail-smtp-in.l.google.com:25
Connecting to 64.233.170.27
CONNECTED(00000005)
depth=2 C=US, O=Google Trust Services LLC, CN=GTS Root R1
verify return:1
depth=1 C=US, O=Google Trust Services, CN=WR2
verify return:1
depth=0 CN=mx.google.com
verify return:1
---
Certificate chain
 0 s:CN=mx.google.com
   i:C=US, O=Google Trust Services, CN=WR2
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jun 13 16:32:52 2024 GMT; NotAfter: Sep  5 16:32:51 2024 GMT
 1 s:C=US, O=Google Trust Services, CN=WR2
   i:C=US, O=Google Trust Services LLC, CN=GTS Root R1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Dec 13 09:00:00 2023 GMT; NotAfter: Feb 20 14:00:00 2029 GMT
 2 s:C=US, O=Google Trust Services LLC, CN=GTS Root R1
   i:C=BE, O=GlobalSign nv-sa, OU=Root CA, CN=GlobalSign Root CA
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jun 19 00:00:42 2020 GMT; NotAfter: Jan 28 00:00:42 2028 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGwjCCBaqgAwIBAgIQcFG/r2M1dxAJrxJhvnVpCDANBgkqhkiG9w0BAQsFADA7
MQswCQYDVQQGEwJVUzEeMBwGA1UEChMVR29vZ2xlIFRydXN0IFNlcnZpY2VzMQww
CgYDVQQDEwNXUjIwHhcNMjQwNjEzMTYzMjUyWhcNMjQwOTA1MTYzMjUxWjAYMRYw
FAYDVQQDEw1teC5nb29nbGUuY29tMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE
MJuda1+cDYnL6+IaJOIZQlMXgRQpxqsk0vqfwRE8ITRQsKZr1ffNy1X9JnypcoG7
9Y2MwLyIYl+B8RxnMb4vBaOCBK4wggSqMA4GA1UdDwEB/wQEAwIHgDATBgNVHSUE
DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBQsz7qPhWTn56YL
dmac1JwIy9Nj9TAfBgNVHSMEGDAWgBTeGx7teRXUPjckwyG77DQ5bUKyMDBYBggr
BgEFBQcBAQRMMEowIQYIKwYBBQUHMAGGFWh0dHA6Ly9vLnBraS5nb29nL3dyMjAl
BggrBgEFBQcwAoYZaHR0cDovL2kucGtpLmdvb2cvd3IyLmNydDCCAoYGA1UdEQSC
An0wggJ5gg1teC5nb29nbGUuY29tgg9zbXRwLmdvb2dsZS5jb22CEmFzcG14Lmwu
Z29vZ2xlLmNvbYIXYWx0MS5hc3BteC5sLmdvb2dsZS5jb22CF2FsdDIuYXNwbXgu
bC5nb29nbGUuY29tghdhbHQzLmFzcG14LmwuZ29vZ2xlLmNvbYIXYWx0NC5hc3Bt
eC5sLmdvb2dsZS5jb22CGmdtYWlsLXNtdHAtaW4ubC5nb29nbGUuY29tgh9hbHQx
LmdtYWlsLXNtdHAtaW4ubC5nb29nbGUuY29tgh9hbHQyLmdtYWlsLXNtdHAtaW4u
bC5nb29nbGUuY29tgh9hbHQzLmdtYWlsLXNtdHAtaW4ubC5nb29nbGUuY29tgh9h
bHQ0LmdtYWlsLXNtdHAtaW4ubC5nb29nbGUuY29tghhnbXItc210cC1pbi5sLmdv
b2dsZS5jb22CHWFsdDEuZ21yLXNtdHAtaW4ubC5nb29nbGUuY29tgh1hbHQyLmdt
ci1zbXRwLWluLmwuZ29vZ2xlLmNvbYIdYWx0My5nbXItc210cC1pbi5sLmdvb2ds
ZS5jb22CHWFsdDQuZ21yLXNtdHAtaW4ubC5nb29nbGUuY29tgg1teDEuc210cC5n
b29ngg1teDIuc210cC5nb29ngg1teDMuc210cC5nb29ngg1teDQuc210cC5nb29n
ghVhc3BteDIuZ29vZ2xlbWFpbC5jb22CFWFzcG14My5nb29nbGVtYWlsLmNvbYIV
YXNwbXg0Lmdvb2dsZW1haWwuY29tghVhc3BteDUuZ29vZ2xlbWFpbC5jb22CEWdt
ci1teC5nb29nbGUuY29tMBMGA1UdIAQMMAowCAYGZ4EMAQIBMDYGA1UdHwQvMC0w
K6ApoCeGJWh0dHA6Ly9jLnBraS5nb29nL3dyMi83NXI0WnlBM3ZBMC5jcmwwggEC
BgorBgEEAdZ5AgQCBIHzBIHwAO4AdQB2/4g/Crb7lVHCYcz1h7o0tKTNuyncaEIK
n+ZnTFo6dAAAAZASqNw5AAAEAwBGMEQCIFmcvAlG8GRqEs/7CixYyuplrj+Ai5BI
Fuviqod5BGblAiALWpzNG9MZVPNDDLp+6WAj1Bds+7KmgzJN+wp4ddCUZwB1ANq2
v2s/tbYin5vCu1xr6HCRcWy7UYSFNL2kPTBI1/urAAABkBKo3FEAAAQDAEYwRAIg
Ulm5cDDRx75d+2DDebfbDugHGSrQo/lW3Z2Ko2W3AOACIDFi9JmF59PmInRWCrvp
gdKPAvhZtwbgAKAmSfbW0BWaMA0GCSqGSIb3DQEBCwUAA4IBAQCQnB3M6LCwJmw6
CZNlkZlCOodgCJTcxGaMpdy/REf832sQxdywddaYOMFBu8cwDeMX1hFlbFtRmlQH
KMcCT3GwCUxFZtZ/ilq6wOZAlH5NfLs7Ks3FoKb4mkHw7W833jBGDVsvrGsEgiEX
TYJ2ri5tgXz2aFBi+kvaQqqOP3RfgqewUPpNWFOFuIcoOKP9eSIMLJ77gqgtDYJY
vmHEF1Qy5TRg/M0hp/+iLvktYXJZuxBvFWS5n7tzOx4KVrhpCF/oUEgvB7jIfOcS
BjhbKrGjPevrYQm8GBbPB3WaUl6z3j15oACIcq1sCz7znmhklNq8sqcQ+YAuvTAD
nYACHArZ
-----END CERTIFICATE-----
subject=CN=mx.google.com
issuer=C=US, O=Google Trust Services, CN=WR2
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4970 bytes and written 447 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
250 SMTPUTF8

What is happens if you try that with your relay?

russellballestrini commented 3 months ago

@ruslandoga I run postfix on all my server/docker hosts, it's a localhost only relay so it shouldn't need tls. Does this mean I should look into disabling tls from postfix side and things should start working by defaults? (:if_available)

iex(plausible@c4b28e92e971)2> 
BREAK: (a)bort (A)bort with dump (c)ontinue (p)roc info (i)nfo
       (l)oaded (v)ersion (k)ill (D)b-tables (d)istribution
^Croot@analytics:/opt/plausible# telnet 127.0.0.1 25
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
220 analytics.unturf.com ESMTP Postfix
EHLO localhost
250-analytics.unturf.com
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250-DSN
250-SMTPUTF8
250 CHUNKING

ruslandoga commented 3 months ago

it's a localhost only relay

What does it mean? Where does it relay to? If it relays to the public internet, you can use Bamboo.Mua without that relay. Also please try the OpenSSL command since your relay seems to support STARTTLS. I wonder what OpenSSL would say about the certificate.

russellballestrini commented 3 months ago

I disabled tls on postfix and now I encounter a different error:

iex(plausible@c4b28e92e971)1> Application.get_env :plausible, Plausible.Mailer
[
  adapter: Bamboo.Mua,
  auth: [username: nil, password: nil],
  relay: "172.17.0.1",
  port: 25,
  tls: false
]
iex(plausible@c4b28e92e971)2> {:ok, _delivered_email} =
...(plausible@c4b28e92e971)2>   Plausible.Mailer.deliver_now(
...(plausible@c4b28e92e971)2>     Bamboo.Email.new_email(
...(plausible@c4b28e92e971)2>       # pick one of your own mailboxes
...(plausible@c4b28e92e971)2>       to: "russell.ballestrini@gmail.com",
...(plausible@c4b28e92e971)2>       from: PlausibleWeb.Email.mailer_email_from(),
...(plausible@c4b28e92e971)2>       text_body: "test"
...(plausible@c4b28e92e971)2>     )
...(plausible@c4b28e92e971)2>   )
** (ArgumentError) construction of binary failed: segment 2 of type 'binary': expected a binary but got: nil
    (mua 0.2.0) lib/mua.ex:378: Mua.auth/4
    (mua 0.2.0) lib/mua.ex:121: Mua.easy_send_one/6
    (mua 0.2.0) lib/mua.ex:84: Mua.easy_send_any/6
    (bamboo_mua 0.2.0) lib/bamboo_mua.ex:59: Bamboo.Mua.deliver/2
    (bamboo 2.3.0) lib/bamboo/mailer.ex:208: Bamboo.Mailer.deliver_now/4
    iex:3: (file)

postfix was using a selfed signed snakeoil cert and I never depended on tls before, this was mistake config. The relay only listens to localhost 127.0.0.1 and 172.17.0.1 and doesn't need tls for the first hop, and it may use TLS for the 2nd hop when it goes over the internet.

Thanks for spending time on this with me today.

ruslandoga commented 3 months ago

I disabled tls on postfix and now I encounter a different error:

Thanks, that's a configuration error which I'll fix tomorrow. I didn't expect relay to be used without auth.

Thanks for spending time on this with me today.

You are welcome. Does it mean the original issue is resolved?

russellballestrini commented 3 months ago

I think we can close this PR since I get the same error now when configured to :if_available when tls is really unavailable it does the right thing.

iex(plausible@55ccde1f37b0)1> Application.get_env :plausible, Plausible.Mailer
[
  adapter: Bamboo.Mua,
  auth: [username: nil, password: nil],
  relay: "172.17.0.1",
  port: 25,
  tls: :if_available
]
iex(plausible@55ccde1f37b0)2> {:ok, _delivered_email} =
...(plausible@55ccde1f37b0)2>   Plausible.Mailer.deliver_now(
...(plausible@55ccde1f37b0)2>     Bamboo.Email.new_email(
...(plausible@55ccde1f37b0)2>       # pick one of your own mailboxes
...(plausible@55ccde1f37b0)2>       to: "<removed>@gmail.com",
...(plausible@55ccde1f37b0)2>       from: PlausibleWeb.Email.mailer_email_from(),
...(plausible@55ccde1f37b0)2>       text_body: "test"
...(plausible@55ccde1f37b0)2>     )
...(plausible@55ccde1f37b0)2>   )
** (ArgumentError) construction of binary failed: segment 2 of type 'binary': expected a binary but got: nil
    (mua 0.2.0) lib/mua.ex:378: Mua.auth/4
    (mua 0.2.0) lib/mua.ex:121: Mua.easy_send_one/6
    (mua 0.2.0) lib/mua.ex:84: Mua.easy_send_any/6
    (bamboo_mua 0.2.0) lib/bamboo_mua.ex:59: Bamboo.Mua.deliver/2
    (bamboo 2.3.0) lib/bamboo/mailer.ex:208: Bamboo.Mailer.deliver_now/4
    iex:3: (file)

russellballestrini commented 3 months ago

@ruslandoga I'll be glad to test the work on anonymous smtp auth highlight me whenever you like!

ruslandoga commented 3 months ago

Just to be clear, :tls option does nothing in Bamboo.Mua, and also, you can use that adapter without a relay :)

Thank you!

russellballestrini commented 3 months ago

cool yeah, I don't know the erlang language all that well (having known about it for quite some time now) and I just started looking at this codebase today (been using it self-hosted for years!). not familiar with elixir or phoenix either.

I do most my coding with Python but I've always thought I would become an erlang person eventually. (maybe with LLMs my confidence will rise to the occasion.)

two years ago I integrated plausible into my app https://git.unturf.com/engineering/make-post-sell/make_post_sell/-/blame/master/make_post_sell/templates/shop_settings.j2?ref_type=heads#L111-120

plausible / analytics