haraka / Haraka

A fast, highly extensible, and event driven SMTP server
https://haraka.github.io
MIT License
5.11k stars 660 forks source link

Haraka fails to read `docs/Plugins.md` after installation on 3.0.4 #3394

Closed tyranron closed 2 months ago

tyranron commented 3 months ago

Describe the bug

npm install -g Haraka@3.0.4
haraka -i /etc/haraka/

haraka -o -c /etc/haraka

Expected behavior

Something like:

loaded TLD files:
  1=1448
  2=8320
  3=3571
loaded 9672 Public Suffixes
loglevel: INFO
log format: DEFAULT

Hook: connect ------------------------------------------------------------------
Plugin                              Method                              Prio T/O
--------------------------------------------------------------------------------
dnsbl                               connect_first                          0  30

Hook: helo ---------------------------------------------------------------------
Plugin                              Method                              Prio T/O
--------------------------------------------------------------------------------
helo.checks                         proto_mismatch_smtp                    0  30
helo.checks                         init                                   0  30
helo.checks                         match_re                               0  30
helo.checks                         bare_ip                                0  30
helo.checks                         dynamic                                0  30
helo.checks                         big_company                            0  30
helo.checks                         valid_hostname                         0  30
helo.checks                         rdns_match                             0  30
helo.checks                         forward_dns                            0  30
helo.checks                         host_mismatch                          0  30
helo.checks                         emit_log                               0  30
helo.checks                         literal_mismatch                       0  30

Hook: ehlo ---------------------------------------------------------------------
Plugin                              Method                              Prio T/O
--------------------------------------------------------------------------------
helo.checks                         proto_mismatch_esmtp                   0  30
helo.checks                         init                                   0  30
helo.checks                         match_re                               0  30
helo.checks                         bare_ip                                0  30
helo.checks                         dynamic                                0  30
helo.checks                         big_company                            0  30
helo.checks                         valid_hostname                         0  30
helo.checks                         rdns_match                             0  30
helo.checks                         forward_dns                            0  30
helo.checks                         host_mismatch                          0  30
helo.checks                         emit_log                               0  30
helo.checks                         literal_mismatch                       0  30

Hook: mail ---------------------------------------------------------------------
Plugin                              Method                              Prio T/O
--------------------------------------------------------------------------------
mail_from.is_resolvable             hook_mail                              0  30
rcpt_to.in_host_list                hook_mail                              0  30

Hook: rcpt ---------------------------------------------------------------------
Plugin                              Method                              Prio T/O
--------------------------------------------------------------------------------
rcpt_to.in_host_list                hook_rcpt                              0  30

Hook: data_post ----------------------------------------------------------------
Plugin                              Method                              Prio T/O
--------------------------------------------------------------------------------
headers                             duplicate_singular                     0  30
headers                             missing_required                       0  30
headers                             invalid_return_path                    0  30
headers                             invalid_date                           0  30
headers                             user_agent                             0  30
headers                             direct_to_mx                           0  30
headers                             from_match                             0  30
headers                             delivered_to                           0  30
headers                             mailing_list                           0  30
headers                             from_phish                             0  30

Hook: queue --------------------------------------------------------------------
Plugin                              Method                              Prio T/O
--------------------------------------------------------------------------------
queue/smtp_forward                  queue_forward                          0  30

Hook: get_mx -------------------------------------------------------------------
Plugin                              Method                              Prio T/O
--------------------------------------------------------------------------------
queue/smtp_forward                  get_mx                                 0  30

Observed behavior

loaded TLD files:
  1=1448
  2=8320
  3=3571
loaded 9672 Public Suffixes
loglevel: INFO
log format: DEFAULT

node:fs:561
  return binding.open(
                 ^

Error: ENOENT: no such file or directory, open 'docs/Plugins.md'
    at Object.openSync (node:fs:561:18)
    at Object.readFileSync (node:fs:445:35)
    at getHooks (/usr/local/lib/node_modules/Haraka/bin/haraka:231:8)
    at Object.<anonymous> (/usr/local/lib/node_modules/Haraka/bin/haraka:371:24)
    at Module._compile (node:internal/modules/cjs/loader:1546:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1691:10)
    at Module.load (node:internal/modules/cjs/loader:1317:32)
    at Module._load (node:internal/modules/cjs/loader:1127:12)
    at TracingChannel.traceSync (node:diagnostics_channel:315:14)
    at wrapModuleLoad (node:internal/modules/cjs/loader:217:24) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: 'docs/Plugins.md'
}

Node.js v22.7.0

Additional context

3.0.3 works OK.

s-aga-r commented 2 months ago

There are additional issues as well. For instance, in the hook_delivered params, the host is mistakenly set as the IP address. Additionally, the mailauth plugin's SPF check consistently fails.

brutus commented 2 months ago

Same here. If I manually copy docs/Plugin.md to the current working directory it no longer bails that way.

But fails like this:

loaded TLD files:
  1=1448
  2=8320
  3=3571
loaded 9672 Public Suffixes
loglevel: INFO
log format: DEFAULT

/opt/haraka/lib/node_modules/Haraka/bin/haraka:371
    for (const hook of getHooks()) {
                       ^

TypeError: getHooks is not a function or its return value is not iterable
    at Object.<anonymous> (/opt/haraka/lib/node_modules/Haraka/bin/haraka:371:24)
    at Module._compile (node:internal/modules/cjs/loader:1364:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1422:10)
    at Module.load (node:internal/modules/cjs/loader:1203:32)
    at Module._load (node:internal/modules/cjs/loader:1019:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:128:12)
    at node:internal/main/run_main_module:28:49

I guess b/c the getHooks function is missing an return?

lnedry commented 2 months ago

Plugins.md needs to be installed in the docs folder, e.g. /etc/haraka/docs/

msimerson commented 2 months ago

Hey folks, @lnedry has a partial fix I merged in #3405 and this should be the other needful bit. Please confirm.

lnedry commented 2 months ago

I confirmed that Plugins.md is being installed in the docs folder and help is working as expected.

tyranron commented 1 month ago

@msimerson still not fixed for 3.0.5. Steps for reproducing are the same:

npm install -g Haraka@3.0.5
haraka -i /etc/haraka/

haraka -o -c /etc/haraka
output ``` / # npm install -g Haraka@3.0.5 npm warn deprecated @npmcli/move-file@1.1.2: This functionality has been moved to @npmcli/fs npm warn deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported npm warn deprecated are-we-there-yet@3.0.1: This package is no longer supported. npm warn deprecated npmlog@6.0.2: This package is no longer supported. npm warn deprecated rimraf@3.0.2: Rimraf versions prior to v4 are no longer supported npm warn deprecated inflight@1.0.6: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful. npm warn deprecated gauge@4.0.4: This package is no longer supported. npm warn deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported added 317 packages in 47s 26 packages are looking for funding run `npm fund` for details / # haraka -i /etc/haraka/ create: /etc/haraka create: /etc/haraka/plugins create: /etc/haraka/docs create: /etc/haraka/config create: /etc/haraka/docs/Plugins.md create: /etc/haraka/config/smtp.ini create: /etc/haraka/config/log.ini create: /etc/haraka/config/plugins / # haraka -o -c /etc/haraka loaded TLD files: 1=1445 2=8416 3=3642 loaded 9773 Public Suffixes loglevel: INFO log format: DEFAULT node:fs:561 return binding.open( ^ Error: ENOENT: no such file or directory, open 'docs/Plugins.md' at Object.openSync (node:fs:561:18) at Object.readFileSync (node:fs:445:35) at getHooks (/usr/local/lib/node_modules/Haraka/bin/haraka:231:15) at Object. (/usr/local/lib/node_modules/Haraka/bin/haraka:371:24) at Module._compile (node:internal/modules/cjs/loader:1546:14) at Module._extensions..js (node:internal/modules/cjs/loader:1691:10) at Module.load (node:internal/modules/cjs/loader:1317:32) at Module._load (node:internal/modules/cjs/loader:1127:12) at TracingChannel.traceSync (node:diagnostics_channel:315:14) at wrapModuleLoad (node:internal/modules/cjs/loader:217:24) { errno: -2, code: 'ENOENT', syscall: 'open', path: 'docs/Plugins.md' } Node.js v22.9.0 ```
full reproducing steps ``` docker run --rm -it node:22-alpine sh apk update && apk upgrade apk add python3 g++ make npm install -g Haraka@3.0.5 haraka -i /etc/haraka/ haraka -o -c /etc/haraka ```
msimerson commented 1 month ago

@tyranron , please try this patch against the haraka 'binary' in your path. It should resolve the issue, and if not it'll at least provide a useful message.

diff --git a/bin/haraka b/bin/haraka
index 170bfbe7..0db61428 100755
--- a/bin/haraka
+++ b/bin/haraka
@@ -228,7 +228,8 @@ const plugin_doc = [

 function getHooks () {   // see haraka/Haraka#3306
-    return fs.readFileSync('docs/Plugins.md').toString()
+    const pi_path = path.resolve(parsed.configs, 'docs', 'Plugins.md')
+    return fs.readFileSync(pi_path).toString()
         .split('## Available Hooks')[1]    // discard everything before this string
         .split('### rcpt')[0]              // discard everything after this string
         .match(/\*\s(\S+)/gm)              // grab the first word of lines starting with '* '

Reminder to self: paths should always be assembled using path.join or path.resolve, so that we don't needlessly harm Windows compatibility.

msimerson commented 1 month ago

Additionally, the mailauth plugin's SPF check consistently fails.

@s-aga-r, Please create separate issues for stuff like this. Preferably in the repo where the issue is. postalsys/haraka-plugin-mailauth#3 should fix that.

lnedry commented 1 month ago

I just installed in on a clean Debian 12 and it is working as expected.

# haraka -o -c /etc/haraka
loaded TLD files:
  1=1445
  2=8416
  3=3642
loaded 9773 Public Suffixes
loglevel: INFO
log format: DEFAULT

Hook: mail ---------------------------------------------------------------------
Plugin                              Method                              Prio T/O
--------------------------------------------------------------------------------
mail_from.is_resolvable             hook_mail                              0  30
rcpt_to.in_host_list                hook_mail                              0  30

Hook: rcpt ---------------------------------------------------------------------
Plugin                              Method                              Prio T/O
--------------------------------------------------------------------------------
rcpt_to.in_host_list                hook_rcpt                              0  30

Hook: queue --------------------------------------------------------------------
Plugin                              Method                              Prio T/O
--------------------------------------------------------------------------------
queue/smtp_forward                  queue_forward                          0  30

Hook: get_mx -------------------------------------------------------------------
Plugin                              Method                              Prio T/O
--------------------------------------------------------------------------------
queue/smtp_forward                  get_mx                                 0  30
lnedry commented 1 month ago

I also confirmed that @msimerson's patch also works.

tyranron commented 1 month ago

@msimerson I do confirm that the patch works.

repro ``` $ docker run --rm -it node:22-alpine3.20 sh / # apk update && apk upgrade fetch https://dl-cdn.alpinelinux.org/alpine/v3.20/main/x86_64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.20/community/x86_64/APKINDEX.tar.gz v3.20.3-96-g1827497eea7 [https://dl-cdn.alpinelinux.org/alpine/v3.20/main] v3.20.3-104-g27788c02ed4 [https://dl-cdn.alpinelinux.org/alpine/v3.20/community] OK: 24163 distinct packages available Upgrading critical system libraries and apk-tools: (1/1) Upgrading apk-tools (2.14.4-r0 -> 2.14.4-r1) Executing busybox-1.36.1-r29.trigger Continuing the upgrade transaction with new apk-tools: OK: 10 MiB in 16 packages / # apk add python3 g++ make (1/29) Installing libstdc++-dev (13.2.1_git20240309-r0) (2/29) Installing jansson (2.14-r4) (3/29) Installing zstd-libs (1.5.6-r0) (4/29) Installing binutils (2.42-r0) (5/29) Installing libgomp (13.2.1_git20240309-r0) (6/29) Installing libatomic (13.2.1_git20240309-r0) (7/29) Installing gmp (6.3.0-r1) (8/29) Installing isl26 (0.26-r1) (9/29) Installing mpfr4 (4.2.1-r0) (10/29) Installing mpc1 (1.3.1-r1) (11/29) Installing gcc (13.2.1_git20240309-r0) (12/29) Installing musl-dev (1.2.5-r0) (13/29) Installing g++ (13.2.1_git20240309-r0) (14/29) Installing make (4.4.1-r2) (15/29) Installing libbz2 (1.0.8-r6) (16/29) Installing libexpat (2.6.3-r0) (17/29) Installing libffi (3.4.6-r0) (18/29) Installing gdbm (1.23-r1) (19/29) Installing xz-libs (5.6.2-r0) (20/29) Installing mpdecimal (4.0.0-r0) (21/29) Installing ncurses-terminfo-base (6.4_p20240420-r1) (22/29) Installing libncursesw (6.4_p20240420-r1) (23/29) Installing libpanelw (6.4_p20240420-r1) (24/29) Installing readline (8.2.10-r0) (25/29) Installing sqlite-libs (3.45.3-r1) (26/29) Installing python3 (3.12.6-r0) (27/29) Installing python3-pycache-pyc0 (3.12.6-r0) (28/29) Installing pyc (3.12.6-r0) (29/29) Installing python3-pyc (3.12.6-r0) Executing busybox-1.36.1-r29.trigger OK: 255 MiB in 45 packages / # npm install -g Haraka@3.0.5 npm warn deprecated @npmcli/move-file@1.1.2: This functionality has been moved to @npmcli/fs npm warn deprecated npmlog@6.0.2: This package is no longer supported. npm warn deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported npm warn deprecated rimraf@3.0.2: Rimraf versions prior to v4 are no longer supported npm warn deprecated are-we-there-yet@3.0.1: This package is no longer supported. npm warn deprecated inflight@1.0.6: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful. npm warn deprecated gauge@4.0.4: This package is no longer supported. npm warn deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported added 317 packages in 1m 26 packages are looking for funding run `npm fund` for details / # apk add patch (1/1) Installing patch (2.7.6-r10) Executing busybox-1.36.1-r29.trigger OK: 255 MiB in 46 packages / # vi wtf.patch / # patch /usr/local/lib/node_modules/Haraka/bin/haraka < wtf.patch patching file /usr/local/lib/node_modules/Haraka/bin/haraka / # haraka -i /etc/haraka/ create: /etc/haraka create: /etc/haraka/plugins create: /etc/haraka/docs create: /etc/haraka/config create: /etc/haraka/docs/Plugins.md create: /etc/haraka/config/smtp.ini create: /etc/haraka/config/log.ini create: /etc/haraka/config/plugins / # haraka -o -c /etc/haraka loaded TLD files: 1=1445 2=8416 3=3642 loaded 9773 Public Suffixes loglevel: INFO log format: DEFAULT Hook: mail --------------------------------------------------------------------- Plugin Method Prio T/O -------------------------------------------------------------------------------- mail_from.is_resolvable hook_mail 0 30 rcpt_to.in_host_list hook_mail 0 30 Hook: rcpt --------------------------------------------------------------------- Plugin Method Prio T/O -------------------------------------------------------------------------------- rcpt_to.in_host_list hook_rcpt 0 30 Hook: queue -------------------------------------------------------------------- Plugin Method Prio T/O -------------------------------------------------------------------------------- queue/smtp_forward queue_forward 0 30 Hook: get_mx ------------------------------------------------------------------- Plugin Method Prio T/O -------------------------------------------------------------------------------- queue/smtp_forward get_mx 0 30 ```