search

sipcapture / paStash

pastaʃ'ʃ = Spaghetti I/O Event Data Processing, Interpolation, Correlation and beyond :spaghetti:
http://sipcapture.io
Apache License 2.0
102 stars 28 forks source link

Filter GROK not working in container env. #234

Closed spady7 closed 2 months ago

spady7 commented 2 months ago

Hi, I have a problem when I try to use the "grok" filter. From what I read, if I understand correctly, it was part of the pastash installation. Then it was moved as an additional filter. If I try to install it I get the following error:

# npm install --unsafe-perm -g @pastash/filter_grok
npm error code E404
npm error 404 Not Found - GET https://registry.npmjs.org/@pastash%2ffilter_grok - Not found
npm error 404
npm error 404  '@pastash/filter_grok@*' is not in this registry.
npm error 404
npm error 404 Note that you can also install from a
npm error 404 tarball, folder, http url, or git url.
npm error A complete log of this run can be found in: /root/.npm/_logs/2024-08-21T14_40_24_796Z-debug-0.log

If I try to start a configuration that uses grok, at startup I get the following error:

[Wed, 21 Aug 2024 14:42:38 GMT] NOTICE Starting pastash 1.0.82
[Wed, 21 Aug 2024 14:42:38 GMT] INFO Max http socket 100
[Wed, 21 Aug 2024 14:42:38 GMT] INFO Loading config files from : /config
[Wed, 21 Aug 2024 14:42:38 GMT] INFO Files loaded from directory, 4 urls found
[Wed, 21 Aug 2024 14:42:38 GMT] INFO Loading config : 7 urls
[Wed, 21 Aug 2024 14:42:38 GMT] INFO Initializing output Stdout
[Wed, 21 Aug 2024 14:42:38 GMT] INFO Initializing output file
[Wed, 21 Aug 2024 14:42:38 GMT] INFO Start output to file /usr/src/app/output_testing.json
[Wed, 21 Aug 2024 14:42:38 GMT] INFO Initializing filter AddHost
[Wed, 21 Aug 2024 14:42:38 GMT] INFO Initializing filter AddTimestamp
[Wed, 21 Aug 2024 14:42:38 GMT] INFO Initializing filter AddVersion
Loading npm module... @pastash/filter_grok
[Wed, 21 Aug 2024 14:42:38 GMT] ERROR Unable to load urls from command line
[Wed, 21 Aug 2024 14:42:38 GMT] ERROR TypeError: Cannot read properties of undefined (reading 'create')
    at LogstashAgent.configure (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:143:21)
    at LogstashAgent.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:215:10)
    at /usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:3113:16
    at replenish (/usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:1014:17)
    at /usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:1019:9
    at eachLimit$1 (/usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:3199:24)
    at Object.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:1049:16)
    at LogstashAgent.start_modules (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:214:9)
    at LogstashAgent.start_filters (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:255:8)
    at LogstashAgent.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:204:10)

Can you help me? Regards

lmangani commented 2 months ago

Then it was moved as an additional filter.

The Grok filter was never moved into a plugin. I just ran a standard test and it seems to pass using an npm installed version.

input {
  stdin{}
}

filter {
  grok {
    match => '%{WORD:w1} %{NUMBER:num1}'
  }
}

output {
  stdout{}
}

Could you confirm this is the case when you run using an npm installed version? Are you using the ghcr.io/sipcapture/pastash@latest image in your testing?

spady7 commented 2 months ago

Hi @lmangani tried with your official image (https://hub.docker.com/r/qxip/pastash/tags) and I get error. Tried with npm version (and get issue):

/usr/src/app # npm list -g
/usr/local/lib
+-- @pastash/filter_app_audiocodes@1.1.5
+-- @pastash/output_loki@1.0.10
+-- @pastash/pastash@1.0.82
+-- async@3.2.6
+-- corepack@0.29.3
+-- csv-parser@3.0.0
+-- log4node@0.1.6
+-- lru-cache@11.0.0
+-- mkdirp@3.0.1
+-- n@9.2.3
+-- npm@10.8.2
+-- optimist@0.6.1
`-- requireg@0.2.2

I think pastash@1.0.82 is latest.

lmangani commented 2 months ago

The official images are right here on github: https://github.com/sipcapture/paStash/pkgs/container/pastash

spady7 commented 2 months ago

Ok, so the same I'am using. It happens only on container. Try yourself using following Dockerfile

FROM node:lts-alpine

# Install jq
RUN apk update && apk add --no-cache jq

RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

COPY package.json /usr/src/app/
COPY install-global-packages-alpine.sh /usr/src/app/

# Ensure the script is executable
RUN chmod +x install-global-packages-alpine.sh

# Run the script to install global npm packages
RUN ./install-global-packages-alpine.sh

RUN mkdir -p /config

EXPOSE 9999/tcp 514/udp 9999/udp 514/tcp
CMD [ "/usr/local/bin/pastash", "--config_dir", "/config" ]

and install-global-packages-alpine.sh :

#!/bin/sh

# Exit immediately if a command exits with a non-zero status
set -e

# Check if package.json exists
if [ ! -f package.json ]; then
  echo "package.json not found!"
  exit 1
fi

npm install --unsafe-perm -g @pastash/pastash

# Extract dependencies from package.json and install them globally
jq -r '.dependencies | keys[]' package.json | while read -r package; do
  npm install -g "$package"
done

echo "Global npm packages installed successfully."
lmangani commented 2 months ago

I see the issue. Oniguruma won't build without build tools. Working on a solution.

lmangani commented 2 months ago

@spady7 could you try again with ghcr.io/sipcapture/pastash:latest (no cache) this image is built in multi-stages and should include the compiled modules required by the imports

spady7 commented 2 months ago

@lmangani seams to work:

root@d4aa1a4af9bd:/app# node bin/pastash --config_dir=/config
[Thu, 22 Aug 2024 20:53:49 GMT] NOTICE Starting pastash 1.0.82
[Thu, 22 Aug 2024 20:53:49 GMT] INFO Max http socket 100
[Thu, 22 Aug 2024 20:53:49 GMT] INFO Loading config files from : /config
[Thu, 22 Aug 2024 20:53:49 GMT] INFO Files loaded from directory, 4 urls found
[Thu, 22 Aug 2024 20:53:49 GMT] INFO Loading config : 7 urls
[Thu, 22 Aug 2024 20:53:49 GMT] INFO Initializing output Stdout
[Thu, 22 Aug 2024 20:53:49 GMT] INFO Initializing output file
[Thu, 22 Aug 2024 20:53:49 GMT] INFO Start output to file /usr/src/app/output_testing.json
[Thu, 22 Aug 2024 20:53:49 GMT] INFO Initializing filter AddHost
[Thu, 22 Aug 2024 20:53:49 GMT] INFO Initializing filter AddTimestamp
[Thu, 22 Aug 2024 20:53:49 GMT] INFO Initializing filter AddVersion
[Thu, 22 Aug 2024 20:53:49 GMT] INFO Initializing filter Grok
[Thu, 22 Aug 2024 20:53:49 GMT] INFO Initializing grok filter, pattern: %{TIMESTAMP_ISO8601:timestamp}: %{GREEDYDATA:message}
[Thu, 22 Aug 2024 20:53:49 GMT] INFO Loading grok patterns
[Thu, 22 Aug 2024 20:53:49 GMT] INFO Grok patterns loaded from patterns directories 188
[Thu, 22 Aug 2024 20:53:49 GMT] INFO Initializing input Udp
[Thu, 22 Aug 2024 20:53:49 GMT] INFO Start listening on udp 0.0.0.0:30514
[Thu, 22 Aug 2024 20:53:49 GMT] INFO Config loaded.

If I want to build my own image, like Dockerfile posted before, what Have I to add to get Oniguruma compiled? Regards

lmangani commented 2 months ago

Please refer to the updated Dockerfile You can do the same with alpine and just copy the resulting node_modules to your image If you have suggestions, feel free to fire in a PR for the Dockerfile

spady7 commented 2 months ago

@lmangani If try to use external plugin, as loki (installed with npm install --unsafe-perm -g @pastash/output_loki) i get error "module not found". Tried to copy content of output_loki under "/app/lib/outputs" but it does not find module. Can you point me how to use external installed plugin? Regards

lmangani commented 2 months ago

@spady7 i don't understand the report and the module clearly exists on npm so if you want provide the Dockerfile and we can take a look.

spady7 commented 2 months ago

@lmangani sure. Following what I've done and errors.

Used Dockerfile:

# PaStash Docker Builder
FROM node:20-slim as builder
RUN apt update && apt install -y python3 make gcc g++
RUN mkdir -p /app
WORKDIR /app
COPY package.json /app/
RUN npm install
COPY . /app

# PaStash Docker Container
FROM node:20-slim

RUN mkdir -p /config
COPY --from=builder /app /app
WORKDIR /app

EXPOSE 9999/tcp 514/udp 9999/udp 514/tcp
CMD [ "bin/pastash", "--config_dir", "/config" ]

packajes.json (modified so plugin app_audiocodes and output_loki are installed):

{
  "name": "@pastash/pastash",
  "description": "Spaghetti I/O Processing, Interpolation, Correlation and beyond - now with plugins",
  "version": "1.0.82",
  "author": "Lorenzo Mangani <lorenzo.mangani@gmail.com>",
  "contributors": [
    {
      "name": "Bertrand Paquet",
      "email": "bertrand.paquet@gmail.com>"
    }
  ],
  "keywords": [
    "logstash",
    "log",
    "zmq",
    "zeromq",
    "hep",
    "eep",
    "freeswitch",
    "asterisk",
    "avaya",
    "janus",
    "homer",
    "hepic",
    "splunk",
    "bencode",
    "redis",
    "cep",
    "elasticsearch",
    "beats",
    "influxdb",
    "statsd",
    "netflow"
  ],
  "main": "./lib/agent.js",
  "homepage": "https://github.com/sipcapture/pastash",
  "repository": {
    "type": "git",
    "url": "git://github.com/sipcapture/pastash.git"
  },
  "license": "Apache-2.0",
  "devDependencies": {
    "istanbul": "0.4.x",
    "jison": "0.4.x",
    "js-beautify": "1.15.x",
    "jshint": "^2.9.6",
    "rimraf": "5.0.x",
    "vows": "0.7.0",
    "vows-batch-retry": "0.0.4",
    "which": "^4.0.0"
  },
  "scripts": {
    "test": "./test-runner.sh"
  },
  "dependencies": {
    "async": "2.6.4",
    "csv-parser": "^2.3.3",
    "log4node": "0.1.6",
    "lru-cache": "4.1.x",
    "mkdirp": "0.5.1",
    "optimist": "0.6.1",
    "requireg": "^0.2.1"
  },
  "optionalDependencies": {
    "amqplib": "0.5.1",
    "fast-json-stringify": "^5.15.1",
    "geoip-lite": "1.4.10",
    "gun": "0.2019.1228",
    "hep-js": "^1.0.20",
    "hoek": "~6.1.3",
    "http-proxy-agent": "3.x",
    "https-proxy-agent": "7.0.4",
    "lumberjack-protocol": "git://github.com/bpaquet/node-lumberjack-protocol.git",
    "maxmind": "0.6.x",
    "maxmind-geolite-mirror": "1.3.x",
    "moment": "2.30.1",
    "msgpack": "1.0.x",
    "mustache": "4.2.0",
    "object.omit": "3.0.0",
    "oniguruma": "7.2.x",
    "redis": "4.6.14",
    "tinymath": "1.2.1",
    "ws": "8.18.0",
    "zmq": "*",
    "@pastash/filter_app_audiocodes": "latest",
    "@pastash/output_loki": "latest"
  },
  "directories": {
    "test": "./test",
    "bin": "./bin"
  }
}

here my conf file:

input {
  udp {
    host => 0.0.0.0
    port => 514
    tags => CDR
  }
}

filter {
  compute_field {
    field => origin
    value => "AUDIOCODES"
    }
  grok {
    match => '<%{NUMBER:syslog_pri}>\[S=%{NUMBER:session_id}\] \|%{WORD:action}\s*\|%{DATA:source}\s*\|%{NUMBER:code1}\s*\|%{NUMBER:code2}\s*\|%{DATA:unique_id}\s*%{DATA:unique_id}\s*\|%{TIME:time}\s+%{WORD:timezone} %{WORD:day} %{MONTH:month} %{MONTHDAY:monthday} %{YEAR:year}\|%{NUMBER:value1}\s*\|%{NUMBER:value2}\s*\|%{WORD:timezone_final}\s*\|%{NUMBER:chiamante}\s*\|%{NUMBER:chiamante2}\s*\|%{NUMBER:chiamato}\s*\|%{NUMBER:chiamato2}\s*\|%{IP:ip_src}\s*\|%{IP:ip_dst}\s*\|%{DATA:call_status}\s*\|%{WORD:sip_method}\s*\|%{DATA:provider_src}\s*\|%{DATA:provider_dst}\s*\|%{NUMBER:flag}'
  }
}

 output {
     stdout {}
     loki {
       host => "qryn"
       port => 3100
       path => "/loki/api/v1/push"
     }
   }

}

So when I start container I get following error:

[Fri, 23 Aug 2024 10:07:10 GMT] INFO Changing log_level debug
[Fri, 23 Aug 2024 10:07:10 GMT] NOTICE Starting pastash 1.0.82
[Fri, 23 Aug 2024 10:07:10 GMT] INFO Max http socket 100
[Fri, 23 Aug 2024 10:07:10 GMT] INFO Loading config file : /opt/cdr_audiocodes.conf
[Fri, 23 Aug 2024 10:07:10 GMT] INFO File loaded, 5 urls found
[Fri, 23 Aug 2024 10:07:10 GMT] INFO Loading config : 8 urls
[Fri, 23 Aug 2024 10:07:10 GMT] DEBUG Loading urls [
  'filter://add_host://',
  'filter://add_timestamp://',
  'filter://add_version://',
  'input://udp://?host=0.0.0.0&port=514&tags=CDR',
  'filter://compute_field://?field=origin&value=AUDIOCODES',
  'filter://grok://?match=%3C%25%7BNUMBER%3Asyslog_pri%7D%3E%5C%5BS%3D%25%7BNUMBER%3Asession_id%7D%5C%5D%20%5C%7C%25%7BWORD%3Aaction%7D%5Cs*%5C%7C%25%7BDATA%3Asource%7D%5Cs*%5C%7C%25%7BNUMBER%3Acode1%7D%5Cs*%5C%7C%25%7BNUMBER%3Acode2%7D%5Cs*%5C%7C%25%7BDATA%3Aunique_id%7D%5Cs*%25%7BDATA%3Aunique_id%7D%5Cs*%5C%7C%25%7BTIME%3Atime%7D%5Cs%2B%25%7BWORD%3Atimezone%7D%20%25%7BWORD%3Aday%7D%20%25%7BMONTH%3Amonth%7D%20%25%7BMONTHDAY%3Amonthday%7D%20%25%7BYEAR%3Ayear%7D%5C%7C%25%7BNUMBER%3Avalue1%7D%5Cs*%5C%7C%25%7BNUMBER%3Avalue2%7D%5Cs*%5C%7C%25%7BWORD%3Atimezone_final%7D%5Cs*%5C%7C%25%7BNUMBER%3Achiamante%7D%5Cs*%5C%7C%25%7BNUMBER%3Achiamante2%7D%5Cs*%5C%7C%25%7BNUMBER%3Achiamato%7D%5Cs*%5C%7C%25%7BNUMBER%3Achiamato2%7D%5Cs*%5C%7C%25%7BIP%3Aip_src%7D%5Cs*%5C%7C%25%7BIP%3Aip_dst%7D%5Cs*%5C%7C%25%7BDATA%3Acall_status%7D%5Cs*%5C%7C%25%7BWORD%3Asip_method%7D%5Cs*%5C%7C%25%7BDATA%3Aprovider_src%7D%5Cs*%5C%7C%25%7BDATA%3Aprovider_dst%7D%5Cs*%5C%7C%25%7BNUMBER%3Aflag%7D',
  'output://stdout://',
  'output://loki://?host=qryn&port=3100&path=%2Floki%2Fapi%2Fv1%2Fpush'
]
[Fri, 23 Aug 2024 10:07:10 GMT] DEBUG Initializing module output
[Fri, 23 Aug 2024 10:07:10 GMT] INFO Initializing output Stdout
[Fri, 23 Aug 2024 10:07:10 GMT] DEBUG Initializing module output
[Fri, 23 Aug 2024 10:07:10 GMT] DEBUG no local module found outputs output_loki Error: Cannot find module 'outputs/output_loki'
Require stack:
- /app/lib/agent.js
- /app/bin/pastash
    at Module._resolveFilename (node:internal/modules/cjs/loader:1225:15)
    at Module._load (node:internal/modules/cjs/loader:1051:27)
    at Module.require (node:internal/modules/cjs/loader:1311:19)
    at require (node:internal/modules/helpers:179:18)
    at LogstashAgent.configure (/app/lib/agent.js:133:17)
    at LogstashAgent.<anonymous> (/app/lib/agent.js:215:10)
    at /app/node_modules/async/dist/async.js:3113:16
    at replenish (/app/node_modules/async/dist/async.js:1014:17)
    at /app/node_modules/async/dist/async.js:1019:9
    at eachLimit$1 (/app/node_modules/async/dist/async.js:3199:24) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '/app/lib/agent.js', '/app/bin/pastash' ]
}
Loading npm module... @pastash/output_loki
[Fri, 23 Aug 2024 10:07:10 GMT] DEBUG NestedError: Could not require module '@pastash/output_loki'
    at requireg (/app/node_modules/requireg/lib/requireg.js:14:11)
    at LogstashAgent.configure (/app/lib/agent.js:139:18)
    at LogstashAgent.<anonymous> (/app/lib/agent.js:215:10)
    at /app/node_modules/async/dist/async.js:3113:16
    at replenish (/app/node_modules/async/dist/async.js:1014:17)
    at /app/node_modules/async/dist/async.js:1019:9
    at eachLimit$1 (/app/node_modules/async/dist/async.js:3199:24)
    at Object.<anonymous> (/app/node_modules/async/dist/async.js:1049:16)
    at LogstashAgent.start_modules (/app/lib/agent.js:214:9)
    at LogstashAgent.start_outputs (/app/lib/agent.js:270:8)
Caused By: Error: Cannot find module '@pastash/pastash'
Require stack:
- /app/node_modules/@pastash/output_loki/abstract_http.js
- /app/node_modules/@pastash/output_loki/output_loki.js
- /app/node_modules/requireg/lib/requireg.js
- /app/lib/agent.js
- /app/bin/pastash
    at Module._resolveFilename (node:internal/modules/cjs/loader:1225:15)
    at Module._load (node:internal/modules/cjs/loader:1051:27)
    at Module.require (node:internal/modules/cjs/loader:1311:19)
    at require (node:internal/modules/helpers:179:18)
    at Object.<anonymous> (/app/node_modules/@pastash/output_loki/abstract_http.js:1:19)
    at Module._compile (node:internal/modules/cjs/loader:1469:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1548:10)
    at Module.load (node:internal/modules/cjs/loader:1288:32)
    at Module._load (node:internal/modules/cjs/loader:1104:12)
    at Module.require (node:internal/modules/cjs/loader:1311:19) {
  nested: Error: Cannot find module '@pastash/pastash'
  Require stack:
  - /app/node_modules/@pastash/output_loki/abstract_http.js
  - /app/node_modules/@pastash/output_loki/output_loki.js
  - /app/node_modules/requireg/lib/requireg.js
  - /app/lib/agent.js
  - /app/bin/pastash
      at Module._resolveFilename (node:internal/modules/cjs/loader:1225:15)
      at Module._load (node:internal/modules/cjs/loader:1051:27)
      at Module.require (node:internal/modules/cjs/loader:1311:19)
      at require (node:internal/modules/helpers:179:18)
      at Object.<anonymous> (/app/node_modules/@pastash/output_loki/abstract_http.js:1:19)
      at Module._compile (node:internal/modules/cjs/loader:1469:14)
      at Module._extensions..js (node:internal/modules/cjs/loader:1548:10)
      at Module.load (node:internal/modules/cjs/loader:1288:32)
      at Module._load (node:internal/modules/cjs/loader:1104:12)
      at Module.require (node:internal/modules/cjs/loader:1311:19) {
    code: 'MODULE_NOT_FOUND',
    requireStack: [
      '/app/node_modules/@pastash/output_loki/abstract_http.js',
      '/app/node_modules/@pastash/output_loki/output_loki.js',
      '/app/node_modules/requireg/lib/requireg.js',
      '/app/lib/agent.js',
      '/app/bin/pastash'
    ]
  }
}
[Fri, 23 Aug 2024 10:07:10 GMT] ERROR Unable to load urls from command line
[Fri, 23 Aug 2024 10:07:10 GMT] ERROR TypeError: Cannot read properties of undefined (reading 'create')
    at LogstashAgent.configure (/app/lib/agent.js:143:21)
    at LogstashAgent.<anonymous> (/app/lib/agent.js:215:10)
    at /app/node_modules/async/dist/async.js:3113:16
    at replenish (/app/node_modules/async/dist/async.js:1014:17)
    at /app/node_modules/async/dist/async.js:1019:9
    at eachLimit$1 (/app/node_modules/async/dist/async.js:3199:24)
    at Object.<anonymous> (/app/node_modules/async/dist/async.js:1049:16)
    at LogstashAgent.start_modules (/app/lib/agent.js:214:9)
    at LogstashAgent.start_outputs (/app/lib/agent.js:270:8)
    at LogstashAgent.start (/app/lib/agent.js:200:8)

but loki module is installed:

root@f39e7c6fffd5:/app# ls -la node_modules/\@pastash/
total 16
drwxr-xr-x   4 root root 4096 Aug 23 09:51 .
drwxr-xr-x 199 root root 4096 Aug 23 09:52 ..
drwxr-xr-x   3 root root 4096 Aug 23 09:51 filter_app_audiocodes
drwxr-xr-x   2 root root 4096 Aug 23 09:51 output_loki

What Am I wrong?

Regards

lmangani commented 2 months ago

packajes.json (modified so plugin app_audiocodes and output_loki are installed):

That's a very convoluted way. Plugins must be installed as global. Just install them in the final container.

# PaStash Docker Builder
FROM node:20-slim as builder
RUN apt update && apt install -y python3 make gcc g++
RUN mkdir -p /app
WORKDIR /app
COPY package.json /app/
RUN npm install
COPY . /app

# PaStash Docker Container
FROM node:20-slim
RUN mkdir -p /config
COPY --from=builder /app /app
RUN npm install -g @pastash/output_loki @pastash/filter_app_audiocodes
WORKDIR /app
EXPOSE 9999/tcp 514/udp 9999/udp 514/tcp
CMD [ "bin/pastash", "--config_dir", "/config" ]
spady7 commented 2 months ago

Already done/tested, but same error:

root@f39e7c6fffd5:/app# npm install --unsafe-perm -g @pastash/output_loki

added 7 packages in 4s

and after that:

[Fri, 23 Aug 2024 10:24:10 GMT] INFO Changing log_level debug
[Fri, 23 Aug 2024 10:24:10 GMT] NOTICE Starting pastash 1.0.82
[Fri, 23 Aug 2024 10:24:10 GMT] INFO Max http socket 100
[Fri, 23 Aug 2024 10:24:10 GMT] INFO Loading config file : /opt/cdr_audiocodes.conf
[Fri, 23 Aug 2024 10:24:10 GMT] INFO File loaded, 5 urls found
[Fri, 23 Aug 2024 10:24:10 GMT] INFO Loading config : 8 urls
[Fri, 23 Aug 2024 10:24:10 GMT] DEBUG Loading urls [
  'filter://add_host://',
  'filter://add_timestamp://',
  'filter://add_version://',
  'input://udp://?host=0.0.0.0&port=514&tags=CDR',
  'filter://compute_field://?field=origin&value=AUDIOCODES',
  'filter://grok://?match=%3C%25%7BNUMBER%3Asyslog_pri%7D%3E%5C%5BS%3D%25%7BNUMBER%3Asession_id%7D%5C%5D%20%5C%7C%25%7BWORD%3Aaction%7D%5Cs*%5C%7C%25%7BDATA%3Asource%7D%5Cs*%5C%7C%25%7BNUMBER%3Acode1%7D%5Cs*%5C%7C%25%7BNUMBER%3Acode2%7D%5Cs*%5C%7C%25%7BDATA%3Aunique_id%7D%5Cs*%25%7BDATA%3Aunique_id%7D%5Cs*%5C%7C%25%7BTIME%3Atime%7D%5Cs%2B%25%7BWORD%3Atimezone%7D%20%25%7BWORD%3Aday%7D%20%25%7BMONTH%3Amonth%7D%20%25%7BMONTHDAY%3Amonthday%7D%20%25%7BYEAR%3Ayear%7D%5C%7C%25%7BNUMBER%3Avalue1%7D%5Cs*%5C%7C%25%7BNUMBER%3Avalue2%7D%5Cs*%5C%7C%25%7BWORD%3Atimezone_final%7D%5Cs*%5C%7C%25%7BNUMBER%3Achiamante%7D%5Cs*%5C%7C%25%7BNUMBER%3Achiamante2%7D%5Cs*%5C%7C%25%7BNUMBER%3Achiamato%7D%5Cs*%5C%7C%25%7BNUMBER%3Achiamato2%7D%5Cs*%5C%7C%25%7BIP%3Aip_src%7D%5Cs*%5C%7C%25%7BIP%3Aip_dst%7D%5Cs*%5C%7C%25%7BDATA%3Acall_status%7D%5Cs*%5C%7C%25%7BWORD%3Asip_method%7D%5Cs*%5C%7C%25%7BDATA%3Aprovider_src%7D%5Cs*%5C%7C%25%7BDATA%3Aprovider_dst%7D%5Cs*%5C%7C%25%7BNUMBER%3Aflag%7D',
  'output://stdout://',
  'output://loki://?host=qryn&port=3100&path=%2Floki%2Fapi%2Fv1%2Fpush'
]
[Fri, 23 Aug 2024 10:24:10 GMT] DEBUG Initializing module output
[Fri, 23 Aug 2024 10:24:10 GMT] INFO Initializing output Stdout
[Fri, 23 Aug 2024 10:24:10 GMT] DEBUG Initializing module output
[Fri, 23 Aug 2024 10:24:10 GMT] DEBUG no local module found outputs output_loki Error: Cannot find module 'outputs/output_loki'
Require stack:
- /app/lib/agent.js
- /app/bin/pastash
    at Module._resolveFilename (node:internal/modules/cjs/loader:1225:15)
    at Module._load (node:internal/modules/cjs/loader:1051:27)
    at Module.require (node:internal/modules/cjs/loader:1311:19)
    at require (node:internal/modules/helpers:179:18)
    at LogstashAgent.configure (/app/lib/agent.js:133:17)
    at LogstashAgent.<anonymous> (/app/lib/agent.js:215:10)
    at /app/node_modules/async/dist/async.js:3113:16
    at replenish (/app/node_modules/async/dist/async.js:1014:17)
    at /app/node_modules/async/dist/async.js:1019:9
    at eachLimit$1 (/app/node_modules/async/dist/async.js:3199:24) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '/app/lib/agent.js', '/app/bin/pastash' ]
}
Loading npm module... @pastash/output_loki
[Fri, 23 Aug 2024 10:24:10 GMT] DEBUG NestedError: Could not require module '@pastash/output_loki'
    at requireg (/app/node_modules/requireg/lib/requireg.js:14:11)
    at LogstashAgent.configure (/app/lib/agent.js:139:18)
    at LogstashAgent.<anonymous> (/app/lib/agent.js:215:10)
    at /app/node_modules/async/dist/async.js:3113:16
    at replenish (/app/node_modules/async/dist/async.js:1014:17)
    at /app/node_modules/async/dist/async.js:1019:9
    at eachLimit$1 (/app/node_modules/async/dist/async.js:3199:24)
    at Object.<anonymous> (/app/node_modules/async/dist/async.js:1049:16)
    at LogstashAgent.start_modules (/app/lib/agent.js:214:9)
    at LogstashAgent.start_outputs (/app/lib/agent.js:270:8)
Caused By: Error: Cannot find module '@pastash/pastash'
Require stack:
- /app/node_modules/@pastash/output_loki/abstract_http.js
- /app/node_modules/@pastash/output_loki/output_loki.js
- /app/node_modules/requireg/lib/requireg.js
- /app/lib/agent.js
- /app/bin/pastash
    at Module._resolveFilename (node:internal/modules/cjs/loader:1225:15)
    at Module._load (node:internal/modules/cjs/loader:1051:27)
    at Module.require (node:internal/modules/cjs/loader:1311:19)
    at require (node:internal/modules/helpers:179:18)
    at Object.<anonymous> (/app/node_modules/@pastash/output_loki/abstract_http.js:1:19)
    at Module._compile (node:internal/modules/cjs/loader:1469:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1548:10)
    at Module.load (node:internal/modules/cjs/loader:1288:32)
    at Module._load (node:internal/modules/cjs/loader:1104:12)
    at Module.require (node:internal/modules/cjs/loader:1311:19) {
  nested: Error: Cannot find module '@pastash/pastash'
  Require stack:
  - /app/node_modules/@pastash/output_loki/abstract_http.js
  - /app/node_modules/@pastash/output_loki/output_loki.js
  - /app/node_modules/requireg/lib/requireg.js
  - /app/lib/agent.js
  - /app/bin/pastash
      at Module._resolveFilename (node:internal/modules/cjs/loader:1225:15)
      at Module._load (node:internal/modules/cjs/loader:1051:27)
      at Module.require (node:internal/modules/cjs/loader:1311:19)
      at require (node:internal/modules/helpers:179:18)
      at Object.<anonymous> (/app/node_modules/@pastash/output_loki/abstract_http.js:1:19)
      at Module._compile (node:internal/modules/cjs/loader:1469:14)
      at Module._extensions..js (node:internal/modules/cjs/loader:1548:10)
      at Module.load (node:internal/modules/cjs/loader:1288:32)
      at Module._load (node:internal/modules/cjs/loader:1104:12)
      at Module.require (node:internal/modules/cjs/loader:1311:19) {
    code: 'MODULE_NOT_FOUND',
    requireStack: [
      '/app/node_modules/@pastash/output_loki/abstract_http.js',
      '/app/node_modules/@pastash/output_loki/output_loki.js',
      '/app/node_modules/requireg/lib/requireg.js',
      '/app/lib/agent.js',
      '/app/bin/pastash'
    ]
  }
}
[Fri, 23 Aug 2024 10:24:10 GMT] ERROR Unable to load urls from command line
[Fri, 23 Aug 2024 10:24:10 GMT] ERROR TypeError: Cannot read properties of undefined (reading 'create')
    at LogstashAgent.configure (/app/lib/agent.js:143:21)
    at LogstashAgent.<anonymous> (/app/lib/agent.js:215:10)
    at /app/node_modules/async/dist/async.js:3113:16
    at replenish (/app/node_modules/async/dist/async.js:1014:17)
    at /app/node_modules/async/dist/async.js:1019:9
    at eachLimit$1 (/app/node_modules/async/dist/async.js:3199:24)
    at Object.<anonymous> (/app/node_modules/async/dist/async.js:1049:16)
    at LogstashAgent.start_modules (/app/lib/agent.js:214:9)
    at LogstashAgent.start_outputs (/app/lib/agent.js:270:8)
    at LogstashAgent.start (/app/lib/agent.js:200:8)

Same error

spady7 commented 2 months ago

I suspect this is the problem.

root@f39e7c6fffd5:/app# npm list -g
/usr/local/lib
+-- @pastash/output_loki@1.0.10
+-- corepack@0.29.3
`-- npm@10.8.2

I have the plugin globally, but there is no "pastash". That's because pastash was not installed by npm, right? So, is there a way to copy somewhere the loki files? To make discoverable by pastash program?

lmangani commented 2 months ago

That's because pastash was not installed by npm, right?

Yes and no. It shouldn't matter but it seems indeed the global path somehow is not matching what node expects on module imports from the code. Perhaps you can use this approach:

# PaStash Docker Builder
FROM ghcr.io/sipcapture/pastash:latest
RUN npm install -g @pastash/pastash @pastash/output_loki @pastash/filter_app_audiocodes
RUN mkdir -p /config
EXPOSE 9999/tcp 514/udp 9999/udp 514/tcp
CMD [ "bin/pastash", "--config_dir", "/config" ]
[Fri, 23 Aug 2024 10:46:01 GMT] NOTICE Starting pastash 1.0.82
[Fri, 23 Aug 2024 10:46:01 GMT] INFO Max http socket 100
[Fri, 23 Aug 2024 10:46:01 GMT] INFO Loading config files from : /config
[Fri, 23 Aug 2024 10:46:01 GMT] INFO Files loaded from directory, 4 urls found
[Fri, 23 Aug 2024 10:46:01 GMT] INFO Loading config : 7 urls
[Fri, 23 Aug 2024 10:46:01 GMT] INFO Initializing output Stdout
Loading npm module... @pastash/output_loki
[Fri, 23 Aug 2024 10:46:03 GMT] INFO Initializing output Loki
[Fri, 23 Aug 2024 10:46:03 GMT] INFO Start HTTP output to LOKI http loki:3100/api/prom/push
[Fri, 23 Aug 2024 10:46:03 GMT] INFO Initializing filter AddHost
[Fri, 23 Aug 2024 10:46:03 GMT] INFO Initializing filter AddTimestamp
[Fri, 23 Aug 2024 10:46:03 GMT] INFO Initializing filter AddVersion
[Fri, 23 Aug 2024 10:46:03 GMT] INFO Initializing filter Grok
[Fri, 23 Aug 2024 10:46:03 GMT] INFO Initializing grok filter, pattern: %{WORD:w1} %{NUMBER:num1}
[Fri, 23 Aug 2024 10:46:03 GMT] INFO Loading grok patterns
[Fri, 23 Aug 2024 10:46:03 GMT] INFO Grok patterns loaded from patterns directories 188
[Fri, 23 Aug 2024 10:46:03 GMT] INFO Initializing input Stdin
[Fri, 23 Aug 2024 10:46:03 GMT] INFO Config loaded.
[Fri, 23 Aug 2024 10:46:12 GMT] INFO SIGINT received.
[Fri, 23 Aug 2024 10:46:12 GMT] INFO Closing stdin
[Fri, 23 Aug 2024 10:46:12 GMT] INFO Closing stdout
[Fri, 23 Aug 2024 10:46:12 GMT] INFO Closing HTTP Post output to loki 3100 ssl false
[Fri, 23 Aug 2024 10:46:12 GMT] INFO Closing agent
[Fri, 23 Aug 2024 10:46:12 GMT] INFO Quitting.
spady7 commented 2 months ago

Thank you @lmangani for your help. Finally I found a way, that in my case best fit my own needs, and I want to share. Maybe could be usefull for someone else.

Dockerfile

# PaStash Docker Builder
FROM node:20-slim as builder

RUN apt update && apt install -y python3 make gcc g++ jq
RUN mkdir -p /app
WORKDIR /app
COPY package.json /app/
COPY install-global-packages.sh /app/
RUN chmod +x install-global-packages.sh
RUN ./install-global-packages.sh

# PaStash Docker Container
FROM node:20-slim

RUN mkdir -p /config
COPY --from=builder /usr/local/lib/node_modules /usr/local/lib/node_modules
RUN ln -s /usr/local/lib/node_modules/@pastash/pastash/bin/pastash /usr/local/bin/pastash
WORKDIR /config

EXPOSE 9999/tcp 514/udp 9999/udp 514/tcp
CMD [ "/usr/local/bin/pastash", "--config_dir", "/config" ]

and install-global-packages.sh

#!/bin/bash

# Exit immediately if a command exits with a non-zero status
set -e

# Check if package.json exists
if [ ! -f package.json ]; then
  echo "package.json not found!"
  exit 1
fi

npm install --unsafe-perm -g @pastash/pastash @pastash/output_loki @pastash/filter_app_audiocodes

# Extract dependencies from package.json and install them globally
jq -r '.dependencies | keys[]' package.json | while read -r package; do
  npm install -g "$package"
done

echo "Global npm packages installed successfully."

and finally good logs ;-) 

[Fri, 23 Aug 2024 12:14:21 GMT] NOTICE Starting pastash 1.0.82
[Fri, 23 Aug 2024 12:14:21 GMT] INFO Max http socket 100
[Fri, 23 Aug 2024 12:14:21 GMT] INFO Loading config files from : /config
[Fri, 23 Aug 2024 12:14:21 GMT] INFO Files loaded from directory, 5 urls found
[Fri, 23 Aug 2024 12:14:21 GMT] INFO Loading config : 8 urls
[Fri, 23 Aug 2024 12:14:21 GMT] INFO Initializing output Stdout
Loading npm module... @pastash/output_loki
[Fri, 23 Aug 2024 12:14:21 GMT] INFO Initializing output Loki
[Fri, 23 Aug 2024 12:14:21 GMT] INFO Start HTTP output to LOKI http qryn:3100/loki/api/v1/push
[Fri, 23 Aug 2024 12:14:21 GMT] INFO Initializing filter AddHost
[Fri, 23 Aug 2024 12:14:21 GMT] INFO Initializing filter AddTimestamp
[Fri, 23 Aug 2024 12:14:21 GMT] INFO Initializing filter AddVersion
[Fri, 23 Aug 2024 12:14:21 GMT] INFO Initializing filter ComputeField
[Fri, 23 Aug 2024 12:14:21 GMT] INFO Initialized compute field filter on field: origin, value: AUDIOCODES
[Fri, 23 Aug 2024 12:14:21 GMT] INFO Initializing filter Grok
[Fri, 23 Aug 2024 12:14:21 GMT] INFO Initializing grok filter, pattern: <%{NUMBER:syslog_pri}>\[S=%{NUMBER:session_id}\] \|%{WORD:action}\s*\|%{DATA:source}\s*\|%{NUMBER:code1}\s*\|%{NUMBER:code2}\s*\|%{DATA:unique_id}\s*%{DATA:unique_id}\s*\|%{TIME:time}\s+%{WORD:timezone} %{WORD:day} %{MONTH:month} %{MONTHDAY:monthday} %{YEAR:year}\|%{NUMBER:value1}\s*\|%{NUMBER:value2}\s*\|%{WORD:timezone_final}\s*\|%{NUMBER:chiamante}\s*\|%{NUMBER:chiamante2}\s*\|%{NUMBER:chiamato}\s*\|%{NUMBER:chiamato2}\s*\|%{IP:ip_src}\s*\|%{IP:ip_dst}\s*\|%{DATA:call_status}\s*\|%{WORD:sip_method}\s*\|%{DATA:provider_src}\s*\|%{DATA:provider_dst}\s*\|%{NUMBER:flag}
[Fri, 23 Aug 2024 12:14:21 GMT] INFO Loading grok patterns
[Fri, 23 Aug 2024 12:14:21 GMT] INFO Grok patterns loaded from patterns directories 188
[Fri, 23 Aug 2024 12:14:21 GMT] INFO Initializing input Udp
[Fri, 23 Aug 2024 12:14:21 GMT] INFO Start listening on udp 0.0.0.0:514
[Fri, 23 Aug 2024 12:14:21 GMT] INFO Config loaded.
lmangani commented 2 months ago

Glad you got it working the way you wanted 👍