search

sipcapture / paStash

pastaʃ'ʃ = Spaghetti I/O Event Data Processing, Interpolation, Correlation and beyond :spaghetti:
http://sipcapture.io
Apache License 2.0
102 stars 27 forks source link

Using simply a grok output an error #79

Open le-ross opened 3 years ago

le-ross commented 3 years ago

Installed the app globaly and LOKI running locally

at first had this

input {
  file {
    path => "./logs/*.log"
  }
}

output {
  loki {
    host => localhost
    port => 3100
    path => "/loki/api/v1/push"
  }
}

and each time I insert logs I get [Wed, 20 Jan 2021 16:34:15 GMT] ERROR error Wrong HTTP Post return code: 400

So I taught maybe I need to parse better each line with this :

filter { 
  grok { 
    match => '%{TIMESTAMP_ISO8601:timestamp}: %{GREEDYDATA:message}' 
  }
}

Now I only get this error of loading module.

Loading npm module... @pastash/filter_grok
[Wed, 20 Jan 2021 16:16:13 GMT] ERROR Unable to load urls from command line
[Wed, 20 Jan 2021 16:16:13 GMT] ERROR TypeError: Cannot read property 'create' of undefined
    at LogstashAgent.configure (C:\Users\LeChat\AppData\Roaming\npm\node_modules\@pastash\pastash\lib\agent.js:143:21)
    at LogstashAgent.<anonymous> (C:\Users\LeChat\AppData\Roaming\npm\node_modules\@pastash\pastash\lib\agent.js:215:10)
    at C:\Users\LeChat\AppData\Roaming\npm\node_modules\@pastash\pastash\node_modules\async\dist\async.js:3096:16
    at replenish (C:\Users\LeChat\AppData\Roaming\npm\node_modules\@pastash\pastash\node_modules\async\dist\async.js:998:17)
    at iterateeCallback (C:\Users\LeChat\AppData\Roaming\npm\node_modules\@pastash\pastash\node_modules\async\dist\async.js:983:17)
    at C:\Users\LeChat\AppData\Roaming\npm\node_modules\@pastash\pastash\node_modules\async\dist\async.js:958:16
    at LogstashAgent.<anonymous> (C:\Users\LeChat\AppData\Roaming\npm\node_modules\@pastash\pastash\lib\agent.js:265:5)
    at LogstashAgent.<anonymous> (C:\Users\LeChat\AppData\Roaming\npm\node_modules\@pastash\pastash\lib\agent.js:220:7)
    at LogstashAgent.<anonymous> (C:\Users\LeChat\AppData\Roaming\npm\node_modules\@pastash\pastash\lib\agent.js:156:9)
    at FilterAddVersion.<anonymous> (C:\Users\LeChat\AppData\Roaming\npm\node_modules\@pastash\pastash\lib\lib\base_filter.js:36:5)

I dont have the easier time using that application. what I am missing

lmangani commented 3 years ago

Could you paste your full pastash config? Which version of Loki are you running? (or is it the one above?)

lmangani commented 3 years ago

The grok filter is indeed no longer part of the default build - but that's a parallel issue - please post the loki errors and/or capture the socket traffic with ngrep -W byline port 3100 -d any and attach the output during the failure.

lmangani commented 3 years ago

I also suspect your target system might lack the build tools required to compile the grok module - this might cause the 2nd failure.

le-ross commented 3 years ago

I started to work with promtail to see if I can make it work and now I suspect it's the same error 400

promtail_1 | level=error ts=2021-01-20T20:06:26.4438654Z caller=client.go:305 component=client host=loki:3100 msg=“final error sending batch” status=400 error=“server returned HTTP status 400 Bad Request (400): entry for stream '{filename=”/var/log/AAAAAA.log", job=“varlogs”}’ has timestamp too old: 2020-12-19 12:54:30 +0000 UTC"

I dont understand why this error. It's weird that I cannot ingest old logs.

For the second error I am still looking.

le-ross commented 3 years ago

Am I supposed to install like the output_loki? because I dont see on npm or the in repo.

$ npm install --unsafe-perm -g @pastash/filter_grok npm ERR! code E404 npm ERR! 404 Not Found - GET https://registry.npmjs.org/@pastash%2ffilter_grok - Not found npm ERR! 404 npm ERR! 404 '@pastash/filter_grok@latest' is not in the npm registry. npm ERR! 404 You should bug the author to publish it (or use the name yourself!) npm ERR! 404 npm ERR! 404 Note that you can also install from a npm ERR! 404 tarball, folder, http url, or git url. npm ERR! A complete log of this run can be found in: npm ERR! C:\Users\LeChat\AppData\Roaming\npm-cache\_logs\2021-01-20T23_24_40_297Z-debug.log

le-ross commented 3 years ago

This is the logs when I install pastash. I dont see any reference for filters or grok

pastash logs.txt

lmangani commented 3 years ago

@le-ross I must be honest i never tried running this on windows. In either case, you do not need grok, the error is elsewhere. Please run ngrep -W byline port 3100 -d any and while running. Try inserting data into Loki (either way). Attach the output.

le-ross commented 3 years ago

I will need to install ubuntu becase I am not able to find ngrep for windows

I tried to wrap the package in a docker container like this

FROM node:12

RUN npm install -g @pastash/pastash @pastash/output_loki

WORKDIR /app

CMD pastash --config_file=./pastash_loki.json

It doesnt make sense that I keep having this filter_grok error also. are you sure it's not missing somewhere? I need that grok otherwise all my logs are insert with today's timestamp

I was able to insert the logs with graylog instead of loki to see that all I am missing is the grok filter

le-ross commented 3 years ago

tried under my ubuntu session. same grok error.

[Thu, 21 Jan 2021 20:03:38 GMT] NOTICE Starting pastash 1.0.71
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Max http socket 100
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Loading config file : ./pastash-config.json
[Thu, 21 Jan 2021 20:03:38 GMT] INFO File loaded, 4 urls found
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Loading config : 7 urls
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Initializing output Gelf
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Start output to  Gelf localhost:12201
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Initializing output file
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Start output to file ./tmp/file.log
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Initializing filter AddHost
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Initializing filter AddTimestamp
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Initializing filter AddVersion
Loading npm module... @pastash/filter_grok
[Thu, 21 Jan 2021 20:03:38 GMT] ERROR Unable to load urls from command line
[Thu, 21 Jan 2021 20:03:38 GMT] ERROR TypeError: Cannot read property 'create' of undefined
    at LogstashAgent.configure (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/lib/agent.js:143:21)
    at LogstashAgent.<anonymous> (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/lib/agent.js:215:10)
    at /home/rossm/.npm-global/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:3096:16
    at replenish (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:998:17)
    at iterateeCallback (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:983:17)
    at /home/rossm/.npm-global/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:958:16
    at LogstashAgent.<anonymous> (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/lib/agent.js:265:5)
    at LogstashAgent.<anonymous> (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/lib/agent.js:220:7)
    at LogstashAgent.<anonymous> (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/lib/agent.js:156:9)
    at FilterAddVersion.<anonymous> (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/lib/lib/base_filter.js:36:5)
lmangani commented 3 years ago

Do you have build-essential meta package installed?

lmangani commented 3 years ago

Btw this still does not explain the error you're receiving you're receiving: 

each time I insert logs I get [Wed, 20 Jan 2021 16:34:15 GMT] ERROR error Wrong HTTP Post return code: 400

You forgot to run the ngrep command to see what we're sending out. Is the date on your server in sync with the world?

le-ross commented 3 years ago

Oh yeah, got distracted when I saw a image ready to use. I don't have the problem of grok with that image which is version 1.0.50

here is the log with a dummy file also attached dummy.log

interface: any
filter: ( port 3100 ) and (ip || ip6)
####################################
T 127.0.0.1:50314 -> 127.0.0.1:3100 [AP] #36
POST /loki/api/v1/push HTTP/1.1.
Content-Type: application/json.
Content-Length: 354.
Host: localhost:3100.
Connection: close.
.
{"streams":[{"labels":"{path=\"./logs/dummy.log\",type=\"undefined\",host=\"****\"}","entries":[{"ts":"2021-01-21T21:40:06.820Z","line":"2021-01-20T14:01:29: Job status 87f7c37f-e8df-427f-b810-xxxx STOPPED RUNNING 2"},{"ts":"2021-01-21T21:40:06.820Z","line":"2021-01-20T14:01:31: Job status 87f7c37f-e8df-427f-b810-xxxx STOPPED RUNNING 2"}]}]}
########
T 172.19.0.1:51500 -> 172.19.0.3:3100 [AP] #44
POST /loki/api/v1/push HTTP/1.1.
Content-Type: application/json.
Content-Length: 354.
Host: localhost:3100.
Connection: close.
.
{"streams":[{"labels":"{path=\"./logs/dummy.log\",type=\"undefined\",host=\"****\"}","entries":[{"ts":"2021-01-21T21:40:06.820Z","line":"2021-01-20T14:01:29: Job status 87f7c37f-e8df-427f-b810-xxxx STOPPED RUNNING 2"},{"ts":"2021-01-21T21:40:06.820Z","line":"2021-01-20T14:01:31: Job status 87f7c37f-e8df-427f-b810-xxxx STOPPED RUNNING 2"}]}]}
#
T 172.19.0.1:51500 -> 172.19.0.3:3100 [AP] #45
POST /loki/api/v1/push HTTP/1.1.
Content-Type: application/json.
Content-Length: 354.
Host: localhost:3100.
Connection: close.
.
{"streams":[{"labels":"{path=\"./logs/dummy.log\",type=\"undefined\",host=\"****\"}","entries":[{"ts":"2021-01-21T21:40:06.820Z","line":"2021-01-20T14:01:29: Job status 87f7c37f-e8df-427f-b810-xxxx STOPPED RUNNING 2"},{"ts":"2021-01-21T21:40:06.820Z","line":"2021-01-20T14:01:31: Job status 87f7c37f-e8df-427f-b810-xxxx STOPPED RUNNING 2"}]}]}
###
T 172.19.0.3:3100 -> 172.19.0.1:51500 [AP] #48
HTTP/1.1 204 No Content.
Date: Thu, 21 Jan 2021 21:40:08 GMT.
Connection: close.
.

#
T 172.19.0.3:3100 -> 172.19.0.1:51500 [AP] #49
HTTP/1.1 204 No Content.
Date: Thu, 21 Jan 2021 21:40:08 GMT.
Connection: close.
.

###
T 127.0.0.1:3100 -> 127.0.0.1:50314 [AP] #52
HTTP/1.1 204 No Content.
Date: Thu, 21 Jan 2021 21:40:08 GMT.
Connection: close.
.

##########
le-ross commented 3 years ago

here is my docker-compose in case

version: "3"

networks:
  loki:

services:
  loki:
    image: grafana/loki:latest
    ports:
      - 3100:3100
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki

  promtail:
    image: grafana/promtail:latest
    command: -config.file=/etc/promtail/config.yml
    networks:
      - loki

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    networks:
      - loki
lmangani commented 3 years ago

204 is good. From the logs it looks as if they were accepted

HTTP/1.1 204 No Content.
Date: Thu, 21 Jan 2021 21:40:08 GMT.
Connection: close.
le-ross commented 3 years ago

manged to get ngrep with the error 400

interface: any
filter: ( port 3100 ) and (ip || ip6)
#######
T 172.19.0.5:60510 -> 172.19.0.4:3100 [AP] #7
POST /loki/api/v1/push HTTP/1.1.
Content-Type: application/json.
Content-Length: 852.
Host: loki:3100.
Connection: close.
.
{"streams":[{"labels":"{path=\"/var/log/dummy.asdd.log\",type=\"undefined\",host=\"7d46bb4ba73c\"}","entries":[{"ts":"2021-01-22T01:38:13.725Z","line":"T01:57:49: 12 Sep 01:57:49 - [info] [udp out:26d9cdad.cf09d2] udp re-use socket: 4562 -> :8363"},{"ts":"2021-01-22T01:38:13.727Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:23670e42.f33822] udp re-use socket:  -> 127.0.0.1:6250"},{"ts":"2021-01-22T01:38:13.727Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:af1ac3dc.034b7] udp re-use socket: 4562 -> :8363"},{"ts":"2021-01-22T01:38:13.727Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:67ac2c51.a054f4] udp re-use socket:  -> localhost:4562"},{"ts":"2021-01-22T01:38:13.728Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:e9198db3.2a23c] udp re-use socket: 4562 -> :8363"}]}]}
#
T 172.19.0.5:60510 -> 172.19.0.4:3100 [AP] #8
POST /loki/api/v1/push HTTP/1.1.
Content-Type: application/json.
Content-Length: 852.
Host: loki:3100.
Connection: close.
.
{"streams":[{"labels":"{path=\"/var/log/dummy.asdd.log\",type=\"undefined\",host=\"7d46bb4ba73c\"}","entries":[{"ts":"2021-01-22T01:38:13.725Z","line":"T01:57:49: 12 Sep 01:57:49 - [info] [udp out:26d9cdad.cf09d2] udp re-use socket: 4562 -> :8363"},{"ts":"2021-01-22T01:38:13.727Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:23670e42.f33822] udp re-use socket:  -> 127.0.0.1:6250"},{"ts":"2021-01-22T01:38:13.727Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:af1ac3dc.034b7] udp re-use socket: 4562 -> :8363"},{"ts":"2021-01-22T01:38:13.727Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:67ac2c51.a054f4] udp re-use socket:  -> localhost:4562"},{"ts":"2021-01-22T01:38:13.728Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:e9198db3.2a23c] udp re-use socket: 4562 -> :8363"}]}]}
###
T 172.19.0.4:3100 -> 172.19.0.5:60510 [AP] #11
HTTP/1.1 400 Bad Request.
Content-Type: text/plain; charset=utf-8.
X-Content-Type-Options: nosniff.
Date: Fri, 22 Jan 2021 01:38:14 GMT.
Content-Length: 101.
Connection: close.
.
error parsing labels: parse error at line 1, col 2: syntax error: unexpected }, expecting IDENTIFIER

#
T 172.19.0.4:3100 -> 172.19.0.5:60510 [AP] #12
HTTP/1.1 400 Bad Request.
Content-Type: text/plain; charset=utf-8.
X-Content-Type-Options: nosniff.
Date: Fri, 22 Jan 2021 01:38:14 GMT.
Content-Length: 101.
Connection: close.
.
error parsing labels: parse error at line 1, col 2: syntax error: unexpected }, expecting IDENTIFIER

########^Cexit
le-ross commented 3 years ago

it's enough to throw that error with that data

2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] A
2020-09-12T01:57:50: 12 Sep 01:57:49 - [info] B
2020-09-12T01:57:51: 12 Sep 01:57:49 - c
2020-09-12T01:57:52: 12 Sep 01:57:49 d
2020-09-12T01:57:53: 12 Sep 01:e
2020-09-12T01:57:54: 12 Sef
2020-09-12T01:57:55: g
spady7 commented 1 year ago

I get this error too. Is this resolved? [Wed, 02 Aug 2023 09:51:09 GMT] NOTICE Starting pastash 1.0.75 [Wed, 02 Aug 2023 09:51:09 GMT] INFO Max http socket 100 [Wed, 02 Aug 2023 09:51:09 GMT] INFO Loading config file : /opt/pastash/sonus.conf [Wed, 02 Aug 2023 09:51:09 GMT] INFO File loaded, 7 urls found [Wed, 02 Aug 2023 09:51:09 GMT] INFO Loading config : 10 urls [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initializing output HEP/EEP Server [Wed, 02 Aug 2023 09:51:09 GMT] INFO Start output to HEP udp to 127.0.0.1:9060 [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initializing filter AddHost [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initializing filter AddTimestamp [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initializing filter AddVersion [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initializing filter Multiline [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initialized multiline filter with start_line_regex: /^\[\d{4}-\d{2}-\d{2}/, flags: Loading npm module... @pastash/filter_grok [Wed, 02 Aug 2023 09:51:09 GMT] ERROR Unable to load urls from command line [Wed, 02 Aug 2023 09:51:09 GMT] ERROR TypeError: Cannot read properties of undefined (reading 'create') at LogstashAgent.configure (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:143:21) at LogstashAgent.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:215:10) at /usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:3096:16 at replenish (/usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:998:17) at iterateeCallback (/usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:983:17) at /usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:958:16 at LogstashAgent.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:265:5) at LogstashAgent.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:220:7) at LogstashAgent.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:156:9) at FilterMultiline.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/lib/lib/base_filter.js:36:5)

Dletta commented 1 year ago

Could you provide the steps you took to get to this point? What config are you using?

What node version are you using?

On Wed, Aug 2, 2023, 04:56 spady7 @.***> wrote:

I get this error too. Is this resolved? [Wed, 02 Aug 2023 09:51:09 GMT] NOTICE Starting pastash 1.0.75 [Wed, 02 Aug 2023 09:51:09 GMT] INFO Max http socket 100 [Wed, 02 Aug 2023 09:51:09 GMT] INFO Loading config file : /opt/pastash/sonus.conf [Wed, 02 Aug 2023 09:51:09 GMT] INFO File loaded, 7 urls found [Wed, 02 Aug 2023 09:51:09 GMT] INFO Loading config : 10 urls [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initializing output HEP/EEP Server [Wed, 02 Aug 2023 09:51:09 GMT] INFO Start output to HEP udp to 127.0.0.1:9060 [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initializing filter AddHost [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initializing filter AddTimestamp [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initializing filter AddVersion [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initializing filter Multiline [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initialized multiline filter with start_line_regex: /^[\d{4}-\d{2}-\d{2}/, flags: Loading npm module... @pastash/filter_grok [Wed, 02 Aug 2023 09:51:09 GMT] ERROR Unable to load urls from command line [Wed, 02 Aug 2023 09:51:09 GMT] ERROR TypeError: Cannot read properties of undefined (reading 'create') at LogstashAgent.configure @./pastash/lib/agent.js:143:21) at LogstashAgent. @./pastash/lib/agent.js:215:10) at @./pastash/node_modules/async/dist/async.js:3096:16 at replenish @./pastash/node_modules/async/dist/async.js:998:17) at iterateeCallback @./pastash/node_modules/async/dist/async.js:983:17) at @./pastash/node_modules/async/dist/async.js:958:16 at LogstashAgent. @./pastash/lib/agent.js:265:5) at LogstashAgent. @./pastash/lib/agent.js:220:7) at LogstashAgent. @./pastash/lib/agent.js:156:9) at FilterMultiline. @./pastash/lib/lib/base_filter.js:36:5)

— Reply to this email directly, view it on GitHub https://github.com/sipcapture/paStash/issues/79#issuecomment-1661911671, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLWWESIR5H3G5Z64T64IB3XTIP4RANCNFSM4WK6DRVA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

spady7 commented 1 year ago

Hi @Dletta i need to use "pastash" to parse logs from Sonus/Ribbon SBC. i took configuration from wiki. Btw this is my conf:

`input { file { path => "/tmp/webui.log" } }

filter { multiline { start_line_regex => /^[\d{4}-\d{2}-\d{2}/ } grok { match => '(?m)[%{TIMESTAMP_ISO8601:timestamp}] %{WORD:pid} %{WORD:seq} \n%{GREEDYDATA:payload}' } regex { regex => /From:.\@(.)\:(\d+)/ fields => [srcIp,srcPort] } regex { regex => /To:.\@(.)\:(\d+)/ fields => [dstIp,dstPort] } regex { regex => /Call-I.: (.)/ fields => [correlation_id] } }

output { if [tags] != "_grokparsefailure" { hep { host => '127.0.0.1' port => 9060 hep_id => 2222 hep_type => 1 } } }`

And when i try

pastash --config_file=/opt/pastash/sonus.conf

i get errors.

spady7 commented 2 weeks ago

Has anyone found a solution to this problem? Get same error and If i try to install as plugin i get:

npm error code E404
npm error 404 Not Found - GET https://registry.npmjs.org/@pastash%2ffilter_grok - Not found
npm error 404
npm error 404  '@pastash/filter_grok@*' is not in this registry.
npm error 404
npm error 404 Note that you can also install from a
npm error 404 tarball, folder, http url, or git url.
npm error A complete log of this run can be found in: /root/.npm/_logs/2024-08-21T10_34_04_889Z-debug-0.log
Dletta commented 2 weeks ago

@spady7 This may occur if your path definition is insufficient.

Could you run pastash from the /usr/lib/node_modules/@pastash/pastash directory and see if it still occurs?

spady7 commented 2 weeks ago

@Dletta just tried, but is the same:

root@d9c9a162c2e3:/opt# /usr/local/lib/node_modules/\@pastash/pastash/bin/pastash --config_dir=/opt
[Wed, 21 Aug 2024 14:50:13 GMT] NOTICE Starting pastash 1.0.82
[Wed, 21 Aug 2024 14:50:13 GMT] INFO Max http socket 100
[Wed, 21 Aug 2024 14:50:13 GMT] INFO Loading config files from : /opt
[Wed, 21 Aug 2024 14:50:13 GMT] INFO Files loaded from directory, 4 urls found
[Wed, 21 Aug 2024 14:50:13 GMT] INFO Loading config : 7 urls
[Wed, 21 Aug 2024 14:50:13 GMT] INFO Initializing output Stdout
[Wed, 21 Aug 2024 14:50:13 GMT] INFO Initializing output file
[Wed, 21 Aug 2024 14:50:13 GMT] INFO Start output to file /usr/src/app/output_testing.json
[Wed, 21 Aug 2024 14:50:13 GMT] INFO Initializing filter AddHost
[Wed, 21 Aug 2024 14:50:13 GMT] INFO Initializing filter AddTimestamp
[Wed, 21 Aug 2024 14:50:13 GMT] INFO Initializing filter AddVersion
Loading npm module... @pastash/filter_grok
[Wed, 21 Aug 2024 14:50:13 GMT] ERROR Unable to load urls from command line
[Wed, 21 Aug 2024 14:50:13 GMT] ERROR TypeError: Cannot read properties of undefined (reading 'create')
    at LogstashAgent.configure (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:143:21)
    at LogstashAgent.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:215:10)
    at /usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:3113:16
    at replenish (/usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:1014:17)
    at /usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:1019:9
    at eachLimit$1 (/usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:3199:24)
    at Object.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:1049:16)
    at LogstashAgent.start_modules (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:214:9)
    at LogstashAgent.start_filters (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:255:8)
    at LogstashAgent.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:204:10)

If i use others filters they work. For instance app_audiocodes it working.

spady7 commented 2 weeks ago

Hi @Dletta I am investigating the problem thoroughly. First of all, I am using pastash in a docker container, where I have the problem. If I install pastash directly on a host machine (debian) everything works correctly. If I use the container, I have the problem described. I understood that when pastash starts, it loads the filters, if they are specified in the conf file, from the following folder: /usr/local/lib/nodemodules/@pastash/pastash/lib/filters where there are all the "filter*.js" filters. I tried, by modifying the configuration file used by pastash, to load other filters, eg "csv", "grep", etc and it always works. For example, if I rename the file "/usr/local/lib/node_modules/@pastash/pastash/lib/filters/filter_csv.js" to "/usr/local/lib/node_modules/@pastash/pastash/lib/filters/filter_csv_ORI.js" pastash gives me an error when it starts (rightly so because it can't find the specified file. That said, if I try to load only the grok filter I continue to have the problem. How is this possible?

Dletta commented 2 weeks ago

@spady7

Can you share your docker compose or Dockerfile, I 'd like to try myself

spady7 commented 2 weeks ago

@Dletta Sure, see following:

Dockerfile

FROM node:lts-alpine

# Install jq
RUN apk update && apk add --no-cache jq

RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

COPY package.json /usr/src/app/
COPY install-global-packages-alpine.sh /usr/src/app/

# Ensure the script is executable
RUN chmod +x install-global-packages-alpine.sh

# Run the script to install global npm packages
RUN ./install-global-packages-alpine.sh

RUN mkdir -p /config

EXPOSE 9999/tcp 514/udp 9999/udp 514/tcp
CMD [ "/usr/local/bin/pastash", "--config_dir", "/config" ]

Here a I used alpine node container as base. But tried also with "node:20-slim" but same issue. Let me know. Thanks

spady7 commented 2 weeks ago

@Dletta FYI i tried making container starting from qxip's official image (https://hub.docker.com/r/qxip/pastash/tags) and i get same error. 2024-08-22 15 37 37

root@47e8f2dc2cc7:/usr/src/app# node bin/pastash --log_level=debug --config_dir /config
[Thu, 22 Aug 2024 13:39:57 GMT] INFO Changing log_level debug
[Thu, 22 Aug 2024 13:39:57 GMT] NOTICE Starting pastash 1.0.81
[Thu, 22 Aug 2024 13:39:57 GMT] INFO Max http socket 100
[Thu, 22 Aug 2024 13:39:57 GMT] INFO Loading config files from : /config
[Thu, 22 Aug 2024 13:39:57 GMT] INFO Files loaded from directory, 4 urls found
[Thu, 22 Aug 2024 13:39:57 GMT] INFO Loading config : 7 urls
[Thu, 22 Aug 2024 13:39:57 GMT] DEBUG Loading urls [
  'filter://add_host://',
  'filter://add_timestamp://',
  'filter://add_version://',
  'input://udp://?host=0.0.0.0&port=30520&tags=CDR',
  'filter://grok://?match=%25%7BTIMESTAMP_ISO8601%3Atimestamp%7D%3A%20%25%7BGREEDYDATA%3Amessage%7D',
  'output://stdout://',
  'output://file://?path=%2Fusr%2Fsrc%2Fapp%2Foutput_testing.json&serializer=json_logstash'
]
[Thu, 22 Aug 2024 13:39:57 GMT] DEBUG Initializing module output
[Thu, 22 Aug 2024 13:39:57 GMT] INFO Initializing output Stdout
[Thu, 22 Aug 2024 13:39:57 GMT] DEBUG Initializing module output
[Thu, 22 Aug 2024 13:39:57 GMT] INFO Initializing output file
[Thu, 22 Aug 2024 13:39:57 GMT] INFO Start output to file /usr/src/app/output_testing.json
[Thu, 22 Aug 2024 13:39:57 GMT] DEBUG Initializing module filter
[Thu, 22 Aug 2024 13:39:57 GMT] INFO Initializing filter AddHost
[Thu, 22 Aug 2024 13:39:57 GMT] DEBUG Initializing module filter
[Thu, 22 Aug 2024 13:39:57 GMT] INFO Initializing filter AddTimestamp
[Thu, 22 Aug 2024 13:39:57 GMT] DEBUG Initializing module filter
[Thu, 22 Aug 2024 13:39:57 GMT] INFO Initializing filter AddVersion
[Thu, 22 Aug 2024 13:39:57 GMT] DEBUG Initializing module filter
[Thu, 22 Aug 2024 13:39:57 GMT] DEBUG no local module found filters filter_grok Error: Cannot find module 'filters/filter_grok'
Require stack:
- /usr/src/app/lib/agent.js
- /usr/src/app/bin/pastash
    at Module._resolveFilename (node:internal/modules/cjs/loader:1144:15)
    at Module._load (node:internal/modules/cjs/loader:985:27)
    at Module.require (node:internal/modules/cjs/loader:1235:19)
    at require (node:internal/modules/helpers:176:18)
    at LogstashAgent.configure (/usr/src/app/lib/agent.js:133:17)
    at LogstashAgent.<anonymous> (/usr/src/app/lib/agent.js:215:10)
    at /usr/src/app/node_modules/async/dist/async.js:3113:16
    at replenish (/usr/src/app/node_modules/async/dist/async.js:1014:17)
    at /usr/src/app/node_modules/async/dist/async.js:1019:9
    at eachLimit$1 (/usr/src/app/node_modules/async/dist/async.js:3199:24) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '/usr/src/app/lib/agent.js', '/usr/src/app/bin/pastash' ]
}
Loading npm module... @pastash/filter_grok
[Thu, 22 Aug 2024 13:39:57 GMT] DEBUG NestedError: Could not require module '@pastash/filter_grok'
    at requireg (/usr/src/app/node_modules/requireg/lib/requireg.js:14:11)
    at LogstashAgent.configure (/usr/src/app/lib/agent.js:139:18)
    at LogstashAgent.<anonymous> (/usr/src/app/lib/agent.js:215:10)
    at /usr/src/app/node_modules/async/dist/async.js:3113:16
    at replenish (/usr/src/app/node_modules/async/dist/async.js:1014:17)
    at /usr/src/app/node_modules/async/dist/async.js:1019:9
    at eachLimit$1 (/usr/src/app/node_modules/async/dist/async.js:3199:24)
    at Object.<anonymous> (/usr/src/app/node_modules/async/dist/async.js:1049:16)
    at LogstashAgent.start_modules (/usr/src/app/lib/agent.js:214:9)
    at LogstashAgent.start_filters (/usr/src/app/lib/agent.js:255:8)
Caused By: TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received undefined
    at validateString (node:internal/validators:162:11)
    at Module.require (node:internal/modules/cjs/loader:1228:3)
    at require (node:internal/modules/helpers:176:18)
    at requireg (/usr/src/app/node_modules/requireg/lib/requireg.js:12:12)
    at LogstashAgent.configure (/usr/src/app/lib/agent.js:139:18)
    at LogstashAgent.<anonymous> (/usr/src/app/lib/agent.js:215:10)
    at /usr/src/app/node_modules/async/dist/async.js:3113:16
    at replenish (/usr/src/app/node_modules/async/dist/async.js:1014:17)
    at /usr/src/app/node_modules/async/dist/async.js:1019:9
    at eachLimit$1 (/usr/src/app/node_modules/async/dist/async.js:3199:24) {
  nested: TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received undefined
      at validateString (node:internal/validators:162:11)
      at Module.require (node:internal/modules/cjs/loader:1228:3)
      at require (node:internal/modules/helpers:176:18)
      at requireg (/usr/src/app/node_modules/requireg/lib/requireg.js:12:12)
      at LogstashAgent.configure (/usr/src/app/lib/agent.js:139:18)
      at LogstashAgent.<anonymous> (/usr/src/app/lib/agent.js:215:10)
      at /usr/src/app/node_modules/async/dist/async.js:3113:16
      at replenish (/usr/src/app/node_modules/async/dist/async.js:1014:17)
      at /usr/src/app/node_modules/async/dist/async.js:1019:9
      at eachLimit$1 (/usr/src/app/node_modules/async/dist/async.js:3199:24) {
    code: 'ERR_INVALID_ARG_TYPE'
  }
}
[Thu, 22 Aug 2024 13:39:57 GMT] ERROR Unable to load urls from command line
[Thu, 22 Aug 2024 13:39:57 GMT] ERROR TypeError: Cannot read properties of undefined (reading 'create')
    at LogstashAgent.configure (/usr/src/app/lib/agent.js:143:21)
    at LogstashAgent.<anonymous> (/usr/src/app/lib/agent.js:215:10)
    at /usr/src/app/node_modules/async/dist/async.js:3113:16
    at replenish (/usr/src/app/node_modules/async/dist/async.js:1014:17)
    at /usr/src/app/node_modules/async/dist/async.js:1019:9
    at eachLimit$1 (/usr/src/app/node_modules/async/dist/async.js:3199:24)
    at Object.<anonymous> (/usr/src/app/node_modules/async/dist/async.js:1049:16)
    at LogstashAgent.start_modules (/usr/src/app/lib/agent.js:214:9)
    at LogstashAgent.start_filters (/usr/src/app/lib/agent.js:255:8)
    at LogstashAgent.<anonymous> (/usr/src/app/lib/agent.js:204:10)