sipcapture / paStash

pastaʃ'ʃ = Spaghetti I/O Event Data Processing, Interpolation, Correlation and beyond :spaghetti:
http://sipcapture.io
Apache License 2.0
102 stars 28 forks source link

Ingate Plugin to enable syslog -> HEP for Ingate firmwares. #93

Closed systemcrash closed 5 months ago

systemcrash commented 3 years ago

Adds filter_app_ingate.js and an example conf for TCP and UDP.

lmangani commented 3 years ago

Thank you Paul @systemcrash for this wonderful, full round contribution!

lmangani commented 3 years ago

Q: Any reason this should not be a plugin rather than a native module? Most optional parsers are published as plugins.

systemcrash commented 3 years ago

Q: Any reason this should not be a plugin rather than a native module? Most optional parsers are published as plugins.

I wrote it two years ago while everything was like that. I tested it with master and it still works fine. Are there any important things I should be aware of to make it as a plugin? Or is it just a question of changing folders? I tried just changing folders and got some weird stackdumps>

Loading npm module... @pastash/filter_app_ingate
[Thu, 27 May 2021 12:16:33 GMT] ERROR Unable to load urls from command line
[Thu, 27 May 2021 12:16:33 GMT] ERROR TypeError: Cannot read property 'create' of undefined
    at LogstashAgent.configure (/usr/src/app/lib/agent.js:143:21)
    at LogstashAgent.<anonymous> (/usr/src/app/lib/agent.js:215:10)
    at /usr/src/app/node_modules/async/dist/async.js:3096:16
    at replenish (/usr/src/app/node_modules/async/dist/async.js:998:17)
    at iterateeCallback (/usr/src/app/node_modules/async/dist/async.js:983:17)
    at /usr/src/app/node_modules/async/dist/async.js:958:16
    at LogstashAgent.<anonymous> (/usr/src/app/lib/agent.js:265:5)
    at LogstashAgent.<anonymous> (/usr/src/app/lib/agent.js:220:7)
    at LogstashAgent.<anonymous> (/usr/src/app/lib/agent.js:156:9)
    at FilterMultiline.<anonymous> (/usr/src/app/lib/lib/base_filter.js:36:5)

🤷

changed also header to:

var base_filter = require('@pastash/pastash').base_filter,
  util = require('util'),
  logger = require('@pastash/pastash').logger;

which seems to be the source of the crash.

lmangani commented 3 years ago

It's a TINY bit more involved than just changing folders, but really just about importing the pastash base filter. You can take any relevant example plugin filter as example: https://github.com/sipcapture/paStash/tree/next/plugins/filters

var base_filter = require('@pastash/pastash').base_filter,  util = require('util'),  logger = require('@pastash/pastash').logger;<br class="Apple-interchange-newline">
systemcrash commented 3 years ago

It's a TINY bit more involved than just changing folders, but really just about importing the pastash base filter. You can take any relevant example plugin filter as example: https://github.com/sipcapture/paStash/tree/next/plugins/filters

var base_filter = require('@pastash/pastash').base_filter,  util = require('util'),  logger = require('@pastash/pastash').logger;<br class="Apple-interchange-newline">

I guess it should be - but... same crash with those imports (whether or not I move the js to plugins from lib/filter). I've been comparing to other plugins - some of the docs say I should install them - but I docker-compose up --build. Usually works fine, except for here 🤣

lmangani commented 3 years ago

No problem! I can give this a try this weekend, thanks again for contributing this back!!

systemcrash commented 3 years ago

Should we just mash it in and you try to pluginify it?

systemcrash commented 3 years ago

Did you manage to give it a try?

lmangani commented 3 years ago

@systemcrash more or less yeah, but things got busy - please keep this PR open and I'll share a gist - would still like for you to be able to PR it since its your work :)

systemcrash commented 3 years ago

Sooner the better :) but no pressure. Mainly curious what plugins mandate more than the old structure.

systemcrash commented 3 years ago

Did it get messy? 🍰

lmangani commented 3 years ago

@systemcrash you know how we do :)

I don't want to take credit for this, so just opened/closed to share what I thought would work in https://github.com/sipcapture/paStash/pull/96/files (I assume this is the same as you were doing, the only issue was the missing global regex? also ignore the mqtt patch clearly was there by mistake)

systemcrash commented 3 years ago

Any hints on where the missing global regex was? I try to put in place your changes, and just boom same crash. Tried destroying all images and rebuilding... same. Want to test in place and confirm that your 'simple' changes are those necessary.

lmangani commented 3 years ago

All plugins are the same really - is this crashing with @pastash/pastash or a local instance?

systemcrash commented 3 years ago

All plugins are the same really - is this crashing with @pastash/pastash or a local instance?

As above

lmangani commented 3 years ago

To actually check this as it would work once released, you would have to "plant" this in your global node_modules path alongside @pastash/pastash, ie /use/global/path/to/node_modules/@pastash/filter_app_ingate and then run the recipe

systemcrash commented 3 years ago

I'm building an image via docker-compose, and never had to do that before..... 🤷

It's the same result whichever app_* from plugins that I invoke. Nothing via docker-compose works, without some extra magic, which I do not know.

But it works as a core app.

Edit: maybe something from Dockerfile is missing. But basically, I don't know how to do

...you would have to "plant" this in your global node_modules path alongside @pastash/pastash, ie /use/global/path/to/node_modules/@pastash/filter_app_ingate and then run the recipe

systemcrash commented 3 years ago

Welp, I'm OK with this being a plugin, but I can't get paStash to work with any plugin in the container environment without a bit of extra guidance. It seems some special sauce to get those environments working automatically is necessary.

lmangani commented 3 years ago

Hey @systemcrash I promise to dedicate this some time asap but you are right, plugins are really hard to test before they are published to npm unless you manually "simulate" an npm global install by copying them into the expected path - will fix!

systemcrash commented 3 years ago

Hey @systemcrash I promise to dedicate this some time asap but you are right, plugins are really hard to test before they are published to npm unless you manually "simulate" an npm global install by copying them into the expected path - will fix!

Ok - thanks. I consider this low prio; it's not like people are screaming for syslog handling on older Ingate firmwares. But if 'simulating' manually is the only way to get paStash working with its own plugins....

CLAassistant commented 2 years ago

CLA assistant check
All committers have signed the CLA.