search

mentat-is / gulp

g(ULP) - graphical universal log processor
https://gulp.sh
Other
20 stars 3 forks source link
cybersecurity dfir incident-response log-analysis threat-hunting

readme

gULP the generic, universal Log processor for incident response! _made with :heart: by Mentat._

GitHub followers

[Description](#description) - [Architecture](#architecture) - [Installation](#installation) - [Run & examples](#run) - [GUI](#clients) - [Troubleshooting](./docs/Troubleshooting.md)

Description

Gulp is a powerful software tool designed to streamline incident response and analysis. Its core features include:

Current features

Timeline

Here's the rough timeline we put together, it is subject to change:

timeline

Moca 2024 presentation

here is our presentation at MOCA2024.

Installation

TLDR ;)

:warning: WARNING
READ THIS TO INSTALL until we have a fix :)

docker

this will start the provided docker-compose.yml in the current directory and uses gulp_cfg_template.json and default .env as base.

curl https://raw.githubusercontent.com/mentat-is/gulp/refs/heads/develop/bootstrap.sh -o ./bootstrap.sh && chmod 755 ./bootstrap.sh && ./bootstrap.sh

from source

this will install from sources and create a gulp folder, inside the current directory.

curl https://raw.githubusercontent.com/mentat-is/gulp/refs/heads/develop/setup.sh | sudo bash

Exposed services

using the default configuration

Installation details

Environment variables

the following environment variables may be set to override configuration options.

SSL

to use HTTPS, the following certificates must be available:

client certificates for opensearch and postgresql are used if found, opensearch key password is not supported.

Run

[with docker](<./docs/Install Docker.md#run-with-docker-compose>) or [with install from sources](<./docs/Install Dev.md#7-run>)

Test ingestion

# ingest single file
TEST_WS_ID="websocket_id" ./test_scripts/test_ingest.sh -p ./samples/win_evtx/Application_no_crc32.evtx

# ingest zip
TEST_WS_ID="websocket_id" ./test_scripts/test_ingest.sh -p ./test_scripts/test_upload_with_metadata_json.zip

# ingest with filter (GulpIngestFilter)
TEST_WS_ID="websocket_id" TEST_INGESTION_FILTER='{"level":[2]}' ./test_scripts/test_ingest.sh -p ./test_scripts/test_upload_with_metadata_json.zip -f

## multiple concurrent ingestion (using 2 target websockets)
TEST_WS_ID="websocket_id1" TEST_TOKEN=80d5ed1e-c30f-4926-872d-92bcc5a2235d ./test_scripts/test_ingest.sh -p ./samples/win_evtx && TEST_WS_ID="websocket_id2" TEST_TOKEN=80d5ed1e-c30f-4926-872d-92bcc5a2235d TEST_CONTEXT=context2 TEST_OPERATION=testop2 ./test_scripts/test_ingest.sh -p ./samples/win_evtx

# ingest with csv plugin without mapping file ("@timestamp" in the document is mandatory, so in one way or another we must know at least how to obtain a timestamp for ingested event)
TEST_WS_ID="websocket_id" TEST_PLUGIN_PARAMS='{"timestamp_field": "UpdateTimestamp"}' TEST_PLUGIN=csv ./test_scripts/test_ingest.sh -p ./samples/mftecmd/sample_j.csv

# ingest with csv plugin with mapping file
TEST_WS_ID="websocket_id" TEST_PLUGIN_PARAMS='{"mapping_file": "mftecmd_csv.json", "mapping_id": "j"}' TEST_PLUGIN=csv ./test_scripts/test_ingest.sh -p ./samples/mftecmd/sample_j.csv

# example overriding a configuration parameter (will stay overridden until gulp restarts)
TEST_WS_ID="websocket_id" TEST_PLUGIN_PARAMS='{"mapping_file": "mftecmd_csv.json", "mapping_id": "j", "config_override": { "debug_allow_any_token_as_admin": true }' TEST_PLUGIN=csv ./test_scripts/test_ingest.sh -p ./samples/mftecmd/sample_j.csv

# ingest local directory (just for testing, not available in production code and not available when gulp runs in docker)
TEST_WS_ID="websocket_id" ./test_scripts/test_ingest.sh -x -p ./samples/win_evtx

# sample ingestion with filter
TEST_INGESTION_FILTER='{"start_msec":1475719436055, "end_msec": 1475719436211}' TEST_WS_ID=abc ./test_scripts/test_ingest.sh -p ./samples/win_evtx -f

# same as above, send filtered chunks on websocket but store data anyway on database (do not apply filter on-store)
TEST_INGESTION_FILTER='{"start_msec":1475719436055, "end_msec": 1475719436211, "store_all_documents": true}' TEST_WS_ID=abc ./test_scripts/test_ingest.sh -p ./samples/win_evtx -f

  • for testing websocket, a browser extension i.e. chrome websockets browser extension may be used.

    • if TEST_WS_ID is not specified, ingestion happens anyway, but results are not broadcasted.

  • resuming ingestion is supported if the req_id parameter is the same across requests.

Architecture

GULP architecture

Clients

Web Client

If you want to use the web GUI visit it's repository here. It's still heavily under construction, so any contribution is highly appreciated!

This is the intended official wanna-be client for gulp.

.NET Client

[!IMPORTANT] ⚠ the .NET client is to be considered an internal-test version, this is the tool we use internally to testdrive the backend development. It is not feature complete and far from being production ready :). Feel free to open issues, but any contribution should go towards the new WIP web client

If you want to use the .NET client "as-is", you can use the Windows binary from here.

To run the executable on Linux, follow these steps:

  1. start gulp backend with
# add --reset-collab --reset-elastic indexname if reset/initialization is needed
gulp --bind-to 0.0.0.0 8080

  1. start the UI

    on arch linux (endeavouros), other OS should be similar (just package names changes, if any)

    paru -S wine wine-mono winetricks
winetricks -q dotnetdesktop6
chmod 755 ./GUiLP.exe
./GUiLP.exe

  2. make sure the ui points to http://localhost:8080

select host

  1. profit!

UI