Syslifters / sysreptor

A customizable and powerful penetration testing reporting platform for offensive security professionals. Simplify, customize, and automate your pentest reports with ease.
https://docs.sysreptor.com
Other
1.48k stars 141 forks source link

[Urgent Enquiry] Unable to setup Caddy webserver locally to upload Nessus findings to Sysreptor (running locally as well) #303

Closed Zhiyang123 closed 2 months ago

Zhiyang123 commented 3 months ago

I followed the guide to run setup.sh from https://docs.sysreptor.com/setup/webserver/. I wanted to setup Caddy webserver listening on port 80 then forward it to Sysreptor which is listening on port 8000 (by default), without any certificates involved.

When I ran the command, these errors came up:

zy@ubuntu:~/Desktop$ cat ./nessus_v_unknown.nessus | reptor nessus --upload
Reading from stdin...
HTTPSConnectionPool(host='localhost', port=443): Max retries exceeded with url: /api/v1/pentestprojects/a8331730-b0aa-4b3c-a9ae-2344c1c2f822/notes/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)')))
zy@ubuntu:~/Desktop$ cat ./nessus_v_unknown.nessus | reptor nessus --upload -k
Reading from stdin...
405 Client Error: Method Not Allowed for url: https://localhost/api/v1/pentestprojects/a8331730-b0aa-4b3c-a9ae-2344c1c2f822/notes/

The setup.sh that I ran (cloned from sysreptor github repo):

#!/bin/bash
cd `dirname "$BASH_SOURCE"`
if
    test -f Caddyfile
then
    echo "Caddyfile exists. Skipping web server setup."
    read -p "Press any key to continue installation..."
    echo ""
    cd - >/dev/null
    return 2>/dev/null || exit -1  # return if script is source, exit if in process
fi
while [[ "$SYSREPTOR_WEBSERVER" != [yY] && "$SYSREPTOR_WEBSERVER" != [nN] ]]
do
    echo ""
    echo "SysReptor runs on localhost (127.0.0.1) by default."
    read -p "Should we setup a webserver (Caddy in Docker) for you to expose it to your local network or the Internet? [y/n]: " SYSREPTOR_WEBSERVER
    if [[ "$SYSREPTOR_WEBSERVER" == [yY] ]]
    then
        while [[ "$SYSREPTOR_LETSENCRYPT" != [yY] && "$SYSREPTOR_LETSENCRYPT" != [nN] ]]
        do
            echo ""
            echo "Should we take care of your webserver HTTPS certificate using LetsEncrypt?"
            echo "For this, you must set up:"
            echo " 1. a valid domain name resolving to your public IP address"
            echo " 2. port 80 of your must be publicly reachable"
            read -p "Want a LetsEncrypt webserver certificate? [y/n]: " SYSREPTOR_LETSENCRYPT
        done
        while ! case "$SYSREPTOR_CADDY_PORT" in ''|*[!0-9]*) false;;esac;
        do
            if [[ "$SYSREPTOR_LETSENCRYPT" == [yY] ]]
            then
                    default_port=443
            else
                    default_port=80
            fi
            read -p "What port should the webserver use? [$default_port] " SYSREPTOR_CADDY_PORT
            SYSREPTOR_CADDY_PORT=${SYSREPTOR_CADDY_PORT:-$default_port}
            if [[ "$SYSREPTOR_CADDY_PORT" -lt 1 || "$SYSREPTOR_CADDY_PORT" -gt 65535 ]]
            then
                echo "Invalid port number. Please enter a valid port number between 1 and 65535."
                SYSREPTOR_CADDY_PORT=""
            elif [[ "$SYSREPTOR_CADDY_PORT" -eq 8000 ]]
            then
                echo "The Django app uses port 8000 on 127.0.0.1. Please use a different port."
                SYSREPTOR_CADDY_PORT=""
            fi
        done
        if [[ "$SYSREPTOR_LETSENCRYPT" == [yY] ]]
        then
            while [ -z "$SYSREPTOR_CADDY_FQDN" ]
            do
                    read -p "What is your publicly reachable fully qualified domain name (e.g., sysreptor.example.com)? " SYSREPTOR_CADDY_FQDN
            done
        fi
    fi
done
if [[ "$SYSREPTOR_WEBSERVER" == [nN] ]]
then
    echo "Okay. Won't set up a webserver."
else
    # Create config
    echo """# Do not modify. This file is automatically generated.
# Changes will be overwritten.
$SYSREPTOR_CADDY_FQDN:$SYSREPTOR_CADDY_PORT

reverse_proxy http://127.0.0.1:8000""" > Caddyfile
    echo "Setting up your web server..."

    docker_compose_file="../docker-compose.yml"
    include_caddy="  - caddy/docker-compose.yml"
    if ! grep -q "^$include_caddy" "$docker_compose_file"
    then
        # Include Caddy in docker-compose.yml
        sed -i "s#include:#include:\n$include_caddy#" "$docker_compose_file"
    fi
fi
docker container stop sysreptor-caddy 1>/dev/null 2>&1 && docker container rm sysreptor-caddy 1>/dev/null 2>&1 || true
echo ""
cd - >/dev/null
return 2>/dev/null || true  # return if script is source

cd `dirname "$BASH_SOURCE"`
cd ..
docker compose up -d
cd - >/dev/null

config.yaml (in ./sysreptor)

zy@ubuntu:~/.sysreptor$ cat config.yaml 
project_id: a8331730-b0aa-4b3c-a9ae-2344c1c2f822
server: http://localhost
token: sysreptor_<redacted>

docker containers currently up and running

zy@ubuntu:~/Desktop$ docker ps
CONTAINER ID   IMAGE           COMMAND                  CREATED       STATUS                    PORTS                      NAMES
af199aad2194   sysreptor-app   "/bin/sh -c 'python3…"   2 weeks ago   Up 56 minutes (healthy)   127.0.0.1:8000->8000/tcp   sysreptor-app
587376d1c631   postgres:14     "docker-entrypoint.s…"   2 weeks ago   Up 56 minutes (healthy)   5432/tcp                   sysreptor-db
aronmolnar commented 3 months ago

Can you please run the command with --debug for more details?

cat ./nessus_v_unknown.nessus | reptor nessus --upload -k --debug


Von: Zhi Yang @.> Gesendet: Donnerstag, 15. August 2024 08:10 An: Syslifters/sysreptor @.> Cc: Subscribed @.***> Betreff: [Syslifters/sysreptor] [Urgent Enquiry] Unable to setup Caddy webserver locally to upload Nessus findings to Sysreptor (running locally as well) (Issue #303)

I followed the guide to run setup.sh from https://docs.sysreptor.com/setup/webserver/. I wanted to setup Caddy webserver listening on port 80 then forward it to Sysreptor which is listening on port 8000 (by default), without any certificates involved.

When I ran the command, these errors came up:

@.:~/Desktop$ cat ./nessus_v_unknown.nessus | reptor nessus --upload Reading from stdin... HTTPSConnectionPool(host='localhost', port=443): Max retries exceeded with url: /api/v1/pentestprojects/a8331730-b0aa-4b3c-a9ae-2344c1c2f822/notes/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)'))) @.:~/Desktop$ cat ./nessus_v_unknown.nessus | reptor nessus --upload -k Reading from stdin... 405 Client Error: Method Not Allowed for url: https://localhost/api/v1/pentestprojects/a8331730-b0aa-4b3c-a9ae-2344c1c2f822/notes/

The setup.sh that I ran (cloned from sysreptor github repo):

!/bin/bash

cd dirname "$BASH_SOURCE" if test -f Caddyfile then echo "Caddyfile exists. Skipping web server setup." read -p "Press any key to continue installation..." echo "" cd - >/dev/null return 2>/dev/null || exit -1 # return if script is source, exit if in process fi while [[ "$SYSREPTOR_WEBSERVER" != [yY] && "$SYSREPTOR_WEBSERVER" != [nN] ]] do echo "" echo "SysReptor runs on localhost (127.0.0.1) by default." read -p "Should we setup a webserver (Caddy in Docker) for you to expose it to your local network or the Internet? [y/n]: " SYSREPTOR_WEBSERVER if [[ "$SYSREPTOR_WEBSERVER" == [yY] ]] then while [[ "$SYSREPTOR_LETSENCRYPT" != [yY] && "$SYSREPTOR_LETSENCRYPT" != [nN] ]] do echo "" echo "Should we take care of your webserver HTTPS certificate using LetsEncrypt?" echo "For this, you must set up:" echo " 1. a valid domain name resolving to your public IP address" echo " 2. port 80 of your must be publicly reachable" read -p "Want a LetsEncrypt webserver certificate? [y/n]: " SYSREPTOR_LETSENCRYPT done while ! case "$SYSREPTOR_CADDY_PORT" in ''|[!0-9]) false;;esac; do if [[ "$SYSREPTOR_LETSENCRYPT" == [yY] ]] then default_port=443 else default_port=80 fi read -p "What port should the webserver use? [$default_port] " SYSREPTOR_CADDY_PORT SYSREPTOR_CADDY_PORT=${SYSREPTOR_CADDY_PORT:-$default_port} if [[ "$SYSREPTOR_CADDY_PORT" -lt 1 || "$SYSREPTOR_CADDY_PORT" -gt 65535 ]] then echo "Invalid port number. Please enter a valid port number between 1 and 65535." SYSREPTOR_CADDY_PORT="" elif [[ "$SYSREPTOR_CADDY_PORT" -eq 8000 ]] then echo "The Django app uses port 8000 on 127.0.0.1. Please use a different port." SYSREPTOR_CADDY_PORT="" fi done if [[ "$SYSREPTOR_LETSENCRYPT" == [yY] ]] then while [ -z "$SYSREPTOR_CADDY_FQDN" ] do read -p "What is your publicly reachable fully qualified domain name (e.g., sysreptor.example.com)? " SYSREPTOR_CADDY_FQDN done fi fi done if [[ "$SYSREPTOR_WEBSERVER" == [nN] ]] then echo "Okay. Won't set up a webserver." else

Create config

echo """# Do not modify. This file is automatically generated.

Changes will be overwritten.

$SYSREPTOR_CADDY_FQDN:$SYSREPTOR_CADDY_PORT

reverse_proxy http://127.0.0.1:8000""" > Caddyfile echo "Setting up your web server..."

docker_compose_file="../docker-compose.yml"
include_caddy="  - caddy/docker-compose.yml"
if ! grep -q "^$include_caddy" "$docker_compose_file"
then
    # Include Caddy in docker-compose.yml
    sed -i "s#include:#include:\n$include_caddy#" "$docker_compose_file"
fi

fi docker container stop sysreptor-caddy 1>/dev/null 2>&1 && docker container rm sysreptor-caddy 1>/dev/null 2>&1 || true echo "" cd - >/dev/null return 2>/dev/null || true # return if script is source

cd dirname "$BASH_SOURCE" cd .. docker compose up -d cd - >/dev/null

config.yaml (in ./sysreptor)

@.***:~/.sysreptor$ cat config.yaml projectid: a8331730-b0aa-4b3c-a9ae-2344c1c2f822 server: http://localhost token: sysreptor

docker containers currently up and running

@.***:~/Desktop$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES af199aad2194 sysreptor-app "/bin/sh -c 'python3…" 2 weeks ago Up 56 minutes (healthy) 127.0.0.1:8000->8000/tcp sysreptor-app 587376d1c631 postgres:14 "docker-entrypoint.s…" 2 weeks ago Up 56 minutes (healthy) 5432/tcp sysreptor-db

— Reply to this email directly, view it on GitHubhttps://github.com/Syslifters/sysreptor/issues/303, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGANCTYUAE7DQDGEEFD6QD3ZRRA6FAVCNFSM6AAAAABMRUKB5KVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQ3DONBXHA4TCMQ. You are receiving this because you are subscribed to this thread.Message ID: @.***>

Zhiyang123 commented 3 months ago

Here is the full debug output from the command stated above: Sysreptor Debug Output.md

I have put it in a markdown file because the debug logs are too much to input in here. Thank you for the help!

aronmolnar commented 3 months ago

Thank you for the debug output.
It seems like the Nessus report has some unusual format and our parser ends up having lists where indeed dictionaries should be.

I wasn't able to track down the root cause so far. Do you think it would be possible for you to anonymize the report (replace hostnames and ip addresses in the xml file) and send the report to hello@syslifters.com that we can further troubleshoot the issue?

Zhiyang123 commented 2 months ago

Hi, the nessus file that I used was just a sample from DefectDojo, as shown here: https://github.com/DefectDojo/sample-scan-files/blob/master/nessus/nessus_v_unknown.nessus. Could I check if the allowed file extension for nessus scan report is only strictly restricted to .nessus?

aronmolnar commented 2 months ago

I, unfortunately, cannot reproduce your error. After downloading the sample report and using reptor, everything seems to work as expected:

image

Can you check if you have the latest reptor version installed and upgrade if not (pip3 install --upgrade reptor).

The only supported file type for nessus scans is the .nessus extention, which must contain XML data.

Zhiyang123 commented 2 months ago

Could you share with me your config.yaml inside .sysreptor directory (with your token redacted), as well as the docker-compose.yml (for the Caddy webserver) so that I could troubleshoot it on my end? Thank you.

I was thinking if it could be due to the server being either on localhost or insecure (with http)?

p.s The reptor version has been upgraded to the latest, error still persists.

aronmolnar commented 2 months ago

A sample .config.yaml would be:

project_id: 1d2da818-913f-482f-a19c-d9eff0d128f0
server: https://reptortest.sysre.pt/
token: sysreptor_XXXXXXXXXXX

If you think that something is wrong with your Caddy setup, you can run caddy.sh from the SysReptor repo located in deploy/caddy/. This will use a Dockerfile shipped with SysReptor and add the container to your main docker-compose.yml.

My impression is that your SysReptor installation is not the problem.
Can you just try to upload a plain note?

> echo "Upload me" | reptor --notetitle "My Upload" note -k
Reading from stdin...
Note written to "My Upload".
Successfully uploaded to notes.

If this works, it rules out issues with your installation.

Zhiyang123 commented 2 months ago

The uploading of the plain note does not work on my end as shown:

zy@ubuntu:~/sysreptor/deploy/caddy$ echo "Upload me" | reptor --notetitle "My Upload" note -k
Reading from stdin...
'list' object has no attribute 'get'
Zhiyang123 commented 2 months ago

This is what happens when i ran ./setup.sh, please advise on which parts I have to debug.

zy@ubuntu:~/sysreptor/deploy/caddy$ ./setup.sh 
SysReptor runs on localhost (127.0.0.1) by default.
Should we setup a webserver (Caddy in Docker) for you to expose it to your local network or the Internet? [y/n]: y
Should we take care of your webserver HTTPS certificate using LetsEncrypt?
For this, you must set up:
 1. a valid domain name resolving to your public IP address
 2. port 80 of your must be publicly reachable
Want a LetsEncrypt webserver certificate? [y/n]: n
What port should the webserver use? [80] 
Setting up your web server...
service "app" has neither an image nor a build context specified: invalid compose project

Context: I don't have a valid domain name resolving to my public IP address, because I just want to test it locally. Please do let me know if thats the root cause.

aronmolnar commented 2 months ago

The uploading of the plain note does not work on my end as shown:

zy@ubuntu:~/sysreptor/deploy/caddy$ echo "Upload me" | reptor --notetitle "My Upload" note -k
Reading from stdin...
'list' object has no attribute 'get'

Can you run this command with --debug please and send me the output?

Zhiyang123 commented 2 months ago

Debug Information.md

Attached in this post is the file containing the debug output.

p.s I think the issue lies in the fact there is no response when my local machine is trying to establish a HTTPS connection with my localhost (at port 443).

aronmolnar commented 2 months ago

This is what happens when i ran ./setup.sh, please advise on which parts I have to debug.

zy@ubuntu:~/sysreptor/deploy/caddy$ ./setup.sh 
SysReptor runs on localhost (127.0.0.1) by default.
Should we setup a webserver (Caddy in Docker) for you to expose it to your local network or the Internet? [y/n]: y
Should we take care of your webserver HTTPS certificate using LetsEncrypt?
For this, you must set up:
 1. a valid domain name resolving to your public IP address
 2. port 80 of your must be publicly reachable
Want a LetsEncrypt webserver certificate? [y/n]: n
What port should the webserver use? [80] 
Setting up your web server...
service "app" has neither an image nor a build context specified: invalid compose project

Context: I don't have a valid domain name resolving to my public IP address, because I just want to test it locally. Please do let me know if thats the root cause.

You don't need a domain name - this is fine.

The error your receive is also quite strange. Did you change anything in docker-compose.yaml files?

Make sure to implement the following file structure, which should fix your setup.

Your deploy/docker-compose.yaml:

name: sysreptor

include:
  - sysreptor/docker-compose.yml
  - caddy/docker-compose.yaml

Your deploy/caddy/docker-compose.yaml: Here

Your deploy/sysreptor/docker-compose.yaml: Here

If you added/updated those files, run docker compose up -d from the deploy directory.
Make sure that all containers (app, redis, postgres, caddy) are running using docker ps

Zhiyang123 commented 2 months ago

This is my current directory at ~/sysreptor/deploy:

zy@ubuntu:~/sysreptor/deploy$ ls
app.env  app.env.example  caddy  docker-compose.override.yml  docker-compose.yml  setup.sh  sysreptor.nginx

Could I clarify if I am missing a sysreptor directory (deploy/sysreptor)?

aronmolnar commented 2 months ago

Okay, I think we approached the solution. Two things:

  1. Your SysReptor config seems to hold the server URL "http://127.0.0.1:80". This url, however, redirects to http://127.0.0.1:443. This again means, that the requests library follows the redirects, also for POST requests. This causes the note creation to fail, because the redirect is done using a GET request. Solution: Run reptor conf to update your server URL to https://127.0.0.1:443
  2. The setup I explained works with the latest SysReptor version. Your current version is outdated. You should be able to update your installation using update.sh from your SysReptor folder. (This, however, is not the reason why reptor commands fail.)

We will check for redirects in the next version of reptor to communicate this error more clearly.

Zhiyang123 commented 2 months ago

This is what happens when i ran ./setup.sh, please advise on which parts I have to debug.

zy@ubuntu:~/sysreptor/deploy/caddy$ ./setup.sh 
SysReptor runs on localhost (127.0.0.1) by default.
Should we setup a webserver (Caddy in Docker) for you to expose it to your local network or the Internet? [y/n]: y
Should we take care of your webserver HTTPS certificate using LetsEncrypt?
For this, you must set up:
 1. a valid domain name resolving to your public IP address
 2. port 80 of your must be publicly reachable
Want a LetsEncrypt webserver certificate? [y/n]: n
What port should the webserver use? [80] 
Setting up your web server...
service "app" has neither an image nor a build context specified: invalid compose project

Context: I don't have a valid domain name resolving to my public IP address, because I just want to test it locally. Please do let me know if thats the root cause.

You don't need a domain name - this is fine.

The error your receive is also quite strange. Did you change anything in docker-compose.yaml files?

Make sure to implement the following file structure, which should fix your setup.

Your deploy/docker-compose.yaml:

name: sysreptor

include:
  - sysreptor/docker-compose.yml
  - caddy/docker-compose.yaml

Your deploy/caddy/docker-compose.yaml: Here

Your deploy/sysreptor/docker-compose.yaml: Here

If you added/updated those files, run docker compose up -d from the deploy directory. Make sure that all containers (app, redis, postgres, caddy) are running using docker ps

Hi Aron, I managed to setup all the docker containers successfully. I have changed the server URL to https://127.0.0.1:443 and I managed to upload the notes perfectly as shown:

zy@ubuntu:~/sysreptor/deploy$ echo "Upload me" | reptor --notetitle "My Upload" note -k
Reading from stdin...
Note written to "My Upload".
Successfully uploaded to notes.

Thank you for the help and assistance, cheers to a good day for you!

aronmolnar commented 2 months ago

Great - I'm happy to hear this.
I'll keep this ticket open to track improvements of the error messages.

Zhiyang123 commented 2 months ago

So does that mean I won't have to close this issue, so that you can track improvements of the error messages?

aronmolnar commented 2 months ago

Fixed in https://github.com/Syslifters/reptor/commit/fbf28f888d82767e34b581f3bb07ad40fcbe7891