Closed Zhiyang123 closed 2 months ago
Can you please run the command with --debug for more details?
cat ./nessus_v_unknown.nessus | reptor nessus --upload -k --debug
Von: Zhi Yang @.> Gesendet: Donnerstag, 15. August 2024 08:10 An: Syslifters/sysreptor @.> Cc: Subscribed @.***> Betreff: [Syslifters/sysreptor] [Urgent Enquiry] Unable to setup Caddy webserver locally to upload Nessus findings to Sysreptor (running locally as well) (Issue #303)
I followed the guide to run setup.sh from https://docs.sysreptor.com/setup/webserver/. I wanted to setup Caddy webserver listening on port 80 then forward it to Sysreptor which is listening on port 8000 (by default), without any certificates involved.
When I ran the command, these errors came up:
@.:~/Desktop$ cat ./nessus_v_unknown.nessus | reptor nessus --upload Reading from stdin... HTTPSConnectionPool(host='localhost', port=443): Max retries exceeded with url: /api/v1/pentestprojects/a8331730-b0aa-4b3c-a9ae-2344c1c2f822/notes/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)'))) @.:~/Desktop$ cat ./nessus_v_unknown.nessus | reptor nessus --upload -k Reading from stdin... 405 Client Error: Method Not Allowed for url: https://localhost/api/v1/pentestprojects/a8331730-b0aa-4b3c-a9ae-2344c1c2f822/notes/
The setup.sh that I ran (cloned from sysreptor github repo):
cd dirname "$BASH_SOURCE"
if
test -f Caddyfile
then
echo "Caddyfile exists. Skipping web server setup."
read -p "Press any key to continue installation..."
echo ""
cd - >/dev/null
return 2>/dev/null || exit -1 # return if script is source, exit if in process
fi
while [[ "$SYSREPTOR_WEBSERVER" != [yY] && "$SYSREPTOR_WEBSERVER" != [nN] ]]
do
echo ""
echo "SysReptor runs on localhost (127.0.0.1) by default."
read -p "Should we setup a webserver (Caddy in Docker) for you to expose it to your local network or the Internet? [y/n]: " SYSREPTOR_WEBSERVER
if [[ "$SYSREPTOR_WEBSERVER" == [yY] ]]
then
while [[ "$SYSREPTOR_LETSENCRYPT" != [yY] && "$SYSREPTOR_LETSENCRYPT" != [nN] ]]
do
echo ""
echo "Should we take care of your webserver HTTPS certificate using LetsEncrypt?"
echo "For this, you must set up:"
echo " 1. a valid domain name resolving to your public IP address"
echo " 2. port 80 of your must be publicly reachable"
read -p "Want a LetsEncrypt webserver certificate? [y/n]: " SYSREPTOR_LETSENCRYPT
done
while ! case "$SYSREPTOR_CADDY_PORT" in ''|[!0-9]) false;;esac;
do
if [[ "$SYSREPTOR_LETSENCRYPT" == [yY] ]]
then
default_port=443
else
default_port=80
fi
read -p "What port should the webserver use? [$default_port] " SYSREPTOR_CADDY_PORT
SYSREPTOR_CADDY_PORT=${SYSREPTOR_CADDY_PORT:-$default_port}
if [[ "$SYSREPTOR_CADDY_PORT" -lt 1 || "$SYSREPTOR_CADDY_PORT" -gt 65535 ]]
then
echo "Invalid port number. Please enter a valid port number between 1 and 65535."
SYSREPTOR_CADDY_PORT=""
elif [[ "$SYSREPTOR_CADDY_PORT" -eq 8000 ]]
then
echo "The Django app uses port 8000 on 127.0.0.1. Please use a different port."
SYSREPTOR_CADDY_PORT=""
fi
done
if [[ "$SYSREPTOR_LETSENCRYPT" == [yY] ]]
then
while [ -z "$SYSREPTOR_CADDY_FQDN" ]
do
read -p "What is your publicly reachable fully qualified domain name (e.g., sysreptor.example.com)? " SYSREPTOR_CADDY_FQDN
done
fi
fi
done
if [[ "$SYSREPTOR_WEBSERVER" == [nN] ]]
then
echo "Okay. Won't set up a webserver."
else
echo """# Do not modify. This file is automatically generated.
$SYSREPTOR_CADDY_FQDN:$SYSREPTOR_CADDY_PORT
reverse_proxy http://127.0.0.1:8000""" > Caddyfile echo "Setting up your web server..."
docker_compose_file="../docker-compose.yml"
include_caddy=" - caddy/docker-compose.yml"
if ! grep -q "^$include_caddy" "$docker_compose_file"
then
# Include Caddy in docker-compose.yml
sed -i "s#include:#include:\n$include_caddy#" "$docker_compose_file"
fi
fi docker container stop sysreptor-caddy 1>/dev/null 2>&1 && docker container rm sysreptor-caddy 1>/dev/null 2>&1 || true echo "" cd - >/dev/null return 2>/dev/null || true # return if script is source
cd dirname "$BASH_SOURCE"
cd ..
docker compose up -d
cd - >/dev/null
config.yaml (in ./sysreptor)
@.***:~/.sysreptor$ cat config.yaml
projectid: a8331730-b0aa-4b3c-a9ae-2344c1c2f822
server: http://localhost
token: sysreptor
docker containers currently up and running
@.***:~/Desktop$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES af199aad2194 sysreptor-app "/bin/sh -c 'python3…" 2 weeks ago Up 56 minutes (healthy) 127.0.0.1:8000->8000/tcp sysreptor-app 587376d1c631 postgres:14 "docker-entrypoint.s…" 2 weeks ago Up 56 minutes (healthy) 5432/tcp sysreptor-db
— Reply to this email directly, view it on GitHubhttps://github.com/Syslifters/sysreptor/issues/303, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGANCTYUAE7DQDGEEFD6QD3ZRRA6FAVCNFSM6AAAAABMRUKB5KVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQ3DONBXHA4TCMQ. You are receiving this because you are subscribed to this thread.Message ID: @.***>
Here is the full debug output from the command stated above: Sysreptor Debug Output.md
I have put it in a markdown file because the debug logs are too much to input in here. Thank you for the help!
Thank you for the debug output.
It seems like the Nessus report has some unusual format and our parser ends up having lists where indeed dictionaries should be.
I wasn't able to track down the root cause so far. Do you think it would be possible for you to anonymize the report (replace hostnames and ip addresses in the xml file) and send the report to hello@syslifters.com that we can further troubleshoot the issue?
Hi, the nessus file that I used was just a sample from DefectDojo, as shown here: https://github.com/DefectDojo/sample-scan-files/blob/master/nessus/nessus_v_unknown.nessus. Could I check if the allowed file extension for nessus scan report is only strictly restricted to .nessus?
I, unfortunately, cannot reproduce your error. After downloading the sample report and using reptor, everything seems to work as expected:
Can you check if you have the latest reptor version installed and upgrade if not (pip3 install --upgrade reptor
).
The only supported file type for nessus scans is the .nessus extention, which must contain XML data.
Could you share with me your config.yaml inside .sysreptor directory (with your token redacted), as well as the docker-compose.yml (for the Caddy webserver) so that I could troubleshoot it on my end? Thank you.
I was thinking if it could be due to the server being either on localhost or insecure (with http)?
p.s The reptor version has been upgraded to the latest, error still persists.
A sample .config.yaml would be:
project_id: 1d2da818-913f-482f-a19c-d9eff0d128f0
server: https://reptortest.sysre.pt/
token: sysreptor_XXXXXXXXXXX
If you think that something is wrong with your Caddy setup, you can run caddy.sh
from the SysReptor repo located in deploy/caddy/
. This will use a Dockerfile shipped with SysReptor and add the container to your main docker-compose.yml.
My impression is that your SysReptor installation is not the problem.
Can you just try to upload a plain note?
> echo "Upload me" | reptor --notetitle "My Upload" note -k
Reading from stdin...
Note written to "My Upload".
Successfully uploaded to notes.
If this works, it rules out issues with your installation.
The uploading of the plain note does not work on my end as shown:
zy@ubuntu:~/sysreptor/deploy/caddy$ echo "Upload me" | reptor --notetitle "My Upload" note -k
Reading from stdin...
'list' object has no attribute 'get'
This is what happens when i ran ./setup.sh, please advise on which parts I have to debug.
zy@ubuntu:~/sysreptor/deploy/caddy$ ./setup.sh
SysReptor runs on localhost (127.0.0.1) by default.
Should we setup a webserver (Caddy in Docker) for you to expose it to your local network or the Internet? [y/n]: y
Should we take care of your webserver HTTPS certificate using LetsEncrypt?
For this, you must set up:
1. a valid domain name resolving to your public IP address
2. port 80 of your must be publicly reachable
Want a LetsEncrypt webserver certificate? [y/n]: n
What port should the webserver use? [80]
Setting up your web server...
service "app" has neither an image nor a build context specified: invalid compose project
Context: I don't have a valid domain name resolving to my public IP address, because I just want to test it locally. Please do let me know if thats the root cause.
The uploading of the plain note does not work on my end as shown:
zy@ubuntu:~/sysreptor/deploy/caddy$ echo "Upload me" | reptor --notetitle "My Upload" note -k Reading from stdin... 'list' object has no attribute 'get'
Can you run this command with --debug please and send me the output?
Attached in this post is the file containing the debug output.
p.s I think the issue lies in the fact there is no response when my local machine is trying to establish a HTTPS connection with my localhost (at port 443).
This is what happens when i ran ./setup.sh, please advise on which parts I have to debug.
zy@ubuntu:~/sysreptor/deploy/caddy$ ./setup.sh SysReptor runs on localhost (127.0.0.1) by default. Should we setup a webserver (Caddy in Docker) for you to expose it to your local network or the Internet? [y/n]: y Should we take care of your webserver HTTPS certificate using LetsEncrypt? For this, you must set up: 1. a valid domain name resolving to your public IP address 2. port 80 of your must be publicly reachable Want a LetsEncrypt webserver certificate? [y/n]: n What port should the webserver use? [80] Setting up your web server... service "app" has neither an image nor a build context specified: invalid compose project
Context: I don't have a valid domain name resolving to my public IP address, because I just want to test it locally. Please do let me know if thats the root cause.
You don't need a domain name - this is fine.
The error your receive is also quite strange. Did you change anything in docker-compose.yaml files?
Make sure to implement the following file structure, which should fix your setup.
Your deploy/docker-compose.yaml
:
name: sysreptor
include:
- sysreptor/docker-compose.yml
- caddy/docker-compose.yaml
Your deploy/caddy/docker-compose.yaml
: Here
Your deploy/sysreptor/docker-compose.yaml
: Here
If you added/updated those files, run docker compose up -d
from the deploy
directory.
Make sure that all containers (app, redis, postgres, caddy) are running using docker ps
This is my current directory at ~/sysreptor/deploy:
zy@ubuntu:~/sysreptor/deploy$ ls
app.env app.env.example caddy docker-compose.override.yml docker-compose.yml setup.sh sysreptor.nginx
Could I clarify if I am missing a sysreptor directory (deploy/sysreptor
)?
Okay, I think we approached the solution. Two things:
requests
library follows the redirects, also for POST requests. This causes the note creation to fail, because the redirect is done using a GET request.
Solution: Run reptor conf
to update your server URL to https://127.0.0.1:443update.sh
from your SysReptor folder. (This, however, is not the reason why reptor commands fail.)We will check for redirects in the next version of reptor to communicate this error more clearly.
This is what happens when i ran ./setup.sh, please advise on which parts I have to debug.
zy@ubuntu:~/sysreptor/deploy/caddy$ ./setup.sh SysReptor runs on localhost (127.0.0.1) by default. Should we setup a webserver (Caddy in Docker) for you to expose it to your local network or the Internet? [y/n]: y Should we take care of your webserver HTTPS certificate using LetsEncrypt? For this, you must set up: 1. a valid domain name resolving to your public IP address 2. port 80 of your must be publicly reachable Want a LetsEncrypt webserver certificate? [y/n]: n What port should the webserver use? [80] Setting up your web server... service "app" has neither an image nor a build context specified: invalid compose project
Context: I don't have a valid domain name resolving to my public IP address, because I just want to test it locally. Please do let me know if thats the root cause.
You don't need a domain name - this is fine.
The error your receive is also quite strange. Did you change anything in docker-compose.yaml files?
Make sure to implement the following file structure, which should fix your setup.
Your
deploy/docker-compose.yaml
:name: sysreptor include: - sysreptor/docker-compose.yml - caddy/docker-compose.yaml
Your
deploy/caddy/docker-compose.yaml
: HereYour
deploy/sysreptor/docker-compose.yaml
: HereIf you added/updated those files, run
docker compose up -d
from thedeploy
directory. Make sure that all containers (app, redis, postgres, caddy) are running usingdocker ps
Hi Aron, I managed to setup all the docker containers successfully. I have changed the server URL to https://127.0.0.1:443 and I managed to upload the notes perfectly as shown:
zy@ubuntu:~/sysreptor/deploy$ echo "Upload me" | reptor --notetitle "My Upload" note -k
Reading from stdin...
Note written to "My Upload".
Successfully uploaded to notes.
Thank you for the help and assistance, cheers to a good day for you!
Great - I'm happy to hear this.
I'll keep this ticket open to track improvements of the error messages.
So does that mean I won't have to close this issue, so that you can track improvements of the error messages?
I followed the guide to run setup.sh from https://docs.sysreptor.com/setup/webserver/. I wanted to setup Caddy webserver listening on port 80 then forward it to Sysreptor which is listening on port 8000 (by default), without any certificates involved.
When I ran the command, these errors came up:
The setup.sh that I ran (cloned from sysreptor github repo):
config.yaml (in ./sysreptor)
docker containers currently up and running