search

SpecterOps / Nemesis

An offensive data enrichment pipeline
https://specterops.github.io/Nemesis/
Other
613 stars 59 forks source link

Not able to access Nemesis Dashboard after install #19

Closed siesta888 closed 1 year ago

siesta888 commented 1 year ago

user@nemesis:~/Nemesis$ ip a

ens33:
...
inet 192.168.152.131/24
user@nemesis:~/Nemesis$ kubectl get pod
NAME                                      READY   STATUS      RESTARTS      AGE
dotnet-57f5b9dc5b-hgllh                   1/1     Running     0             8m18s
elasticsearch-exporter-566d77f8cc-shpbw   1/1     Running     0             8m54s
enrichment-95cb8f4db-wwwjw                1/1     Running     0             7m36s
gotenberg-85454494f4-69qnq                1/1     Running     0             11m
kibana-init-job-9zzl2                     0/1     Completed   0             11m
minio-7dccd7f8c5-k65xs                    1/1     Running     1 (18m ago)   19m
nemesis-es-default-0                      1/1     Running     0             11m
nemesis-kb-5f9897b569-brpkt               1/1     Running     0             11m
nemesis-rabbitmq-0                        1/1     Running     0             11m
nlp-8b4b65474-7pgm2                       1/1     Running     0             8m18s
passwordcracker-7b5d7d6678-hxvm5          1/1     Running     0             8m17s
pgadmin-0                                 1/1     Running     0             11m
postgres-745b6f6576-4t24c                 1/1     Running     0             11m
postgres-exporter-9b879d87c-rqbvx         1/1     Running     0             8m53s
tensorflow-serving-575b545dff-rkvt8       1/1     Running     0             8m17s
tika-5764c986bf-xnmpg                     1/1     Running     0             11m
user@nemesis:~/Nemesis$ cat nemesis.config
# General options
# The nemesis_http_server must match the port of the ingress-nginx-controller service in skaffold.yaml (port 8080 by default)
nemesis_http_server: http://192.168.152.131:8080
...
(truncated)
...

Just from eyeballing it, it looks like the ingress service never started?

user@nemesis:~/Nemesis$ kubectl get service
NAME                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                           AGE
dotnet                       NodePort    10.106.131.35    <none>        9800:31800/TCP                                    14m
elasticsearch-exporter       NodePort    10.109.241.175   <none>        9108:31108/TCP                                    15m
enrichment-cracklist         NodePort    10.106.28.216    <none>        9900:30089/TCP                                    14m
enrichment-landingpage       NodePort    10.99.213.121    <none>        9920:30809/TCP                                    14m
enrichment-monitoring        NodePort    10.111.77.121    <none>        8000:31693/TCP                                    14m
enrichment-webapi            NodePort    10.101.19.105    <none>        9910:30808/TCP                                    14m
enrichment-yara              NodePort    10.100.242.28    <none>        9700:30079/TCP                                    14m
gotenberg                    NodePort    10.111.211.119   <none>        3000:31300/TCP                                    17m
kubernetes                   ClusterIP   10.96.0.1        <none>        443/TCP                                           49m
minio                        ClusterIP   10.107.183.177   <none>        9000/TCP,9001/TCP                                 25m
nemesis-es-default           ClusterIP   None             <none>        9200/TCP                                          17m
nemesis-es-http              ClusterIP   10.110.148.189   <none>        9200/TCP                                          17m
nemesis-es-internal-http     ClusterIP   10.96.102.71     <none>        9200/TCP                                          17m
nemesis-es-transport         ClusterIP   None             <none>        9300/TCP                                          17m
nemesis-kb-http              ClusterIP   10.99.85.240     <none>        5601/TCP                                          17m
nemesis-rabbitmq-discovery   ClusterIP   None             <none>        5672/TCP,5671/TCP,25672/TCP,15692/TCP,15672/TCP   17m
nemesis-rabbitmq-svc         ClusterIP   10.104.93.36     <none>        5672/TCP,5671/TCP,25672/TCP,15692/TCP,15672/TCP   17m
nlp                          NodePort    10.105.211.65    <none>        9803:31803/TCP,8000:31692/TCP                     14m
passwordcracker              NodePort    10.96.80.56      <none>        9090:31990/TCP                                    14m
pgadmin                      NodePort    10.106.206.250   <none>        80:31288/TCP                                      17m
postgres                     NodePort    10.111.254.211   <none>        5432:31209/TCP                                    17m
postgres-exporter            NodePort    10.99.147.22     <none>        9187:31119/TCP                                    15m
tensorflow-serving           NodePort    10.102.60.137    <none>        8501:31501/TCP                                    14m
tika                         NodePort    10.101.174.23    <none>        9998:31998/TCP                                    17m
$ sudo netstat -punta|grep 8080
..nothing..
leechristensen commented 1 year ago

When you run nemesis-cli.py, it should create minio, elastic operator, and nginx ingress pods. When you run nemesis-cli.py, do you see those pods once it's complete?

Easiest way to start fresh is:

minikube delete    # delete your current cluster
minikube start       # start up minikube again

./nemesis-cli.py     # Setup Nemesis configuration again

kubectl get pods -A # check for minio/ingress/ES operator pods after nemesis-cli.py

skaffold run  --port-forward   # Kick things off
siesta888 commented 1 year ago

Here are the pods I have. I'll try restarting. Looks like I never got the ingress pod.

user@nemesis:~/Nemesis$ kubectl get pod
NAME                                      READY   STATUS      RESTARTS      AGE
dotnet-56997dcdd4-nr9m5                   1/1     Running     0             22m
elasticsearch-exporter-8665548c69-ldmvq   1/1     Running     0             22m
enrichment-645dd99b5d-fqqht               1/1     Running     0             21m
gotenberg-5bcbcd8ff8-4wwm7                1/1     Running     0             23m
kibana-init-job-9zzl2                     0/1     Completed   0             47m
minio-7dccd7f8c5-k65xs                    1/1     Running     1 (54m ago)   55m
nemesis-es-default-0                      1/1     Running     0             47m
nemesis-kb-5f9897b569-brpkt               1/1     Running     0             47m
nemesis-rabbitmq-0                        1/1     Running     0             23m
nlp-68c79c894b-fpvkw                      1/1     Running     0             22m
passwordcracker-74cd54bb58-rtq2z          1/1     Running     0             22m
pgadmin-0                                 1/1     Running     0             23m
postgres-748b6f98f6-psd4p                 1/1     Running     1 (22m ago)   23m
postgres-exporter-646b44b4d9-xwqv2        1/1     Running     0             22m
tensorflow-serving-f68cbf967-hbnt7        1/1     Running     0             22m
tika-7c7bf49657-j8spp                     1/1     Running     0             23m
leechristensen commented 1 year ago

Can you add a -A to you pod listing command? E.g. kubectl get pods -A? The ingress is running in a different namespace

siesta888 commented 1 year ago

I di what you said and rebuilt it, verified the ingress controller is there and I can exec into it and see that nginx is running, but the port forward seems to fail to take effect on the host.

user@nemesis:~/Nemesis$ kubectl get pods -A
NAMESPACE        NAME                                        READY   STATUS      RESTARTS      AGE
default          dotnet-77bb6d9855-p8wlg                     1/1     Running     0             10m
default          elasticsearch-exporter-79bbd7bf68-xxzjp     1/1     Running     0             11m
default          enrichment-59df6c7647-gd85q                 1/1     Running     0             10m
default          gotenberg-5f776d5789-rj9wc                  1/1     Running     0             13m
default          kibana-init-job-b22qn                       0/1     Completed   0             13m
default          minio-66f6748f6-rt268                       1/1     Running     0             21m
default          nemesis-es-default-0                        1/1     Running     0             13m
default          nemesis-kb-5cff666c56-gj5fj                 1/1     Running     0             13m
default          nemesis-rabbitmq-0                          1/1     Running     0             13m
default          nlp-84f4d4ffbc-gm6lp                        1/1     Running     0             10m
default          passwordcracker-65d875c59-sw8j4             1/1     Running     0             10m
default          pgadmin-0                                   1/1     Running     0             13m
default          postgres-74fc86857f-rzvth                   1/1     Running     0             13m
default          postgres-exporter-6c79c46986-vcjhl          1/1     Running     0             11m
default          tensorflow-serving-857f88c7b-fnt9q          1/1     Running     0             10m
default          tika-5469955c58-mj29g                       1/1     Running     0             13m
elastic-system   elastic-operator-0                          1/1     Running     0             20m
ingress-nginx    ingress-nginx-controller-67dd76f85b-l5gxr   1/1     Running     0             20m
kube-system      coredns-5d78c9869d-9gp27                    1/1     Running     0             26m
kube-system      etcd-minikube                               1/1     Running     0             26m
kube-system      fluentd-zj6t2                               1/1     Running     0             11m
kube-system      kube-apiserver-minikube                     1/1     Running     0             26m
kube-system      kube-controller-manager-minikube            1/1     Running     0             26m
kube-system      kube-proxy-7gzb8                            1/1     Running     0             26m
kube-system      kube-scheduler-minikube                     1/1     Running     0             26m
kube-system      kube-state-metrics-5846d4d795-gd48d         1/1     Running     0             11m
kube-system      metrics-server-55c649bc7b-fjm9g             1/1     Running     0             20m
kube-system      storage-provisioner                         1/1     Running     1 (26m ago)   26m
monitoring       alertmanager-7cfd6ccf47-t2mg7               1/1     Running     0             11m
monitoring       grafana-5bc975484f-kvc5b                    1/1     Running     0             11m
monitoring       node-exporter-r5mnf                         1/1     Running     0             11m
monitoring       prometheus-deployment-7dd7fc5bdf-4zv27      1/1     Running     0             11m

user@nemesis:~/Nemesis$ kubectl exec -it --namespace=ingress-nginx ingress-nginx-controller-67dd76f85b-l5gxr -- /bin/bash -c "hostname;netstat -punta|grep 80;ps auxfw|grep nginx"
ingress-nginx-controller-67dd76f85b-l5gxr
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:10246         127.0.0.1:59180         TIME_WAIT   -
tcp        0      0 ::ffff:10.244.0.5:10254 ::ffff:10.244.0.1:56280 TIME_WAIT   -
    1 www-data  0:00 /usr/bin/dumb-init -- /nginx-ingress-controller --publish-service=ingress-nginx/ingress-nginx-controller --election-id=ingress-nginx-leader --controller-class=k8s.io/ingress-nginx --ingress-class=nginx --configmap=ingress-nginx/ingress-nginx-controller --tcp-services-configmap=ingress-nginx/ingress-nginx-tcp --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key
    7 www-data  0:02 /nginx-ingress-controller --publish-service=ingress-nginx/ingress-nginx-controller --election-id=ingress-nginx-leader --controller-class=k8s.io/ingress-nginx --ingress-class=nginx --configmap=ingress-nginx/ingress-nginx-controller --tcp-services-configmap=ingress-nginx/ingress-nginx-tcp --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key
   22 www-data  0:00 nginx: master process /usr/bin/nginx -c /etc/nginx/nginx.conf
  440 www-data  0:00 nginx: worker process
  441 www-data  0:00 nginx: worker process
  442 www-data  0:00 nginx: worker process
  443 www-data  0:00 nginx: worker process
  444 www-data  0:00 nginx: cache manager process
  614 www-data  0:00 /bin/bash -c hostname;netstat -punta|grep 80;ps auxfw|grep nginx
  624 www-data  0:00 grep nginx
leechristensen commented 1 year ago

Some questions:

  1. Did you add --port-forward to your skaffold command?
  2. Can you access the ingress port locally on the host running nemesis using curl? You can't access the ingress port remotely at all, correct?
  3. Can you do a process listing and see if there's a kubectl port-forward command associated with the ingress? Skaffold uses kubectl underneath to setup port forwards.
siesta888 commented 1 year ago

  1. Yes I ran skaffold run --port-forward

  2. No, I can't access on the nemesis host using curl.

    user@nemesis:~/Nemesis$ curl localhost:8080
curl: (7) Failed to connect to localhost port 8080: Connection refused
  3. user@nemesis:~/Nemesis$ ps auxfw|grep kubectl
user     1370881  0.0  0.0   6240   704 pts/0    S+   16:10   0:00  |           \_ grep kubectl
siesta888 commented 1 year ago

Manually running kubectl port-forward --address 192.168.152.131 --namespace=ingress-nginx service/ingress-nginx-controller 8080:80 Allows me to access the dashboard

jcorert commented 1 year ago

I'm having the same issue. Running the manual kubectl port-forward command gets me to the the initial dashboard (with the basic links), but then the main dashboard doesn't open.

leechristensen commented 1 year ago

Looks like there's an oddity in skaffold's port forwarding with the run command due to the recently added dev/jupyter profiles. Just pushed a regression fix and also the updated the setup documentation with a new skaffold command to run Nemesis to reduce the number of port forwards that get setup. The new command to start all the Nemesis services is:

skaffold run -m nemesis --port-forward=user

Let me know if you still run into any issues with port forwards using skaffold run!