search

red-kite-solutions / stalker

Stalker, the Extensible Attack Surface Management tool.
https://wiki.stalker.red-kite.io/
GNU General Public License v3.0
75 stars 4 forks source link

Orchestrator and Flow Manager start before Kafka full completion #103

Closed lm-sec closed 1 year ago

lm-sec commented 1 year ago

Made a ticket to list details @Aboisier

When given enough ressources (RAM + CPU), the k8s deployment in minikube finishes to deploy and initiate the flow manager and the orchestrator before the completion of Kafka's deployment and initiation, resulting in errors and the container's failure. The error is the same on both containers and it states that the flow manager or the orchestrator cannot reach Kafka's port. Logs at the end.

How to reproduce: 

minikube stop
minikube delete
minikube start --driver=docker --memory=8192 --cpus=4
devspace dev -n stalker

Orchestrator: 

%3|1666323050.312|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#10.106.193.42:9092 failed: Connection refused (after 0ms in state CONNECT)                           
%3|1666323050.312|ERROR|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: 1/1 brokers are down                                                                                                                      
%3|1666323050.313|ERROR|rdkafka#producer-1| [thrd:app]: rdkafka#producer-1: kafka:9092/bootstrap: Connect to ipv4#10.106.193.42:9092 failed: Connection refused (after 0ms in state CONNECT)

Flow Manager:

[Nest] 29  - 10/21/2022, 3:31:04 AM   ERROR [ExceptionHandler] Connection error: connect ECONNREFUSED 10.106.193.42:9092                                                                                           
KafkaJSNonRetriableError                                                                                                                                                                                           
  Caused by: KafkaJSConnectionError: Connection error: connect ECONNREFUSED 10.106.193.42:9092
lm-sec commented 1 year ago

J'ai update devspace.yml en fonction de ce que tu m'avais dit, ca semble fonctionner. Par contre, j'ai une nouvelle erreur dans l'orchestrateur : 

dotnet watch ⌚ Exited
dotnet watch ⌚ File changed: /app/Constants.cs
dotnet watch ⌚ MSBuild output from target 'GenerateWatchList':
dotnet watch ⌚ 
dotnet watch ⌚    MSBUILD : error MSB1009: Project file does not exist.
dotnet watch ⌚    Switch: /app/Orchestrator.csproj
dotnet watch ⌚ 
dotnet watch ❌ Error(s) finding watch items project file 'Orchestrator.csproj'
dotnet watch ⌚ Fix the error to continue or press Ctrl+C to exit.
lm-sec commented 1 year ago

J'ai la meme erreur dans #102, on dirait que l'orchestrateur a perdu son csproj somehow. J'ai essaye de le fix dans /orchestrator/orchestrator/dockerfile.dev, sans succes

lm-sec commented 1 year ago

Have not seen this bug in a while, closing