search

OpenCTI-Platform / connectors

OpenCTI Connectors
https://www.opencti.io
Apache License 2.0
366 stars 395 forks source link

[Stream Connectors] - How to use CONNECTOR_LIVE_STREAM_START_TIMESTAMP #2649

Open vishesh-verma-coder opened 1 week ago

vishesh-verma-coder commented 1 week ago

Description

We are running some stream connectors and biggest problem is if we restart docker container, connector loses its state and then starts reading the data from redis stream from starting which reads historical logs also and sends unnecessary requests to stream endpoint [ex : QRadar]

This becomes more problematic when you have a good number of indicators and one day you restart docker container of the stream connector, then it will take a lot of time for the connector to stream historical indicators and there can be a delay in reading real time stream message.

I see a Configuration Parameter CONNECTOR_LIVE_STREAM_START_TIMESTAMP [ex: https://github.com/OpenCTI-Platform/connectors/tree/master/stream/qradar], which should allow us to define start time of the stream but it does not seem to work. In the stream connector code, I do not it getting referred anywhere.

Is this parameter in use CONNECTOR_LIVE_STREAM_START_TIMESTAMP ?

Environment

  1. OS (where OpenCTI server runs): RHEL 8.4
  2. OpenCTI version: 6.2.4
  3. OpenCTI client: frontend
  4. Other environment details: Connector running as docker containers on frontend machine and ingestion cluster is separate handling platform managers and workers.

Reproducible Steps

Steps to create the smallest reproducible scenario:

  1. Configure a stream connector and refer a live stream id... Keep it running for some time
  2. Restart docker container of stream connector. It will restart from beginning and start processing all messages in the stream.

There can be a control based on time/date to define from when stream connector should read the events from redis stream .

helene-nguyen commented 18 hours ago

@vishesh-verma-coder It is the correct configuration variable, but restarting the connector only will not apply immediately the new configuration. You also have to reset the state of the connector. Could you confirm that you have done this step too ?

Ping @NiQuintin