search

OpenCTI-Platform / connectors

OpenCTI Connectors
https://www.opencti.io
Apache License 2.0
374 stars 407 forks source link

Backup connector Error every 10sec when using AlienVault Feed #1626

Closed PeeBee66 closed 8 months ago

PeeBee66 commented 10 months ago

Description

Consistent restarting of the backup connector when using AlienVault

Backup connector https://github.com/OpenCTI-Platform/connectors/tree/master/stream/backup-files

INFO Backup processed event 1701520853906-0 in 20231202T124100Z / relationship--fc89d0ab-3968-4f05-8437-641af617d002 | timestamp=2023-12-17T21:40:39.642305Z name=pycti.connector Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/pycti/connector/opencti_connector_helper.py", line 530, in run self.callback(msg) File "/opt/opencti-connector-backup-files/backup-files.py", line 75, in _process_message data = json.loads(msg.data) ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/init.py", line 346, in loads return _default_decoder.decode(s) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/decoder.py", line 353, in raw_decode obj, end = self.scan_once(s, idx) ^^^^^^^^^^^^^^^^^^^^^^ json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 841 (char 840) Terminated

Environment

  1. OS - Ubuntu
  2. OpenCTI version: 5.12.9
  3. OpenCTI client: frontend
  4. Other environment details: The server only runs the alien vault feed, I have rebuilt it and the same issue is still occurring

Reproducible Steps

Steps to create the smallest reproducible scenario:

  1. Install docker, docker compose,
  2. load up docker compose file
  3. wait a bit for the files to start coming in and backing up

Expected Output

I expect the backup feed to keep backing up with out restarting every 10 sec.

i have a number of other opencti instances and feeds backing up and not one other seems to have this issue

Actual Output

The backup connector restarts every 10 sec and shows this error

AV-connectoer log.txt

Additional information

`###------------------- AV CONNECTOR 1 of 2 PORT 4000-----------------------### version: '3' services:

------------------- REDIS SEARCH -----------------------

redis: image: redis:7.2.3 ports:

------------------- Worker -----------------------

worker: image: opencti/worker:${OPENCTI_VERSION} environment:

------------------- AV BACKUP -----------------------

c.av-backup-files: image: opencti/connector-backup-files:${OPENCTI_VERSION}

privileged: true

environment:      
  - OPENCTI_URL=http://opencti:4000
  - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
  - CONNECTOR_ID=${CONNECTOR_BACKUP_ID}
  - CONNECTOR_TYPE=STREAM
  - CONNECTOR_LIVE_STREAM_ID=live # ID of the live stream created in the OpenCTI UI
  - CONNECTOR_LIVE_STREAM_LISTEN_DELETE=true
  - CONNECTOR_NAME=BackupFiles
  - CONNECTOR_SCOPE=backup
  - CONNECTOR_LOG_LEVEL=info
  - BACKUP_PROTOCOL=local # Protocol for file copy (only `local` is supported for now).
  - BACKUP_PATH=/home/opencti_admin/1.Feeds/${CONNECTOR_FEED} # Path to be used to copy the data, can be relative or absolute.
restart: always
volumes:
  - /home/opencti_admin/1.Feeds/${CONNECTOR_FEED}:/home/opencti_admin/1.Feeds/${CONNECTOR_FEED}
depends_on:
  - opencti

------------------- AV FEED -----------------------

connector-alienvault: image: opencti/connector-alienvault:${OPENCTI_VERSION} environment:

Screenshots (optional)

nino-filigran commented 9 months ago

@axelfahy I think you were the one who created the conenctor. Would you mind having a look?

axelfahy commented 9 months ago

@nino-filigran, sorry, but I have never used this connector.

nino-filigran commented 9 months ago

my bad then! I did look in the wrong place. @daemitus / @Arcelone / @rlynch-ironnet / @maertv I see that you all have done some PRs in the past with this connector. Would any of you mind having a look at this?