CSV feed limit of 5 000 increase possible?

RegturH commented 2 months ago

Prerequisites

[ x] I read the Deployment and Setup section of the OpenCTI documentation as well as the Troubleshooting page and didn't find anything relevant to my problem.
[ x] I went through old GitHub issues and couldn't find anything relevant
[ x] I googled the issue and didn't find anything relevant

Description

Last year I asked a question surrounding the maximum length of csv feeds.

I would like to increase it but it not possible.

I created https://github.com/OpenCTI-Platform/opencti/issues/4636

this has been closed.

I was not able to upgrade our platform due to technical challenges.

When upgrading to 6.2.0 I wanted to upgrade the csv feed.

when reading the wiki: https://docs.opencti.io/latest/usage/feeds/ I saw the limit was still 5000 entries.

Environment

OS (where OpenCTI server runs): red hat
OpenCTI version: 6.2.0
OpenCTI client: front end
Other environment details:

Reproducible Steps

Steps to create the smallest reproducible scenario:

create a feed with more then 5 000 entities in the filter
limit is 5000

Additional information

Issue Summary:

We are currently utilizing a CSV feed to supply our firewall with pertinent malicious IP addresses. However, we are encountering a limitation where the feed can only accommodate up to 5,000 IPs.

The feed operates on a rolling time frame of two months. During this period, we often add more IPs than we remove. This imbalance results in relevant IPs being displaced from the feed by newer ones.

Question:

Is it possible to partition this feed to address the issue?

richard-julien commented 2 months ago

Hi @RegturH ,

Thats an option we forget to put in the documentation. You should be able to modify this number using.

As env variable

DATA_SHARING__MAX_CSV_FEED_RESULT=10000

Or in config file

"data_sharing": {
  "max_csv_feed_result": 10000
},

However you need tro understand that this number can lead to performance degradation depending of your platform and your csv feed configuration. Please do some testing and align this number with your platform capacity to avoid any problem.

@labo-flg can you take some time to add this in the official doc? Thanks.

RegturH commented 2 months ago

Hi Julien,

Thank you for the reply.

Will implement and test this.

Kind regards

Rutger

OpenCTI-Platform / opencti