search

aws / aws-for-fluent-bit

The source of the amazon/aws-for-fluent-bit container image
Apache License 2.0
459 stars 134 forks source link

Is it possible to move custom configuration above FILTER #240

Closed karthikeayan closed 3 weeks ago

karthikeayan commented 3 years ago

Describe the issue

My application writes logs to stdout as well as multiple files in the filesystem. By default firelens is configured to read the logs from stdout. I am able to forward the logs to my destination with tail fluent-bit plugin. However, ECS metadata is not added to the logs coming from tail plugin. Because, custom fluent-bit configuration is added after the "record_modifier" section in the main fluent-bit.conf

Configuration

Task definition:

        {
            "logConfiguration": {
              "logDriver": "awslogs",
              "options": {
                "awslogs-group": "firelens-container",
                "awslogs-region": "${region}",
                "awslogs-create-group": "true",
                "awslogs-stream-prefix": "firelens"
              }
            },
            "firelensConfiguration": {
              "type": "fluentbit",
              "options": {
                "config-file-type": "file",
                "config-file-value": "/fluent-bit/alt/fluent-bit.conf"
              }
            },
            "environment": [
              {
                "name": "REGION",
                "value": "${region}"
              },
              {
                "name": "LOG_GROUP",
                "value": "/ecs/${log_group_name}"
              },
                          {
                "name": "LOG_FILES_PATH",
                "value": "${log_files_path}"
              },
              {
                "name": "ES_HOST",
                "value": "${es_host}"
              },
              {
                "name": "ES_INDEX",
                "value": "${es_index_name}"
              }          
            ],
            "image": "amazon/aws-for-fluent-bit:2.19.1",
            "essential": true,
            "name": "log_router"
        },

fluent-bit custom configuration:

[INPUT]
    Name tail
    Path ${LOG_FILES_PATH}
    Tag ${CONTAINER_NAME}-firelens-filesystem

[OUTPUT]
    Name cloudwatch
    Match *
    auto_create_group true
    log_group_name ${LOG_GROUP}
    log_stream_name $(ecs_task_id)
    log_key log
    region ${REGION}

[OUTPUT]
    Name       es
    Match      *
    Host       ${ES_HOST}
    Port       443
    Index      ${ES_INDEX}
    Type       log
    AWS_Auth   On
    AWS_Region ${REGION}
    tls        On
    Trace_Error On

Dockerfile:

FROM amazon/aws-for-fluent-bit:2.19.1

RUN mkdir /fluent-bit/alt
ADD fluent-bit.conf /fluent-bit/alt/fluent-bit.conf

/fluent-bit/etc/fluent-bit.conf:

.
.
.
[FILTER]
    Name record_modifier
    Match *
    Record ecs_cluster abcd
    Record ecs_task_arn abcd
    Record ecs_task_definition abcd

@INCLUDE /fluent-bit/configs/custom-firelens.conf
.
.
.

Fluent Bit Version Info

amazon/aws-for-fluent-bit:2.19.0

Cluster Details

ECS Fargate

Application Details

Java Spring Boot

DrewZhang13 commented 3 years ago

I am not sure if I understand the question. But the configuration order should have no impact of their function, which means move custom configuration above or below should have the same log results.

giammbo commented 3 years ago

@DrewZhang13 The problem (i have the same issue) the filter section seems ignored. I'm trying to send logs to NewRelic and add some tags in the Filter section for example Record pippo pluto, but inside Newrelic i cant see any difference (filter section seems ignored).

zhonghui12 commented 3 years ago

Hi @karthikeayan and @giammbo, I am not sure if I understand it correctly but could you try put the @include at top-left level? https://github.com/aws-samples/amazon-ecs-firelens-examples. Or maybe you could try @include the filter as well

PettitWesley commented 3 years ago

EDIT: This is wrong-ish and I can't believe I forgot how FireLens/Fluent Bit works, please see Matt's comment: https://github.com/aws/aws-for-fluent-bit/issues/240#issuecomment-996374431

@zhonghui12 @DrewZhang13 This issue requires an understanding of the FireLens architecture and design. Please review these:

@karthikeayan is correct. User provided configuration is always included after the auto-generated filter that adds ECS metadata.

Unfortunately there is not a super nice workaround right now... my best idea is to build a custom Fluent Bit that customizes the entrypoint with a script that calls ECS task metadata and sets metadata values as environment variables and then invokes Fluent Bit.

I think if/when we build a "FireLens V2" we will fix this... or we will eventually implement ECS metadata injection support in Fluent Bit...

PettitWesley commented 3 years ago

EDIT: This is not needed and I can't believe I forgot how FireLens/Fluent Bit works, please see Matt's comment: https://github.com/aws/aws-for-fluent-bit/issues/240#issuecomment-996374431

Actually, I just thought of another workaround that might work...

First, you use the techniques in this blog to create your own config file which will be the "main" config file and then customize the entrypoint to use that file: https://aws.amazon.com/blogs/containers/how-to-set-fluentd-and-fluent-bit-input-parameters-in-firelens/

In that blog, I just ignore the auto-generated config file, which would still be created. However, you could actually @INCLUDE it to import it.

So your main config (in the blog "/fluent-bit/alt/fluent-bit.conf") would have this:

# add your tail input here
[INPUT]
      Name tail

# import the auto-generated file to get the metadata filter
@INCLUDE /fluent-bit/etc/fluent-bit.conf 

# add your output below
[OUTPUT]

Does that make sense? Can you try it out? I probably need to create an example for this...

giammbo commented 3 years ago

@PettitWesley my fluentbit configuration it's like that:

                [SERVICE]
                  Plugins_File plugins.conf
                [OUTPUT]
                  Name                    gelf
                  Match                   *
                  Host                    graylog
                  Port                    12201
                  Mode                    tcp
                  Gelf_Short_Message_Key  log
                [OUTPUT]
                  Name newrelic
                  Match *
                  endpoint https://log-api.eu.newrelic.com/log/v1
                  licenseKey ${NR_LICENSE_KEY}
                [FILTER]
                  Name record_modifier
                  Match *
                  Record ecs_cluster ${ECS_CLUSTER}
                  Record ecs_task_arn ${ECS_TASK_ARN}

my problems is the filter part, i cant see this Record inside graylog and newrelic, im not sure but seems not a problem related to firelens but to newrelic. Have you any tips in order to add some custom tags?

PettitWesley commented 3 years ago

@giammbo Do you have the env vars $ECS_CLUSTER and $ECS_TASK_ARN set on your fluent bit container? How did you set those? Can you please share your task definition as well?

i cant see this Record inside graylog and newreli

Please be more clear on what you mean here.

giammbo commented 3 years ago

@PettitWesley i have also try with Record foo bar but the filter section is not recognised FYI: i also use the Modify plugin with Add foo bar, but nothing changed

All filter section is ignored by the container, also moving between the various section, for example above the SERVICE section or above the OUTPUT section

giammbo commented 3 years ago

After some investigation i found this little "bug". My actual configuration it's this:

               [SERVICE]
                  Log_Level debug
                  Flush 1
                  Grace 30
                [FILTER]
                  Name modify
                  Match *
                  Add foo bar
                [OUTPUT]
                  Name gelf
                  Match *
                  Host graylog.musement.int
                  Port 12201
                  Mode tcp
                  Gelf_Short_Message_Key log
                [OUTPUT]
                  Name newrelic
                  Match *
                  endpoint https://log-api.eu.newrelic.com/log/v1
                  licenseKey XXXX

but when i start the container inside the cloudwatch logs i see that:


* Copyright (C) 2019-2021 The Fluent Bit Authors
--
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
Output plugin 'musement-musement-core-app' cannot be loaded
[2021/10/08 13:41:48] [ info] Configuration:
[2021/10/08 13:41:48] [ info]  flush time     \| 1.000000 seconds
[2021/10/08 13:41:48] [ info]  grace          \| 30 seconds
[2021/10/08 13:41:48] [ info]  daemon         \| 0
[2021/10/08 13:41:48] [ info] ___________
[2021/10/08 13:41:48] [ info]  inputs:
[2021/10/08 13:41:48] [ info]      tcp
[2021/10/08 13:41:48] [ info]      forward
[2021/10/08 13:41:48] [ info]      forward
[2021/10/08 13:41:48] [ info] ___________
[2021/10/08 13:41:48] [ info]  filters:
[2021/10/08 13:41:48] [ info] ___________
[2021/10/08 13:41:48] [ info]  outputs:
[2021/10/08 13:41:48] [ info]      gelf.0
[2021/10/08 13:41:48] [ info]      newrelic.1
[2021/10/08 13:41:48] [ info]      null.2
[2021/10/08 13:41:48] [ info] ___________
[2021/10/08 13:41:48] [ info]  collectors:
[2021/10/08 13:41:48] [ info] [engine] started (pid=11)
[2021/10/08 13:41:48] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2021/10/08 13:41:48] [debug] [storage] [cio stream] new stream registered: tcp.0
[2021/10/08 13:41:48] [debug] [storage] [cio stream] new stream registered: forward.1
[2021/10/08 13:41:48] [debug] [storage] [cio stream] new stream registered: forward.2
[2021/10/08 13:41:48] [ info] [storage] version=1.1.1, initializing...
[2021/10/08 13:41:48] [ info] [storage] in-memory
[2021/10/08 13:41:48] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/10/08 13:41:48] [ info] [input:tcp:tcp.0] listening on 127.0.0.1:8877
[2021/10/08 13:41:48] [ info] [input:forward:forward.1] listening on unix:///var/run/fluent.sock
[2021/10/08 13:41:48] [debug] [in_fw] Listen='127.0.0.1' TCP_Port=24224
[2021/10/08 13:41:48] [ info] [input:forward:forward.2] listening on 127.0.0.1:24224
[2021/10/08 13:41:48] [debug] [gelf:gelf.0] created event channels: read=22 write=23
[2021/10/08 13:41:48] [debug] [newrelic:newrelic.1] created event channels: read=24 write=25
[2021/10/08 13:41:49] [debug] [null:null.2] created event channels: read=26 write=30

The filter section is not loaded correctly, wdyt?

PettitWesley commented 3 years ago

@giammbo What Fluent Bit version are you using?

Also, where does this come from?

Output plugin 'musement-musement-core-app' cannot be loaded

Share your full task def please.

giammbo commented 3 years ago

@PettitWesley i've used the latest and the stable versione, the error is the same for both

this is my task definition:

  BlueTaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      ContainerDefinitions:
        # PHP-FPM
        - Image: !Ref FpmDockerImageName
          Essential: true
          LogConfiguration:
            LogDriver: awsfirelens
            Options:
              Name: !Sub '${Namespace}-${ProjectName}-app'
              region: !Ref AWS::Region
              delivery_stream: !Ref ProjectName
          Name: !Sub '${ProjectName}-php-fpm'
          Secrets:
            - Name: DOTENV_FILE
              ValueFrom: !Ref DotenvSecretArn
            - Name: DOTENV_DOTLOCAL_FILE
              ValueFrom: !Ref DotenvDotLocalSecretArn
        # NGINX
        - Image: !Ref NginxDockerImageName
          Essential: true
          Name: !Sub '${ProjectName}-nginx'
          PortMappings:
            - ContainerPort: 80
              Protocol: tcp
          LogConfiguration:
            LogDriver: awsfirelens
            Options:
              Name: !Sub '${Namespace}-${ProjectName}-app'
              region: !Ref AWS::Region
              delivery_stream: !Ref ProjectName
        # FLUENT-BIT
        # FIX ME: create a docker images and remove the command
        - Command:
            - bash
            - -c
            - >
              set -e
              ; curl -L -o /fluent-bit/newrelic.so "https://github.com/newrelic/newrelic-fluent-bit-output/releases/download/v1.7.0/out_newrelic-linux-amd64-1.7.0.so"
              ; chmod +x /fluent-bit/newrelic.so
              ; printf "[PLUGINS]\n  Path /fluent-bit/newrelic.so\n" > /fluent-bit/etc/plugins.conf
              ; echo "${FLUENTBIT_CONFIG}" > /opt/fluentbit.conf
              ; /entrypoint.sh
          Environment:
            - Name: "FLUENTBIT_CONFIG"
              Value: |
                [SERVICE]
                  Plugins_File plugins.conf
                [OUTPUT]
                  Name                    gelf
                  Match                   *
                  Host                    graylog
                  Port                    12201
                  Mode                    tcp
                  Gelf_Short_Message_Key  log
                [OUTPUT]
                  Name newrelic
                  Match *
                  endpoint https://log-api.eu.newrelic.com/log/v1
                  licenseKey ${NR_LICENSE_KEY}
          Essential: false
          Image: !Sub '906394416424.dkr.ecr.${AWS::Region}.amazonaws.com/aws-for-fluent-bit:stable'
          Name: log_router
          FirelensConfiguration:
            Type: fluentbit
            Options:
              config-file-type: 'file'
              config-file-value: '/opt/fluentbit.conf'
              enable-ecs-log-metadata: 'true'
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: !Sub '${Namespace}-${ProjectName}-fluentbit'
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: !Ref ProjectName
          Secrets:
            - Name: NR_LICENSE_KEY
              ValueFrom: !Ref NewRelicLicenseKeySecretArn
      Cpu: !Select [0, !Ref TaskResources]
      ExecutionRoleArn: !Sub 'arn:aws:iam::${AWS::AccountId}:role/${Namespace}-${ProjectName}-ecs-execution'
      Family: !Ref ProjectName
      Memory: !Select [1, !Ref TaskResources]
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      TaskRoleArn: !Sub 'arn:aws:iam::${AWS::AccountId}:role/${Namespace}-${ProjectName}-ecs-task'
PettitWesley commented 3 years ago

@giammbo I see one obvious problem:

LogConfiguration:
            LogDriver: awsfirelens
            Options:
              Name: !Sub '${Namespace}-${ProjectName}-app'
              region: !Ref AWS::Region
              delivery_stream: !Ref ProjectName

When you use the awsfirelens log driver, the options are turned into an auto-generated output configuration: https://aws.amazon.com/blogs/containers/under-the-hood-firelens-for-amazon-ecs-tasks/

In this case, you will get:

[OUTPUT]
     Name: !Sub '${Namespace}-${ProjectName}-app'
     region: !Ref AWS::Region
     delivery_stream: !Ref ProjectName

With each key replaced with its value... and thus this will be an invalid configuration. Clearly you want to use the kinesis_firehose plugin, that should be the value for Name. The Name key is the name of the output plugin.

Also I'd recommend fixing this too, never know if something is going wrong here... and building an image with your config is the standard/recommended practice:

# FLUENT-BIT
        # FIX ME: create a docker images and remove the command
        - Command:
            - bash
            - -c
            - >
              set -e
              ; curl -L -o /fluent-bit/newrelic.so "https://github.com/newrelic/newrelic-fluent-bit-output/releases/download/v1.7.0/out_newrelic-linux-amd64-1.7.0.so"
              ; chmod +x /fluent-bit/newrelic.so
              ; printf "[PLUGINS]\n  Path /fluent-bit/newrelic.so\n" > /fluent-bit/etc/plugins.conf
              ; echo "${FLUENTBIT_CONFIG}" > /opt/fluentbit.conf
              ; /entrypoint.sh
          Environment:
            - Name: "FLUENTBIT_CONFIG"
matthewfala commented 2 years ago

Hi @karthikeayan Referring to the blog Wesley wrote a while back, https://aws.amazon.com/blogs/containers/under-the-hood-firelens-for-amazon-ecs-tasks/#attachment_544

"If the user provided Fluent Bit config file includes a log source, ECS Metadata will be added to the logs it ingests even though the source comes after the ECS Metadata transformer."

I discussed the functionality of fluent bit in terms of always running all the inputs before the filters with Wesley, and he confirmed that this is the expected behavior of Fluent Bit, even when the fluent bit config file is out of order.

This may have been confused previously with the order of the config file, which will be out of order but should not matter.

I tested the following config file which has record modifier set before the tail input. 

[SERVICE]
     Grace 30
     Log_Level debug
[FILTER]
     Name record_modifier
     Match *
     Record ecs_cluster abcd
     Record ecs_task_arn abcd
     Record ecs_task_definition abcd

[INPUT]
     Name tail
     path ./.vscode/data/*.log
     tag tail-record-modifier

[OUTPUT]
     Name cloudwatch_logs
     Match *
     log_stream_prefix batch3-2_
     log_group_name dsr1
     auto_create_group true
     region us-west-2

The results were normal. You can see that the ecs metadata was added to the logs. normal-ecs-metadata

I noticed that in your config file you have cloudwatch's log_key option set to log. Not sure if this is what you want because it will get rid of all the ecs metadata and only record the value of the log_key key.

Please let me know getting rid of log_key log option resolves your issue. Otherwise I can't think of why you are only seeing ecs metadata appended to some logs and not to others.

To reiterate, a custom aws-for-fluent-bit image should not be needed for the functionality of appending ECS metadata (automatically) to logs from custom input plugins.

PettitWesley commented 2 years ago

It is also important to remember that if you parse your logs with a parser and you do not set Reserve_Data True in the filter definition (see below) then you'll lose the extra metadata fields... I just ran into this myself and was trying to figure out why it happens...

[FILTER]
    Name parser
    Match fb_metrics
    Key_Name exec
    Parser fluentbit_prom_metrics_to_json
    Reserve_Data True
otbe commented 2 years ago

EDIT: This is not needed and I can't believe I forgot how FireLens/Fluent Bit works, please see Matt's comment: #240 (comment)

Actually, I just thought of another workaround that might work...

First, you use the techniques in this blog to create your own config file which will be the "main" config file and then customize the entrypoint to use that file: https://aws.amazon.com/blogs/containers/how-to-set-fluentd-and-fluent-bit-input-parameters-in-firelens/

In that blog, I just ignore the auto-generated config file, which would still be created. However, you could actually @include it to import it.

So your main config (in the blog "/fluent-bit/alt/fluent-bit.conf") would have this:

# add your tail input here
[INPUT]
      Name tail

# import the auto-generated file to get the metadata filter
@INCLUDE /fluent-bit/etc/fluent-bit.conf 

# add your output below
[OUTPUT]

Does that make sense? Can you try it out? I probably need to create an example for this...

@PettitWesley really dont want to hijack this issue but I really wonder if this works the way I think it should work :) What I want to do is basically adding filesystem buffering because my firelens sidecar always goes OOM. So I found this https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/oomkill-prevention and your blog post https://aws.amazon.com/de/blogs/containers/how-to-set-fluentd-and-fluent-bit-input-parameters-in-firelens/. This works in general but I need at least the [Output] part which is auto generated by firelens. This is crucial for me because I want to add custom output configuration per container. So my app logs into our main log system and different sidecar (otel) logs into more lightweight system.

What I come up with after reading this issue is basically something like this:

[SERVICE]
    flush                     1
    storage.path              /var/log/flb-storage/
    storage.sync              normal
    storage.checksum          off
    storage.backlog.mem_limit 5M

[INPUT]
    Name forward
    unix_path /var/run/fluent.sock
    storage.type filesystem
    Mem_Buf_Limit 25MB

@INCLUDE /fluent-bit/etc/fluent-bit.conf

and my task definitions look like this 

new FireLensLogDriver({
  options: {
    Name: 'loki',
    host: 'loki-host',
    port: '3000',
    labels: `source=app, service=foo`,
    tls: 'on',
    'storage.total_limit_size': '500M',
  },
  secretOptions: {
    tenant_id: lokiTenant,
  },
});

For the sake of simlicity both containers that should log via firelens use the same firelens config in their container definitions (besides loki labels). This seems to work. I can see the logs with proper labels. I basically get 3 inputs and 3 outputs configured in fluentbit. The output of the logRouter container

Fluent Bit v1.9.3 | logRouter
-- | --
14/07/2022, 20:10:32 | * Copyright (C) 2015-2022 The Fluent Bit Authors | logRouter
14/07/2022, 20:10:32 | * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd | logRouter
14/07/2022, 20:10:32 | * https://fluentbit.io | logRouter
14/07/2022, 20:10:32 | [2022/07/14 18:10:32] [ info] [fluent bit] version=1.9.3, commit=eb4e2e770f, pid=1 | logRouter
14/07/2022, 20:10:32 | [2022/07/14 18:10:32] [ info] [storage] created root path /var/log/flb-storage/ | logRouter
14/07/2022, 20:10:32 | [2022/07/14 18:10:32] [ info] [storage] version=1.2.0, type=memory+filesystem, sync=normal, checksum=disabled, max_chunks_up=128 | logRouter
14/07/2022, 20:10:32 | [2022/07/14 18:10:32] [ info] [storage] backlog input plugin: storage_backlog.4 | logRouter
14/07/2022, 20:10:32 | [2022/07/14 18:10:32] [ info] [cmetrics] version=0.3.1 | logRouter
14/07/2022, 20:10:32 | [2022/07/14 18:10:32] [ info] [input:forward:forward.0] listening on unix:///var/run/fluent.sock | logRouter
14/07/2022, 20:10:32 | [2022/07/14 18:10:32] [ info] [input:forward:forward.1] listening on unix:///var/run/fluent.sock | logRouter
14/07/2022, 20:10:32 | [2022/07/14 18:10:32] [ info] [input:forward:forward.2] listening on 127.0.0.1:24224 | logRouter
14/07/2022, 20:10:32 | [2022/07/14 18:10:32] [ info] [input:tcp:tcp.3] listening on 127.0.0.1:8877 | logRouter
14/07/2022, 20:10:32 | [2022/07/14 18:10:32] [ info] [input:storage_backlog:storage_backlog.4] queue memory limit: 4.8M | logRouter
14/07/2022, 20:10:32 | [2022/07/14 18:10:32] [ info] [output:null:null.0] worker #0 started | logRouter
14/07/2022, 20:10:32 | [2022/07/14 18:10:32] [ info] [output:loki:loki.1] configured, hostname=loki-host:3000 | logRouter
14/07/2022, 20:10:32 | [2022/07/14 18:10:32] [ info] [output:loki:loki.2] configured, hostname=loki-host:3000 | logRouter
14/07/2022, 20:10:32 | [2022/07/14 18:10:32] [ info] [sp] stream processor started

Am I right if I assume both containers will use the proper inputs (the one I configured with storage.type and Mem_Buf_Limit)?

If this is the way how we can integrate filesystem buffering with firelens generated fluentbit config I would be happy to add it to the examples :)

Thank you!

PettitWesley commented 2 years ago

his works in general but I need at least the [Output] part which is auto generated by firelens. This is crucial for me because I want to add custom output configuration per container.

@otbe You do not need to add your output configuration in the Task Definition logConfiguration options for each container. There is another way.

As noted in this blog, FireLens sets the tag for container stdout/stderr logs to be {container name}-firelens-{task ID}: https://aws.amazon.com/blogs/containers/under-the-hood-firelens-for-amazon-ecs-tasks/

Hence, if you have containers that are for example named app, proxy, and metrics_agent, then you could just have output definitions in your fluent bit config file like:

[OUTPUT]
     # this the output for app logs
     Match app-firelens*
     # other output fields omittted
     ... 

[OUTPUT]
     # this the output for proxy logs
     Match proxy-firelens*
     # other output fields omittted
     ... 

[OUTPUT]
     # this the output for metrics_agent logs
     Match metrics_agent-firelens*
     # other output fields omittted
     ...

So then you would not need to @INCLUDE the auto generated config file. If you include that one it would include all of the auto-generated inputs that do not have mem_buf_limit set... I am not actually sure how this will work if you also have your own forward input listening on the same path... can you please try enabling debug logging: https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#enable-debug-logging

And then if you do that the first thing printed will be how many inputs were registered... I am worried you will end up with two identical forward inputs...

otbe commented 2 years ago

I am worried you will end up with two identical forward inputs... Thats exactly what I thought as well.

Indeed, I can use Match app-firelens* and I already tested this and it works great but we lose flexibility this way somehow. The idea of our custom fluent-bit docker image (based on aws-for-fluent-bit) was that it can be used by many services and must not have to be build for each of them. Thats why I liked the idea to configure it via cloudformation/logdriver settings.

lets see how far we get with this approach :) Load testing still has to be executed.

thank you very much @PettitWesley

sparrc commented 3 weeks ago

Doesn't seem like there's any more actions to take on this issue, please feel free to reopen or open a new issue if any problems persist.