Kinesis ingestion task ignores fetchDelayMillis and recordsPerFetch parameters in Druid 31.0.0

[Arwiim]

I'm experiencing an issue where the fetchDelayMillis and recordsPerFetch parameters in the tuningConfig of a Kinesis ingestion task are not being applied. These parameters seem to be ignored or removed when the task is submitted, resulting in the ingestion task exceeding Kinesis shard read throughput limits and causing ProvisionedThroughputExceededException errors.

Despite specifying these parameters in the ingestion spec, they do not appear in the running task's configuration, and the task continues to exceed the Kinesis API call limits.

Steps to Reproduce: Set Up Kinesis Stream:

A Kinesis stream named imply_ranty_nrules_stream with 6 shards. Approximately 2,000 records per second are entering the stream. Create the Ingestion Spec:

Here's the ingestion spec used:

{
  "type": "kinesis",
  "spec": {
    "dataSchema": {
      "dataSource": "imply_ranty_nrules_stream_v2",
      "timestampSpec": {
        "column": "time",
        "format": "iso",
        "missingValue": null
      },
      "dimensionsSpec": {
        "dimensions": [
          // Dimension definitions
        ],
        "dimensionExclusions": [
          "__time",
          "!!!_no_such_column_!!!"
        ],
        "includeAllDimensions": false,
        "useSchemaDiscovery": false
      },
      "metricsSpec": [],
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "DAY",
        "queryGranularity": {
          "type": "none"
        },
        "rollup": false,
        "intervals": []
      },
      "transformSpec": {
        "filter": null,
        "transforms": [
          // Transform definitions
        ]
      }
    },
    "ioConfig": {
      "type": "kinesis",
      "stream": "imply_ranty_nrules_stream",
      "endpoint": "kinesis.us-east-1.amazonaws.com",
      "inputFormat": {
        "type": "json"
      },
      "replicas": 1,
      "taskCount": 1,
      "taskDuration": "PT3600S",
      "startDelay": "PT5S",
      "period": "PT30S",
      "useEarliestSequenceNumber": false,
      "completionTimeout": "PT1800S",
      "idleConfig": {
        "enabled": false
      }
    },
    "tuningConfig": {
      "type": "KinesisTuningConfig",
      "maxRowsInMemory": 150000,
      "maxRowsPerSegment": 750000,
      "maxTotalRows": 3000000,
      "intermediatePersistPeriod": "PT10M",
      "fetchDelayMillis": 500,
      "recordsPerFetch": 1000,
      // Additional tuning configurations
    }
  },
  "context": null,
  "suspended": false
}

Note: I've set fetchDelayMillis and recordsPerFetch in the tuningConfig.

Observe the Running Task Configuration:

Checked the task details in the Druid console under the Tasks section. Noticed that the fetchDelayMillis and recordsPerFetch parameters are missing from the running task's tuningConfig.

Monitor Logs and Metrics:

.Despite the parameters being set in the ingestion spec, the ingestion task continues to exceed Kinesis read limits. .The following errors appear in the MiddleManager logs:

com.amazonaws.services.kinesis.model.ProvisionedThroughputExceededException: Rate exceeded for Shard - [shard details] (Service: AmazonKinesis; Status Code: 400; Error Code: ProvisionedThroughputExceededException; ...)

Expected Behavior: The ingestion task should apply the fetchDelayMillis and recordsPerFetch parameters from the tuningConfig. The task should respect the Kinesis shard read throughput limits by throttling GetRecords calls according to the specified parameters. The parameters should appear in the running task's configuration when inspected. Actual Behavior: The fetchDelayMillis and recordsPerFetch parameters are not applied. These parameters are missing from the running task's tuningConfig. The ingestion task exceeds Kinesis read throughput limits, resulting in ProvisionedThroughputExceededException errors. Increasing taskCount or adjusting other parameters does not resolve the issue.