splunk_cwl_firehose_processor update projectedSize value

[chrisdipesa]

Log entry below causes reingestion retries that never seem to be ingested.

body size is too long

Splunk support informed me to change projectedSize > 6000000: to projectedSize > 5000000, which has worked. Source code should be updated as such.

Seems to have been a change with AWS and lambda max size. How AWS articulated this to Splunk

We recently discovered an edge case scenario in the kinesis-firehose-cloudwatch-logs-processor-python Lambda blueprint that can cause the underlying function to enter a recursive invocation and record re-ingestion loop. The Kinesis Data Firehose and Lambda teams are actively working on a fix as top priority tracking for 4/23/21. In the interim, we decided to remove the blueprint to avoid potential customer impact.

for idx, rec in enumerate(records):
        if rec['result'] != 'Ok':
            continue
        projectedSize += len(rec['data']) + len(rec['recordId'])
        # 6000000 instead of 6291456 to leave ample headroom for the stuff we didn't account for
        if projectedSize > 5000000:
            totalRecordsToBeReingested += 1
            recordsToReingest.append(
                getReingestionRecord(isSas, dataByRecordId[rec['recordId']])
            )
            records[idx]['result'] = 'Dropped'
            del(records[idx]['data'])

[1davidmichael]

Thanks @chrisdipesa

I had the same issue and this fixed it. I'd appreciate an update from Splunk if possible. For now I have modified the generated CFN template to directly include the function code inline.