Open jonparker opened 9 months ago
So in both cases, there are still no records being sent to the sink? If the export is not fully completed (with all data from the export making it to the sink), then the pipeline will never begin processing the latest shards from streams, as indicated by the INFO log that says no new shards have been acquired. It is difficult to know why the sink would not be receiving any records without knowing the full logs and metrics
@graytaylor0 Actually after another check I realised there is a difference between the two pipelines.
The one that failed had a processor:
processor:
- drop_events:
drop_when: "/source == 'Source1' or contains(/created, '2021-') or contains(/created, '2020-')"
I removed the processor from the second pipeline as I wasn't sure if it was the cause of the issue.
However for the broken pipeline I can see it is logging INFO org.opensearch.dataprepper.plugins.source.dynamodb.export.DataFileScheduler - All Exports are done, streaming can continue...
but then there are no documents sent to the sink and it just logs INFO org.opensearch.dataprepper.plugins.source.dynamodb.leader.ShardManager - Listing shards (DescribeStream call) took 27 milliseconds with 0 shards found
multiple times.
There is currently no data being written to the DDB table so it makes sense that there would be nothing in the DDB stream but it should still send all the records from the S3 export to the sink.
These are the only log messages from the org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink
logger:
The successful pipeline that I created with a bigger min and max OCU was successful and I can see the org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink
logger logged out some warning that the first pipeline didn't:
It looks like in your first pipeline, the drop_events
processor was dropping all of the export data, keeping it from going into OpenSearch.
On this second pipeline, it looks like document errors on the sink. Is this happening for all documents or select documents? It is recommended to configure a DLQ to send failed documents to.
Also, the error messages from OpenSearch are cut off, but it looks like there is something in your data that OpenSearch does not like with the created
field
Ok I guess the processor might have dropped all events but I would have expected some informational logs to indicate that all the records have at least been processed. I guess the processor doesn't know how many records there are or even if they have come from an export or a stream but it would be great if it could log out some high level metrics on what it has processed for visibility.
The errors with the other pipeline are fine as they are only for a few records that have dates that don't have valid UTC values. I can see in the pipeline metrics that the second pipeline successfully sent most of the records to the sink and I can also see them in the Open search collection.
Describe the bug
AWS Opensearch Ingestion pipeline with DynamoDB
The pipeline circuit breaker flips open and closed.
After stopping the pipeline and re-configuring it to have more min and max OCUs (6-10 Ingestion-OCU) the flip flop of the circuit breaker no longer happens but it gets stuck trying to process the shards and continuously reports 0 shards found or no shards acquired.
To Reproduce Steps to reproduce the behavior:
Expected behavior The pipeline recovers and processes all records.
Environment (please complete the following information):
Additional context Add any other context about the problem here.