search

treasure-data / digdag

Workload Automation System
https://www.digdag.io/
Apache License 2.0
1.31k stars 221 forks source link

Revised retry logic for getLog #1626

Closed myui closed 3 years ago

myui commented 3 years ago

Because Cloudwatch log stream became available by an eventually consistency fashion, introduce a retry logic to avoid ResourceNotFoundException.

com.amazonaws.services.logs.model.ResourceNotFoundException: The specified log stream does not exist. (Service: AWSLogs; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: xxxx-xxxx-xxx)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
    at com.amazonaws.services.logs.AWSLogsClient.doInvoke(AWSLogsClient.java:3045)
    at com.amazonaws.services.logs.AWSLogsClient.invoke(AWSLogsClient.java:3012)
    at com.amazonaws.services.logs.AWSLogsClient.invoke(AWSLogsClient.java:3001)
    at com.amazonaws.services.logs.AWSLogsClient.executeGetLogEvents(AWSLogsClient.java:1856)
    at com.amazonaws.services.logs.AWSLogsClient.getLogEvents(AWSLogsClient.java:1827)
    at io.digdag.standards.command.ecs.DefaultEcsClient.lambda$getLog$7(DefaultEcsClient.java:278)
    at io.digdag.standards.command.ecs.DefaultEcsClient.retryOnRateLimit(DefaultEcsClient.java:305)
    at io.digdag.standards.command.ecs.DefaultEcsClient.getLog(DefaultEcsClient.java:278)
    at io.digdag.standards.command.EcsCommandExecutor.fetchLogEvents(EcsCommandExecutor.java:441)