search

aws / aws-sdk-java-v2

The official AWS SDK for Java - Version 2
Apache License 2.0
2.16k stars 835 forks source link

Dynamodb enhanced client does not store and retrieve ddb item once changes are deployed via pipeline. #3739

Closed user-12366 closed 1 year ago

user-12366 commented 1 year ago

Describe the bug

PutItem, GetItem, etc. apis are not storing and retrieving the object into the dynamodb table.

Our service exposes a set of swagger apis that allow for storing and retrieving objects into the dynamodb table. We are using enhanced dynamodb client to make PutItem, GetItem, etc. calls to the table.

When call these swagger apis to store an object, it returns with successfully api response; however, the dynamodb wrapper/repository class that uses the enhanced client to store the value doesn't store it, neither does it throw any error.

Expected Behavior

Expected behavior is when the swagger api is called and then the repository class methods are called, then the object of type ValidationExemptions will be stored in the dynamodb table.

Current Behavior

When calling the swagger apis, the repository class methods are called but it doesn't save the object into the table.

Unit tests using the DynamodbLocal dependency works fine.

Reproduction Steps

Client creation code:

import lombok.AllArgsConstructor;
import lombok.NoArgsConstructor;
import lombok.extern.log4j.Log4j2;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;

@Log4j2
@NoArgsConstructor
@AllArgsConstructor
@Configuration
public class DynamoDBConfiguration {

  @Autowired AwsCredentialsProvider awsCredentialsProvider;

  @Bean
  public DynamoDbEnhancedClient getDynamoDbEnhancedClient() {
    return DynamoDbEnhancedClient.builder().dynamoDbClient(getDynamoDbClient()).build();
  }

  @Bean
  public DynamoDbClient getDynamoDbClient() {
    return DynamoDbClient.builder()
        .credentialsProvider(awsCredentialsProvider)
        .region(Region.US_WEST_2)
        .build();
  }
}

DDB bean class:

import java.util.List;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.EqualsAndHashCode;
import lombok.NoArgsConstructor;
import lombok.ToString;
import lombok.extern.jackson.Jacksonized;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbAttribute;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbBean;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbPartitionKey;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbSortKey;

/** POJO representing validation exemptions stored in Dynamodb table. */
@Jacksonized
@ToString
@Builder
@DynamoDbBean
@NoArgsConstructor
@AllArgsConstructor
@EqualsAndHashCode
public class ValidationExemption {

  private String fullyQualifiedSchemaName;
  private String exemptionType;
  private List<String> fullyQualifiedFieldNames;
  // Unix epoch time format in seconds
  // For details, refer
  // https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/time-to-live-ttl-before-you-start.html#time-to-live-ttl-before-you-start-formatting
  private long timeToLive;

  @DynamoDbPartitionKey
  @DynamoDbAttribute(value = "fully-qualified-schema-name")
  public String getFullyQualifiedSchemaName() {
    return fullyQualifiedSchemaName;
  }

  public void setFullyQualifiedSchemaName(String fullyQualifiedSchemaName) {
    this.fullyQualifiedSchemaName = fullyQualifiedSchemaName;
  }

  @DynamoDbSortKey
  @DynamoDbAttribute(value = "exemption-type")
  public String getExemptionType() {
    return exemptionType;
  }

  public void setExemptionType(String exemptionType) {
    this.exemptionType = exemptionType;
  }

  @DynamoDbAttribute(value = "fully-qualified-field-names")
  public List<String> getFullyQualifiedFieldNames() {
    return fullyQualifiedFieldNames;
  }

  public void setFullyQualifiedFieldNames(List<String> fullyQualifiedFieldNames) {
    this.fullyQualifiedFieldNames = fullyQualifiedFieldNames;
  }

  @DynamoDbAttribute(value = "ttl")
  public long getTimeToLive() {
    return timeToLive;
  }

  public void setTimeToLive(long timeToLive) {
    this.timeToLive = timeToLive;
  }
}

Repository class making CRUD calls.

import java.util.Collections;
import java.util.List;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.stream.Collectors;
import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.ToString;
import lombok.extern.log4j.Log4j2;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Repository;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbTable;
import software.amazon.awssdk.enhanced.dynamodb.Key;
import software.amazon.awssdk.enhanced.dynamodb.TableSchema;
import software.amazon.awssdk.enhanced.dynamodb.model.DeleteItemEnhancedRequest;
import software.amazon.awssdk.enhanced.dynamodb.model.GetItemEnhancedRequest;
import software.amazon.awssdk.enhanced.dynamodb.model.PageIterable;
import software.amazon.awssdk.enhanced.dynamodb.model.PutItemEnhancedRequest;
import software.amazon.awssdk.enhanced.dynamodb.model.QueryConditional;
import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
import software.amazon.awssdk.services.dynamodb.model.ResourceNotFoundException;

/**
 * This class provides a set of methods to perform CRUD actions on dynamodb table storing validation
 * exemptions.
 */
@Log4j2
@AllArgsConstructor
@NoArgsConstructor
@Repository
public class ValidationExemptionRepository {

  @Autowired private DynamoDBConfiguration dynamoDBConfiguration;

  @Autowired private String environment;

  private DynamoDbTable<ValidationExemption> getValidationExemptionTable() {
    // Create a table scheme to scan our bean class ValidationExemption
    String VALIDATION_EXEMPTION_TABLE_NAME = "schema-validator-exemptions-" + environment;
    return dynamoDBConfiguration
        .getDynamoDbEnhancedClient()
        .table(VALIDATION_EXEMPTION_TABLE_NAME, TableSchema.fromBean(ValidationExemption.class));
  }

  public void recordExemption(final ValidationExemption validationExemption) {
    try {
      DynamoDbTable<ValidationExemption> validationExemptionTable = getValidationExemptionTable();
      validationExemptionTable.putItem(
          (PutItemEnhancedRequest.Builder<ValidationExemption> requestBuilder) ->
              requestBuilder.item(validationExemption));
    } catch (UnsupportedOperationException | DynamoDbException e) {
      log.error(e.getMessage());
    }
  }

  public ExemptionUpdateStatus updateExemption(
      final String fullyQualifiedSchemaName,
      final String exemptionType,
      final List<String> newFieldNames,
      final int newTimeToLive) {
    try {
      DynamoDbTable<ValidationExemption> validationExemptionTable = getValidationExemptionTable();
      Key key =
          Key.builder().partitionValue(fullyQualifiedSchemaName).sortValue(exemptionType).build();
      ValidationExemption validationExemption =
          validationExemptionTable.getItem(
              (GetItemEnhancedRequest.Builder requestBuilder) -> requestBuilder.key(key));
      AtomicBoolean isChangeRequested = new AtomicBoolean(false);
      if (validationExemption != null) {
        if (newTimeToLive > 0 && newTimeToLive != validationExemption.getTimeToLive()) {
          validationExemption.setTimeToLive(newTimeToLive);
          isChangeRequested.set(true);
        }
        if (!newFieldNames.isEmpty()) {
          List<String> existingFieldNames = validationExemption.getFullyQualifiedFieldNames();
          newFieldNames.forEach(
              newFieldName -> {
                if (!existingFieldNames.contains(newFieldName)) {
                  existingFieldNames.add(newFieldName);
                  isChangeRequested.set(true);
                }
              });
          validationExemption.setFullyQualifiedFieldNames(existingFieldNames);
        }
        if (isChangeRequested.get()) {
          validationExemptionTable.putItem(
              (PutItemEnhancedRequest.Builder<ValidationExemption> requestBuilder) ->
                  requestBuilder.item(validationExemption));
          return ExemptionUpdateStatus.UPDATED;
        }
        return ExemptionUpdateStatus.NOT_MODIFIED;
      }
    } catch (UnsupportedOperationException | DynamoDbException e) {
      log.error(e.getMessage());
    }
    return ExemptionUpdateStatus.NOT_FOUND;
  }

  public boolean deleteExemption(
      final String fullyQualifiedSchemaName, final String exemptionType) {
    ValidationExemption validationExemption = null;
    try {
      DynamoDbTable<ValidationExemption> validationExemptionTable = getValidationExemptionTable();
      Key key =
          Key.builder().partitionValue(fullyQualifiedSchemaName).sortValue(exemptionType).build();
      validationExemption =
          validationExemptionTable.deleteItem(
              (DeleteItemEnhancedRequest.Builder requestBuilder) -> requestBuilder.key(key));
    } catch (UnsupportedOperationException | DynamoDbException e) {
      log.error(e.getMessage());
    }
    return validationExemption != null;
  }

  public List<ValidationExemption> fetchExemption(
      final String fullyQualifiedSchemaName, final String exemptionType) {
    DynamoDbTable<ValidationExemption> validationExemptionTable = getValidationExemptionTable();
    Key key =
        Key.builder().partitionValue(fullyQualifiedSchemaName).sortValue(exemptionType).build();
    try {
      ValidationExemption validationExemption =
          validationExemptionTable.getItem(
              (GetItemEnhancedRequest.Builder requestBuilder) -> requestBuilder.key(key));
      return validationExemption != null
          ? Collections.singletonList(validationExemption)
          : Collections.emptyList();
    } catch (ResourceNotFoundException e) {
      return Collections.emptyList();
    } catch (UnsupportedOperationException | DynamoDbException e) {
      log.error(e.getMessage());
      return null;
    }
  }

  public List<ValidationExemption> fetchExemptions(final String fullyQualifiedSchemaName) {
    DynamoDbTable<ValidationExemption> validationExemptionTable = getValidationExemptionTable();
    QueryConditional queryConditional =
        QueryConditional.keyEqualTo(Key.builder().partitionValue(fullyQualifiedSchemaName).build());
    try {
      PageIterable<ValidationExemption> resultPages =
          validationExemptionTable.query(r -> r.queryConditional(queryConditional));
      return resultPages.items().stream().collect(Collectors.toList());
    } catch (ResourceNotFoundException e) {
      return Collections.emptyList();
    } catch (UnsupportedOperationException | DynamoDbException e) {
      log.error(e.getMessage());
      return null;
    }
  }

  public List<ValidationExemption> fetchAllExemptions(final String exemptionType) {
    PageIterable<ValidationExemption> resultPages = scanAllExemptions();
    return resultPages.items().stream()
        .filter(item -> item.getExemptionType().equals(exemptionType))
        .collect(Collectors.toList());
  }

  public List<ValidationExemption> fetchAllExemptions() {
    PageIterable<ValidationExemption> resultPages = scanAllExemptions();
    return resultPages.items().stream().collect(Collectors.toList());
  }

  private PageIterable<ValidationExemption> scanAllExemptions() {
    try {
      DynamoDbTable<ValidationExemption> validationExemptionTable = getValidationExemptionTable();
      return validationExemptionTable.scan();
    } catch (UnsupportedOperationException | DynamoDbException e) {
      log.error(e.getMessage());
      return null;
    }
  }

  @Getter
  @ToString
  public enum ExemptionUpdateStatus {
    UPDATED("UPDATED"),
    NOT_MODIFIED("NOT_MODIFIED"),
    NOT_FOUND("NOT_FOUND");

    private final String updateStatus;

    ExemptionUpdateStatus(String updateStatus) {
      this.updateStatus = updateStatus;
    }
  }
}

Possible Solution

No response

Additional Information/Context

Log statements with aws sdk java request ids.

msg [m[32m2023-02-07T23:11:45.142Z [http-nio-8080-exec-2] INFO  *.schemavalidator.controller.AdminController - Successfully added validation exemption: ValidationExemption(fullyQualifiedSchemaName=testing.zillow.com.testingonfeb7thmudra, exemptionType=tagging, fullyQualifiedFieldNames=[filed1], timeToLive=1672578000)
--
msg [m[32m2023-02-07T23:11:45.137Z [http-nio-8080-exec-2] INFO  *.schemavalidator.repositories.ValidationExemptionRepository - Successfully made put api call
msg [m[36m2023-02-07T23:11:45.121Z [http-nio-8080-exec-2] DEBUG software.amazon.awssdk.request - Received successful response: 200, Request ID: 8SP4SUBONME4AQ6F3QC0Q5KPR7VV4KQNSO5AEMVJF66Q9ASUAAJG, Extended Request ID: not available
msg [m[36m2023-02-07T23:11:45.115Z [http-nio-8080-exec-2] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=POST, protocol=https, host=dynamodb.us-west-2.amazonaws.com, encodedPath=/, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, User-Agent, X-Amz-Target], queryParameters=[])
msg [m[32m2023-02-07T23:11:45.091Z [http-nio-8080-exec-2] INFO  *.schemavalidator.repositories.ValidationExemptionRepository - Accessing table: schema-validator-exemptions-dev
msg [m[36m2023-02-07T23:11:45.073Z [http-nio-8080-exec-2] DEBUG software.amazon.awssdk.request - Received successful response: 200, Request ID: VF1NIABC43CVAA4E57OHDMDIKNVV4KQNSO5AEMVJF66Q9ASUAAJG, Extended Request ID: not available
msg [m[36m2023-02-07T23:11:44.936Z [http-nio-8080-exec-2] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=POST, protocol=https, host=dynamodb.us-west-2.amazonaws.com, encodedPath=/, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, User-Agent, X-Amz-Target], queryParameters=[])
msg [m[36m2023-02-07T23:11:44.850Z [http-nio-8080-exec-2] DEBUG software.amazon.awssdk.request - Received successful response: 200, Request ID: 83b6bfe5-484f-4846-b042-4e95daced768, Extended Request ID: not available
msg [m[36m2023-02-07T23:11:44.551Z [http-nio-8080-exec-2] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=POST, protocol=https, host=sts.us-west-2.amazonaws.com, encodedPath=, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, User-Agent], queryParameters=[])
msg [m[32m2023-02-07T23:11:44.124Z [http-nio-8080-exec-2] INFO  *.schemavalidator.repositories.ValidationExemptionRepository - Accessing table: schema-validator-exemptions-dev
msg [m[32m2023-02-07T23:11:44.123Z [http-nio-8080-exec-2] INFO  *.schemavalidator.controller.AdminController - Validation exemption request body: ValidationExemptionRequest(schemaName=testingonfeb7thmudra, schemaNamespace=testing.zillow.com, fieldNames=[filed1], timeToLive=1672578000, exemptionType=ExemptionType.tagging(exemptionTypeName=tagging))

"msg
2023-02-07T23:11:45.142Z [http-nio-8080-exec-2] INFO  *.schemavalidator.controller.AdminController - Successfully added validation exemption: ValidationExemption(fullyQualifiedSchemaName=testing.zillow.com.testingonfeb7thmudra, exemptionType=tagging, fullyQualifiedFieldNames=[filed1], timeToLive=1672578000)"
"msg
2023-02-07T23:11:45.137Z [http-nio-8080-exec-2] INFO  *.schemavalidator.repositories.ValidationExemptionRepository - Successfully made put api call"
"msg
2023-02-07T23:11:45.121Z [http-nio-8080-exec-2] DEBUG software.amazon.awssdk.request - Received successful response: 200, Request ID: 8SP4SUBONME4AQ6F3QC0Q5KPR7VV4KQNSO5AEMVJF66Q9ASUAAJG, Extended Request ID: not available"
"msg
[m[36m2023-02-07T23:11:45.115Z [http-nio-8080-exec-2] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=POST, protocol=https, host=[dynamodb.us-west-2.amazonaws.com](http://dynamodb.us-west-2.amazonaws.com/), encodedPath=/, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, User-Agent, X-Amz-Target], queryParameters=[])"
"msg
2023-02-07T23:11:45.091Z [http-nio-8080-exec-2] INFO  *.schemavalidator.repositories.ValidationExemptionRepository - Accessing table: schema-validator-exemptions-dev"
"msg
2023-02-07T23:11:45.073Z [http-nio-8080-exec-2] DEBUG software.amazon.awssdk.request - Received successful response: 200, Request ID: VF1NIABC43CVAA4E57OHDMDIKNVV4KQNSO5AEMVJF66Q9ASUAAJG, Extended Request ID: not available"
"msg
[m[36m2023-02-07T23:11:44.936Z [http-nio-8080-exec-2] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=POST, protocol=https, host=[dynamodb.us-west-2.amazonaws.com](http://dynamodb.us-west-2.amazonaws.com/), encodedPath=/, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, User-Agent, X-Amz-Target], queryParameters=[])"
"msg
2023-02-07T23:11:44.850Z [http-nio-8080-exec-2] DEBUG software.amazon.awssdk.request - Received successful response: 200, Request ID: 83b6bfe5-484f-4846-b042-4e95daced768, Extended Request ID: not available"
"msg
[m[36m2023-02-07T23:11:44.551Z [http-nio-8080-exec-2] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=POST, protocol=https, host=[sts.us-west-2.amazonaws.com](http://sts.us-west-2.amazonaws.com/), encodedPath=, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, User-Agent], queryParameters=[])"
"msg
2023-02-07T23:11:44.124Z [http-nio-8080-exec-2] INFO  *.schemavalidator.repositories.ValidationExemptionRepository - Accessing table: schema-validator-exemptions-dev"
"msg
[m[32m2023-02-07T23:11:44.123Z [http-nio-8080-exec-2] INFO  *.schemavalidator.controller.AdminController - Validation exemption request body: ValidationExemptionRequest(schemaName=testingonfeb7thmudra, schemaNamespace=[testing.zillow.com](http://testing.zillow.com/), fieldNames=[filed1], timeToLive=1672578000, exemptionType=ExemptionType.tagging(exemptionTypeName=tagging))"

AWS Java SDK version used

implementation "software.amazon.awssdk:bom:2.19.28"
implementation "software.amazon.awssdk:dynamodb:2.19.28"
implementation "software.amazon.awssdk:dynamodb-enhanced:2.19.28"
dynamodb "com.amazonaws:DynamoDBLocal:1.16.0"
dynamodb fileTree (dir: 'lib', include: ["*.dylib", "*.so", "*.dll"])

JDK version used

11

Operating System and version

Kubernetes cluster

user-12366 commented 1 year ago

@debora-ito Hello Debora, could you help get this issue triaged? It is blocking our use of enhanced dynamodb client. Gist: The ddb api calls are happening successfully but the ddb items aren't being stored or retrieved from the table successfully. I have shared the aws sdk request ids as well for further investigation.

debora-ito commented 1 year ago

@user-12366 apologies for the delay in response.

Can you enable the verbose wirelogs and check if the PutItem operation is sending the right attributes?

If the low-level DynamoDB APIs are being called successfully, my guess is that some attributes are not matching the expected values. The wirelogs would confirm this.

If you choose to share the wirelogs here, please make sure to redact any sensitive information (like access keys).

For instructions on how to enable wirelogs see https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/logging-slf4j.html#sdk-java-logging-verbose.

github-actions[bot] commented 1 year ago

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.