DynamoDBMapperConfig.PaginationLoadingStrategy.ITERATION_ONLY is not compatible with scan queries

sshevlyagin commented 4 years ago

Expected Behavior

It's possible to run queries that filter by a value with DynamoDBMapperConfig.PaginationLoadingStrategy.ITERATION_ONLY

Actual Behavior

You get the following error:

java.lang.UnsupportedOperationException: The list could only be iterated once in ITERATION_ONLY mode.
    at com.amazonaws.services.dynamodbv2.datamodeling.PaginatedList$PaginatedListIterator.<init>(PaginatedList.java:231)
    at com.amazonaws.services.dynamodbv2.datamodeling.PaginatedList.iterator(PaginatedList.java:204)
    at java.util.Spliterators$IteratorSpliterator.estimateSize(Spliterators.java:1821)

Steps to Reproduce the Problem

Set up the following DynamoDBMapperConfig

@Bean
public DynamoDBMapperConfig dynamoDBMapperConfig() {
    return new DynamoDBMapperConfig.Builder()
   .withSaveBehavior(DynamoDBMapperConfig.SaveBehavior.UPDATE_SKIP_NULL_ATTRIBUTES)
   .withConsistentReads(DynamoDBMapperConfig.ConsistentReads.EVENTUAL)
.withPaginationLoadingStrategy(DynamoDBMapperConfig.PaginationLoadingStrategy.ITERATION_ONLY)
.withBatchWriteRetryStrategy(DynamoDBMapperConfig.DefaultBatchWriteRetryStrategy.INSTANCE)
.withBatchLoadRetryStrategy(DynamoDBMapperConfig.DefaultBatchLoadRetryStrategy.INSTANCE)
.withTypeConverterFactory(DynamoDBTypeConverterFactory.standard())
.withConversionSchema(ConversionSchemas.V2)
.build();

}

In your repository have a findBy method

@EnableScan
public interface FooRepository extends CrudRepository<FooTableItem, String>{
Iterable<FooItem> findByStatus(String status);
}

Execute the query and try to iterate over the results

private void doIt {
    Iterable<FooItem> items = fooRepository.findByStatus("NEW");
    List<Bar> results = new ArrayList<>();
    items.forEach(item -> {
        try {
            results.add(item.toContext());
        } catch (Throwable t) {
            log.error("Failed to deserialize failure", t);
        }
    });
    return results;
}

Note: Setting a breakpoint on https://github.com/aws/aws-sdk-java/blob/1.11.794/aws-java-sdk-dynamodb/src/main/java/com/amazonaws/services/dynamodbv2/datamodeling/PaginatedList.java#L230 will show you that the PaginatedListIterator was called before the forLoop got there with the following stack.

<init>:234, PaginatedList$PaginatedListIterator (com.amazonaws.services.dynamodbv2.datamodeling)
iterator:204, PaginatedList (com.amazonaws.services.dynamodbv2.datamodeling)
requiresConversion:188, QueryExecutionResultHandler (org.springframework.data.repository.core.support)
postProcessInvocationResult:157, QueryExecutionResultHandler (org.springframework.data.repository.core.support)
postProcessInvocationResult:77, QueryExecutionResultHandler (org.springframework.data.repository.core.support)
invoke:605, RepositoryFactorySupport$QueryExecutorMethodInterceptor (org.springframework.data.repository.core.support)
proceed:186, ReflectiveMethodInvocation (org.springframework.aop.framework)
invoke:95, ExposeInvocationInterceptor (org.springframework.aop.interceptor)
proceed:186, ReflectiveMethodInvocation (org.springframework.aop.framework)
invoke:212, JdkDynamicAopProxy (org.springframework.aop.framework)
findByStatus:-1, $Proxy154 (com.sun.proxy)
-REDACTED-
-REDACTED-
invoke0:-1, NativeMethodAccessorImpl (sun.reflect)
invoke:62, NativeMethodAccessorImpl (sun.reflect)
invoke:43, DelegatingMethodAccessorImpl (sun.reflect)
invoke:498, Method (java.lang.reflect)
run:84, ScheduledMethodRunnable (org.springframework.scheduling.support)
run:54, DelegatingErrorHandlingRunnable (org.springframework.scheduling.support)
call:511, Executors$RunnableAdapter (java.util.concurrent)
runAndReset$$$capture:308, FutureTask (java.util.concurrent)
runAndReset:-1, FutureTask (java.util.concurrent)
 - Async stack trace
<init>:151, FutureTask (java.util.concurrent)
<init>:219, ScheduledThreadPoolExecutor$ScheduledFutureTask (java.util.concurrent)
scheduleAtFixedRate:570, ScheduledThreadPoolExecutor (java.util.concurrent)
scheduleAtFixedRate:348, ThreadPoolTaskScheduler (org.springframework.scheduling.concurrent)
scheduleFixedRateTask:479, ScheduledTaskRegistrar (org.springframework.scheduling.config)
scheduleFixedRateTask:453, ScheduledTaskRegistrar (org.springframework.scheduling.config)
scheduleTasks:374, ScheduledTaskRegistrar (org.springframework.scheduling.config)
afterPropertiesSet:349, ScheduledTaskRegistrar (org.springframework.scheduling.config)
finishRegistration:302, ScheduledAnnotationBeanPostProcessor (org.springframework.scheduling.annotation)
onApplicationEvent:233, ScheduledAnnotationBeanPostProcessor (org.springframework.scheduling.annotation)
onApplicationEvent:105, ScheduledAnnotationBeanPostProcessor (org.springframework.scheduling.annotation)
doInvokeListener:172, SimpleApplicationEventMulticaster (org.springframework.context.event)
invokeListener:165, SimpleApplicationEventMulticaster (org.springframework.context.event)
multicastEvent:139, SimpleApplicationEventMulticaster (org.springframework.context.event)
publishEvent:403, AbstractApplicationContext (org.springframework.context.support)
publishEvent:360, AbstractApplicationContext (org.springframework.context.support)
finishRefresh:897, AbstractApplicationContext (org.springframework.context.support)
finishRefresh:162, ServletWebServerApplicationContext (org.springframework.boot.web.servlet.context)
refresh:553, AbstractApplicationContext (org.springframework.context.support)
refresh:141, ServletWebServerApplicationContext (org.springframework.boot.web.servlet.context)
refresh:747, SpringApplication (org.springframework.boot)
refreshContext:397, SpringApplication (org.springframework.boot)
run:315, SpringApplication (org.springframework.boot)
run:1226, SpringApplication (org.springframework.boot)
run:1215, SpringApplication (org.springframework.boot)
REDACTED

Specifications

Spring Data DynamoDB Version: 5.2.4 (2.2)
Spring Data Version: 2.2.7.RELEASE
AWS SDK Version: 1.11.790
Java Version: 1.8.0_181 - Java HotSpot(TM) 64-Bit Server VM 25.181-b13
Platform Details: Mac OS X 10.15.5

boostchicken commented 4 years ago

I have been looking into this for a while. So far what I have found is Spring Data made some changes upstream regarding their conversion service. This means the query get's iterated before it gets back to you. I have not been successful in finding a way to get it to not think that these results are candidates for conversion.

I'll leave this issue open for tracking purposes.

boostchicken commented 4 years ago

@sshevlyagin https://github.com/spring-projects/spring-data-commons/blob/210ab249e373c535796b1ff6f9ccfe1ab9c55a7b/src/main/java/org/springframework/data/repository/core/support/QueryExecutionResultHandler.java#L175-L196

You can see there that they loop through the collection before the result ever gets back to us to verify it does not have to do any type conversion.

sshevlyagin commented 4 years ago

Thanks! Is that a bug on the part of SpringData or is ITERATION_ONLY simply not compatible anymore?

boostchicken commented 4 years ago

for the moment I am going to say ITERATION_ONLY is simply not compatible anymore. Spring Data has changed a lot and if i circumvent this stuff its going to break a lot of expected functionality

boostchicken / spring-data-dynamodb