`.query()` over large data sets with a `filter_condition` does not return expected results

[cooncesean]

Issue Description

When running Dyntastic.query() over a large data set, with a filter_condition, the resulting generator contains zero items (ie: raises StopIteration). The exact same query, with the exact same filter_condition, executed on a smaller data set, returns the expected items from Dynamo.

Video

Slightly difficult to explain over text, so I created a Loom to demonstrate the issue.

Steps To Reproduce

1. Create a new dynatastic model w/ at least one str attribute.
2. Create >1000 records of that model associated to the same p_key
3. Query that model using a filter_expression on the attribute.

[nayaverdier]

Hey, thanks for the detailed video and this definitely sounds like a bug! It might have a very easy fix. Go ahead and install dyntastic==0.11.0a2 and let me know if the issue is resolved. Note that some other changes are also present from the first alpha release, see the changelog.

For more context, the case it seems to cause issues for me is when per_page (or defaults to 1MB of data) is large enough that all records on a "page" are filtered out (since DynamoDB counts records before running the filter condition on them).

And you're correct, you definitely don't want to use a scan here. That will iterate over all records in the entire table on Dynamo's side, rather than just the single partition that the p_key maps to.

[cooncesean]

Great - gonna try this out right now. Thanks so much for the quick response on this 👍.

[cooncesean]

What a legend!

Yup, this works as expected -- thanks for the fix 🏁.

[cooncesean]

Closed with 4b90aaa5d8dec7013de4c9692da44a7aa2766b0d