search

ghdna / athena-express

Athena-Express can simplify executing SQL queries in Amazon Athena AND fetching cleaned-up JSON results in the same synchronous or asynchronous request - well suited for web applications.
https://www.npmjs.com/package/athena-express
MIT License
179 stars 70 forks source link

Pagination causing data loss #65

Closed devanshumadan closed 3 years ago

devanshumadan commented 3 years ago

Hi,

While using pagination (and NextToken), I noticed some data loss. When I debugged the code, it looks like the 'getQueryResults' method responds with column names only when 'NextToken' is not present in the request.

In line 96, helper.js - we don't need to add + 1 every time.

Could you please look into this issue?

I have made few changes in the file for your reference (please do a diff check).

To reproduce the issue: Set limit to '3' in the query, and set page size as '1'. You will notice that we get the first record in the first response (without NextToken), but in the subsequent request (with NextToken), we get the third record, and the second record is getting skipped.

Thank you!

ghdna commented 3 years ago

Valid issue. Looking into it.

devanshumadan commented 3 years ago

@ghdna Gary, I tested the changes in the file which I attached. Seems to be working fine. The solution is not very clean though.

ghdna commented 3 years ago

Your solution was pretty solid. I updated it slightly and incorporated in v7.1.0. Do test it out and let me know if I can close this. Good catch on this issue!

devanshumadan commented 3 years ago

Thank you @ghdna. I tested the updated version. The bug is fixed. Closing the issue.