Open Almouro opened 9 months ago
Hi @Almouro ,
Thanks for reaching out. The CLI might be sending some max value to get all the results by default. Without inspecting the raw request and response of both the CLI and the SDK we cannot know for sure.
You can do that by running your CLI command with the --debug
flag, and similarly you can write a small middleware that will capture the same raw logs like so:
client.middlewareStack.add(next => async (args) => {
console.log(args.request)
const result = await next(args);
console.log(result);
return next(args);
}, {step: 'finalizeRequest'})
By cross referencing the two you could see which values are actually being sent and the response.
Also, if you think the results are being paginated, you can refer to this blogpost for more info about how to paginate through responses.
Please let me know if you need any further assistance. Thanks again, Ran~
Thanks @RanVaknin for your answer!
So the CLI SDK actually send 6 requests, each one with a different nextToken
. So pretty much it paginates over all the results
This is the request the JS SDK sends:
HttpRequest {
method: 'POST',
hostname: 'devicefarm.us-west-2.amazonaws.com',
port: undefined,
query: {},
headers: {
// headers
},
body: '{"arn": <PROJECT_ARN>,"type":"APPIUM_NODE_TEST_PACKAGE"}',
protocol: 'https:',
path: '/',
username: undefined,
password: undefined,
fragment: undefined
}
I'm getting 1 result
If I send it with a nextToken
, I'm getting the 2 next results
So all in all this is unexpected behavior on server side for me.
If I have say 10 uploads of type APPIUM_NODE_TEST_PACKAGE
, I expect my request to return the 10 results because it is not too large a payload to not be returned in a single response.
Here it seems that the server queries the first "page" of uploads in the db, then filters the results by type. Expected behavior for me would be that the server queries directly by type in the db
Hi @Almouro ,
I did a small repro on my own and Im able to see more than one result returned at a time:
import { DeviceFarmClient, ListUploadsCommand } from "@aws-sdk/client-device-farm";
const client = new DeviceFarmClient({ region: "us-west-2" });
const projectArn = "arn:aws:devicefarm:us-west-2:REDACTED:project:REDACTED";
const uploadType = UploadType.APPIUM_JAVA_JUNIT_TEST_SPEC;
try {
const response = await client.send(new ListUploadsCommand({
arn: projectArn,
type: uploadType
}))
console.log(response)
} catch (error) {
console.log(error)
}
Response:
{
'$metadata': {
httpStatusCode: 200,
requestId: 'd03299b1-1b4d-4c2c-82cf-2b898df2a562',
extendedRequestId: undefined,
cfId: undefined,
attempts: 1,
totalRetryDelay: 0
},
uploads: [
{
arn: 'arn:aws:devicefarm:us-west-2::upload:100e31e8-12ac-11e9-ab14-d663b5a4a906',
category: 'CURATED',
created: 2023-01-10T15:37:02.386Z,
name: 'Default TestSpec for Android Appium Java Junit v4.0 (adds support for the latest versions of Chromedriver)',
status: 'SUCCEEDED',
type: 'APPIUM_JAVA_JUNIT_TEST_SPEC',
url: 'https://prod-us-west-2-uploads-testspec.s3-us-west-2.amazonaws.com/public-yaml-files/appium_1222_node14191_junit_android_chromedriver.yml?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20231130T233257Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Credential=AKIAUJHLTYS5AWNTRO6L%2F20231130%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=c6a038d5bf74238d82f4ef8ad1a2b5606f2df2ee878d605b4b5005e7312ead49'
},
{
arn: 'arn:aws:devicefarm:us-west-2::upload:100e31e8-12ac-11e9-ab14-d663b5a4a916',
category: 'CURATED',
created: 2023-11-08T19:00:00.000Z,
metadata: '{"valid":true,"androidAmazonLinux2HostFlag":true}',
name: 'Default Android Test Spec for Appium Java JUnit (updated 2023-11-08)',
status: 'SUCCEEDED',
type: 'APPIUM_JAVA_JUNIT_TEST_SPEC',
url: 'https://prod-us-west-2-uploads-testspec.s3-us-west-2.amazonaws.com/public-yaml-files/al2_appium_java_junit_v1.yml?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20231130T233257Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Credential=AKIAUJHLTYS5AWNTRO6L%2F20231130%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=bca7232d4103720ce1b11a8bb83388d59971b74481c809db1a21154cfa0aba63'
},
{
arn: 'arn:aws:devicefarm:us-west-2::upload:100e31e8-12ac-11e9-ab14-d663b5a4a926',
category: 'CURATED',
created: 2023-11-17T19:00:00.000Z,
metadata: '{"valid":true}',
name: 'Default iOS Test Spec for Appium Java JUnit (updated 2023-11-17)',
status: 'SUCCEEDED',
type: 'APPIUM_JAVA_JUNIT_TEST_SPEC',
url: 'https://prod-us-west-2-uploads-testspec.s3-us-west-2.amazonaws.com/public-yaml-files/appium_2_ios_java_junit_v1.yml?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20231130T233257Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Credential=AKIAUJHLTYS5AWNTRO6L%2F20231130%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=826fd284671cacf0501af482817acaaece0db6e766451c28d210cfe4c40a3560'
},
{
arn: 'arn:aws:devicefarm:us-west-2::upload:100e31e8-12ac-11e9-ab14-d663bd873c81',
category: 'CURATED',
created: 2022-10-31T16:42:02.386Z,
name: 'TestSpec v7.0 for iOS Appium Java JUnit (adds support for Appium 1.22.2)',
status: 'SUCCEEDED',
type: 'APPIUM_JAVA_JUNIT_TEST_SPEC',
url: 'https://prod-us-west-2-uploads-testspec.s3-us-west-2.amazonaws.com/public-yaml-files/appium_1222_node14193_java_junit_ios.yml?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20231130T233257Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Credential=AKIAUJHLTYS5AWNTRO6L%2F20231130%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=7a89ee5f7c253031a2f417f1ee06339c320c84a94973e241299f824785d71e41'
},
{
arn: 'arn:aws:devicefarm:us-west-2::upload:101e31e8-12ac-11e9-ab14-d663bd873d81',
category: 'CURATED',
created: 2022-04-12T12:36:17.474Z,
name: 'Default TestSpec for Android Appium Java Junit v3.0 (adds support for Appium 1.22.2)',
status: 'SUCCEEDED',
type: 'APPIUM_JAVA_JUNIT_TEST_SPEC',
url: 'https://prod-us-west-2-uploads-testspec.s3-us-west-2.amazonaws.com/public-yaml-files/appium_1222_node14191_junit_android.yml?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20231130T233257Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Credential=AKIAUJHLTYS5AWNTRO6L%2F20231130%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=2776a586c14b46b82a2f44773834efec8c67365965004687b42bd29814715009'
},
{
arn: 'arn:aws:devicefarm:us-west-2::upload:1502522c-e4d2-4387-9c8e-95aedf626981',
category: 'CURATED',
created: 2020-12-02T15:37:02.386Z,
name: 'TestSpec v6.0 for iOS Appium Java JUnit (adds support for Appium 1.19.0)',
status: 'SUCCEEDED',
type: 'APPIUM_JAVA_JUNIT_TEST_SPEC',
url: 'https://prod-us-west-2-uploads-testspec.s3-us-west-2.amazonaws.com/public-yaml-files/appium_119_junit_ios.yml?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20231130T233257Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Credential=AKIAUJHLTYS5AWNTRO6L%2F20231130%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=51d81351ef537a8945115701513769d8e1ba2f28787779eb798da5423e06ac7b'
},
{
arn: 'arn:aws:devicefarm:us-west-2::upload:245e31e8-12ac-11e9-ab14-d663bd873d08',
category: 'CURATED',
created: 2020-09-03T12:36:17.474Z,
name: 'TestSpec v5.0 for iOS Appium Java JUnit (adds support for Appium 1.18.1)',
status: 'SUCCEEDED',
type: 'APPIUM_JAVA_JUNIT_TEST_SPEC',
url: 'https://prod-us-west-2-uploads-testspec.s3-us-west-2.amazonaws.com/public-yaml-files/appium_118_junit_ios.yml?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20231130T233257Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86399&X-Amz-Credential=AKIAUJHLTYS5AWNTRO6L%2F20231130%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=2573ec7c9fda4b3ec13c4b49e696fd5f891a445ace6a858bc407922650476661'
},
{
arn: 'arn:aws:devicefarm:us-west-2::upload:245e31e8-12ac-11e9-ab14-d663bd873d69',
category: 'CURATED',
created: 2020-07-31T12:36:17.474Z,
name: 'TestSpec for iOS Appium Java JUnit v4.0 (adds support for Appium 1.17.1)',
status: 'SUCCEEDED',
type: 'APPIUM_JAVA_JUNIT_TEST_SPEC',
url: 'https://prod-us-west-2-uploads-testspec.s3-us-west-2.amazonaws.com/public-yaml-files/appium_117_junit_ios.yml?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20231130T233257Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Credential=AKIAUJHLTYS5AWNTRO6L%2F20231130%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=7ccfdf034123ae3cac7f3a59448fb57528bb244afc2ebe73e844ea00ae8bc3bf'
},
{
arn: 'arn:aws:devicefarm:us-west-2::upload:295e31e8-12ac-11e9-ab14-d663bd873d54',
category: 'CURATED',
created: 2020-04-01T12:36:17.474Z,
name: 'Default TestSpec for Android Appium Java Junit v2.0 (adds support for Appium 1.14+)',
status: 'SUCCEEDED',
type: 'APPIUM_JAVA_JUNIT_TEST_SPEC',
url: 'https://prod-us-west-2-uploads-testspec.s3-us-west-2.amazonaws.com/public-yaml-files/appium_114_115_116_junit_android.yml?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20231130T233257Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Credential=AKIAUJHLTYS5AWNTRO6L%2F20231130%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=a6ddded7f9ccc09df766707c045039dca6a4312c57d644a9ffa83b39dc96e887'
},
{
arn: 'arn:aws:devicefarm:us-west-2::upload:295e31e8-12ac-11e9-ab14-d663bd873d55',
category: 'CURATED',
created: 2020-04-01T12:36:17.474Z,
name: 'Default TestSpec for iOS Appium Java Junit v3.0 (adds support for Appium 1.14+)',
status: 'SUCCEEDED',
type: 'APPIUM_JAVA_JUNIT_TEST_SPEC',
url: 'https://prod-us-west-2-uploads-testspec.s3-us-west-2.amazonaws.com/public-yaml-files/appium_114_115_116_junit_ios.yml?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20231130T233257Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Credential=AKIAUJHLTYS5AWNTRO6L%2F20231130%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=53e524e80b7f60fb71a88a9766806e51883c043419510a00ed1c80b02d1c3444'
},
{
arn: 'arn:aws:devicefarm:us-west-2::upload:4f8bd0a4-7be5-11e8-adc0-fa7ae01bbebc',
category: 'CURATED',
created: 2018-06-20T12:36:17.474Z,
name: 'Default TestSpec for Android Appium Java Junit',
status: 'SUCCEEDED',
type: 'APPIUM_JAVA_JUNIT_TEST_SPEC',
url: 'https://prod-us-west-2-uploads-testspec.s3-us-west-2.amazonaws.com/public-yaml-files/appium_java_junit_android.yml?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20231130T233257Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Credential=AKIAUJHLTYS5AWNTRO6L%2F20231130%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=ced4091567d5a5bd69feec8028a8bd302468c23bebfb8244e438c269fa370560'
},
{
arn: 'arn:aws:devicefarm:us-west-2::upload:8a6c4758-9689-11e8-9eb6-529269fb1458',
category: 'CURATED',
created: 2018-11-20T12:36:17.474Z,
name: 'Default TestSpec for iOS Appium 1.9.1 Junit (Support for iOS 12)',
status: 'SUCCEEDED',
type: 'APPIUM_JAVA_JUNIT_TEST_SPEC',
url: 'https://prod-us-west-2-uploads-testspec.s3-us-west-2.amazonaws.com/public-yaml-files/appium_191_java_junit_ios.yml?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20231130T233257Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Credential=AKIAUJHLTYS5AWNTRO6L%2F20231130%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=7115298266c1e0160e96e216604d8bdbfe94ffe4c0a997b699776318da47ee7a'
},
{
arn: 'arn:aws:devicefarm:us-west-2::upload:8a6c4758-9689-11e8-9eb6-529269fb1459',
category: 'CURATED',
created: 2018-06-20T12:36:17.474Z,
name: 'Default TestSpec for iOS Appium Java Junit',
status: 'SUCCEEDED',
type: 'APPIUM_JAVA_JUNIT_TEST_SPEC',
url: 'https://prod-us-west-2-uploads-testspec.s3-us-west-2.amazonaws.com/public-yaml-files/appium_java_junit_ios.yml?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20231130T233257Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86399&X-Amz-Credential=AKIAUJHLTYS5AWNTRO6L%2F20231130%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=3eed5e7b70b05896cb7e6f74d313680392afe848a29a6d12e9a8b28c3dfb8d61'
}
]
}
Can you share your full code snippet so we can look a bit further into it?
Thanks, Ran~
Hi @RanVaknin, I've tried to reproduce it with a test project but can't seem to reproduce it and I don't really get how pagination is implemented:
Here's what I tried:
This works as expected ✅
This is the snippet:
const { project } = await client.send(
new CreateProjectCommand({
name: "test",
})
);
if (!project) throw new Error("Project not created");
await client.send(
new CreateUploadCommand({
name: `NODE Upload`,
projectArn: project.arn,
type: UploadType.APPIUM_NODE_TEST_PACKAGE,
})
);
for (let i = 0; i < MANY_UPLOAD_COUNT; i++) {
console.log(`Creating upload ${i}`);
await client.send(
new CreateUploadCommand({
name: `OTHER UPLOAD`,
projectArn: project.arn,
type: UploadType.APPIUM_PYTHON_TEST_PACKAGE,
})
);
}
await client.send(
new CreateUploadCommand({
name: `NODE Upload`,
projectArn: project.arn,
type: UploadType.APPIUM_NODE_TEST_PACKAGE,
})
);
const result = await client.send(
new ListUploadsCommand({
arn: project.arn,
type: UploadType.APPIUM_NODE_TEST_PACKAGE,
})
);
// Should have 2 results
console.log(result);
However, what's interesting is that no pagination actually occurs. If I request all the uploads with:
const fullResult = await client.send(
new ListUploadsCommand({
arn: project.arn,
})
);
Then I retrieve ALL the results (more than 1000) which seems like way too much, and I get no nextToken
On the actual project I was facing the bug, calling:
const fullResult = await client.send(
new ListUploadsCommand({
arn: project.arn,
})
);
returns only 484 elements, even though I have more (the request returns a nextToken
)
If I iterate through all the results, I cannot find any consistency in the paging. For each page, I log:
page size
do {
const result = await client.send(
new ListUploadsCommand({
arn: PROJECT_ARN,
nextToken,
})
);
nextToken = result.nextToken;
console.log(result.uploads?.[0]);
console.log(result.uploads?.[result.uploads?.length - 1].created);
console.log(result.uploads?.length);
console.log("------------------------");
} while (nextToken);
Here are my results:
2023-01-10T15:37:02.386Z
2023-11-27T11:04:18.339Z
484
------------------------
2023-11-27T09:58:16.326Z
2023-11-13T15:12:44.297Z
628
------------------------
2023-11-13T11:29:27.533Z
2023-11-23T11:04:32.259Z
731
------------------------
2023-07-24T13:00:51.894Z
2023-11-14T12:02:09.702Z
739
------------------------
2023-11-14T19:05:34.528Z
2023-11-13T11:28:04.584Z
764
------------------------
2023-10-17T08:16:39.267Z
2023-10-30T01:08:57.624Z
381
------------------------
So how is the pagination implemented?
Hi @Almouro ,
Thanks for the investigation. It seems like pagination was not properly implemented server side.
It seems like the device-farm service in this case only relies on nextToken, but returns inconsistent results, ie;
In properly supported paginatable operations, there is typically an parameter like maxResults
or in this case maxUploads
that will limit the results per page.
We can look at how Smithy defines what a paginator should look like:
The
paginated
trait is a structure that contains the following members:Property | Type | Description -- | -- | -- inputToken | string | The name of the operation input member that contains a continuation token. When this value is provided as input, the service returns results from where the previous response left off. This input member MUST NOT be marked as required and SHOULD target a string shape. It can, but SHOULD NOT target a map shape.When contained within a service, a paginated operation MUST either configure inputToken on the operation itself or inherit it from the service that contains the operation. outputToken | string | The path to the operation output member that contains an optional continuation token. When this value is present and not empty in operation output, it indicates that there are more results to retrieve. To get the next page of results, the client passes the received output continuation token to the input continuation token of the next request. This output member MUST NOT be marked as required and SHOULD target a string shape. It can, but SHOULD NOT target a map shape.When contained within a service, a paginated operation MUST either configure outputToken on the operation itself or inherit it from the service that contains the operation. items | string | The path to an output member of the operation that contains the data that is being paginated across many responses. The named output member, if specified, MUST target a list or map. pageSize | string | The name of an operation input member that limits the maximum number of results to include in the operation output. This input member SHOULD NOT be required and SHOULD target an integer shape. It can, but SHOULD NOT target a byte, short, or long shape.WarningDo not attempt to fill response pages to meet the value provided for the pageSize member of a paginated operation. Attempting to match a target number of elements results in an unbounded API with an unpredictable latency.In the case of device farm
items
isuploads
, but thepageSize
argument is missing basically making predictable pagination impossible.With regards to the different ordering based on dates, this is not something that is guaranteed to happen. Sorting is usually not required in pagination so I can't fault the service in this case.
So how is the pagination implemented?
Unfortunately I'm not sure, as we on the SDK team do not have insight to service side implementations. I'll have to reach out to the device-farm team and ask them to redesign their pagination mechanism since its not functional as it is.
Your latest messages suggests something different than the initial bug report. At first you mentioned you are expecting to see 17 total uploads, but you saw 0. However in your latest message you suggested that although the results are not predictable, you are able to work with them (sometimes getting all results, and sometimes getting some with
nextToken
).From this, it seems to me like you are unblocked for the specific scenario you opened the issue for?
Again, I appreciate you experimenting and clarifying everything. The more details you can provide, the better I'll be able to present this to the device-farm team to make this actionable.
Thank you very much, Ran~
Hi @RanVaknin,
Indeed, I'm not blocked in the sense that I can bypass this issue with other less ideal means. Any chance you got an update from the server team about this?
Hi @Almouro,
Sorry for the long wait. This fell off of my radar. For the service team to make this actionable can you please share any request IDs that present both type of behavior?
I tried reproducing this behavior again, and I'm not seeing any inconsistencies. When I paginate through the results, I consistently get the same number of results.
As far as the pagination goes here, since we last talked I've seen more and more services that actually use dynamic pagination. I.e; non-structured data, where as long as there is nextToken, it qualifies as a paginatable operation.
Regarding the statement about the un-ordered records that you get back, the API doesn't guarantee the records would be in any specific order.
Can you please check again if this is an issue?
Thanks, Ran~
Checkboxes for prior research
Describe the bug
Using
ListUploadsCommand
with a type doesn't seem to properly paginate on server sideFor instance:
currently returns 1 result for me while in CLI:
returns 17
At some point,
ListUploadsCommand
was actually returning 0 results for me, even though the CLI had 17Wild guess but it feels like:
SDK version number
@aws-sdk/client-device-farm@3.387.0
Which JavaScript Runtime is this issue in?
Node.js
Details of the browser/Node.js/ReactNative version
v18.18.0
Reproduction Steps
aws devicefarm list-uploads --arn PROJECT_ARN --type APPIUM_NODE_TEST_PACKAGE
Observed Behavior
Results are returned with
aws devicefarm list-uploads --arn PROJECT_ARN --type APPIUM_NODE_TEST_PACKAGE
But only zero or one result is returned with
client.send(new ListUploadsCommand({ arn: PROJECT_ARN, type: UploadType.APPIUM_NODE_TEST_PACKAGE }));
Expected Behavior
client.send(new ListUploadsCommand({ arn: PROJECT_ARN, type: UploadType.APPIUM_NODE_TEST_PACKAGE }));
returns properly paginated resultsPossible Solution
Wondering if filtering is happening server side or not
Additional Information/Context
No response