Open guymguym opened 2 years ago
I don't know how to do this on the top of my head, but there is for sure a way, we will find it. It makes a lot of sense to use the full S3 model here.
Thanks @crisidev
Perhaps codegen-server-test can include all aws services from aws/sdk/aws-models/
?
I know it can be done with a separate subproject that does not depend on smithy-aws-protocol-tests.
This is how I run it -
Thanks for the code pointer! Regarding having all the SDK models in codegen-server-test, we first need to implement all the protocols and quirks that these models are bringing in or the build will never succeed..
I'd build full AWS SDK models for the server in a separate CI target like the client does, to keep codegen-server-test
quick for development / smoke tests CI target.
We should put interesting failures in a separate extras suite, see https://github.com/awslabs/smithy-rs/issues/1164. @guymguym If you have a minimal protocol test that reproduces the failure we can put it in that suite once we hook it up, that would be great.
I think it's fair to keep the codegen-server-test
clean and with the target of development / smoke tests and have a separate CI target for the rest.
I took a look on how to use the full s3 model in here and I honestly cannot find another way apart from the suggestion from @david-perez to create a new codegen folder and a new CI target.
The issue is that, as you noticed, the conflicting model is coming from the protocol tests, which we need inside server-codegen-test
project.
So I will be working first on #1159 and fix the codegen. After that we can discuss more about how to have the full S3 model and potentially others built during CI
I noticed that when building
:codegen-server-test:test
the output for s3 is very partial, which makes it easy to miss issues such as #1159. So I tried to add a symlink to the full model intocodegen-server-test/model/
, similar to the one for ebs.json. But then when I build it I get errors that we are loading conflicting s3 api definitions, first from the protocol tests jar, and then from the file. Is there a way to bypass that to use the full S3 when building codegen-server-test?The difference in model size...
This is the model from protocol tests jar - ``` smithy-rs $ cloc codegen-server-test/build/smithyprojections/codegen-server-test/s3/rust-server-codegen/src/ 13 text files. 13 unique files. 0 files ignored. github.com/AlDanial/cloc v 1.92 T=0.02 s (830.3 files/s, 163751.2 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Rust 13 47 111 2406 ------------------------------------------------------------------------------- SUM: 13 47 111 2406 ------------------------------------------------------------------------------- ``` And this is from aws/sdk/aws-modes/s3.json - ``` $ cloc s3d/build/crates/s3d-smithy-codegen-server-s3/src/ 14 text files. 14 unique files. 0 files ignored. github.com/AlDanial/cloc v 1.92 T=0.19 s (73.5 files/s, 453209.5 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Rust 14 1532 9994 74829 ------------------------------------------------------------------------------- SUM: 14 1532 9994 74829 ------------------------------------------------------------------------------- ```The full build output...
``` $ ./gradlew :codegen-server-test:test > Configure project :codegen-server-test (scanned and found a Smithy CLI version 1.16.1. You will need to add an explicit dependency on smithy-model if publishing a JAR) > Configure project :codegen-test (scanned and found a Smithy CLI version 1.16.1. You will need to add an explicit dependency on smithy-model if publishing a JAR) > Configure project :aws:sdk null Discovered protocol tests for batch.json Discovered protocol tests for ebs.json Discovered protocol tests for glacier.json Discovered protocol tests for route53.json Discovered protocol tests for s3.json Discovered protocol tests for sqs.json (scanned and found a Smithy CLI version 1.16.1. You will need to add an explicit dependency on smithy-model if publishing a JAR) > Configure project :aws:sdk-codegen-test (scanned and found a Smithy CLI version 1.17.0. You will need to add an explicit dependency on smithy-model if publishing a JAR) Building Smithy model sources: [/Users/gu/code/smithy-rs/codegen-server-test/model] Loading Smithy configs: [/Users/gu/code/smithy-rs/codegen-server-test/smithy-build.json] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#ListObjectsV2 defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [266, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [7763, 43] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#GetBucketLocation defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [290, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [4113, 47] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#CommonPrefix defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [296, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [1125, 42] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#GetBucketLocationOutput defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [307, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [4131, 53] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#ListObjectsV2Request defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [311, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [7872, 50] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#ListObjectsV2Output defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [344, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [7790, 49] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#NoSuchBucket defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [373, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [8461, 42] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#Object defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [375, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [8603, 36] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#Owner defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [389, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [9054, 35] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#CommonPrefixList defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [395, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [1139, 46] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#ObjectList defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [399, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [8725, 40] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#AccountId defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [403, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [183, 39] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#BucketName defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [405, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [912, 40] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#Delimiter defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [407, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [3244, 39] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#DisplayName defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [409, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [3301, 41] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#EncodingType defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [417, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [3313, 42] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#ETag defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [419, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [3304, 34] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#FetchOwner defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [421, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [3614, 40] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#ID defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [423, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [6163, 32] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#IsTruncated defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [425, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [6680, 41] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#KeyCount defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [427, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [6729, 38] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#LastModified defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [429, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [6778, 42] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#MaxKeys defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [431, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [8196, 37] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#NextToken defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [433, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [8452, 39] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#ObjectKey defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [438, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [8717, 39] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#ObjectStorageClass defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [474, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [8901, 48] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#Prefix defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [476, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [9236, 36] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#RequestPayer defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [484, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [11552, 42] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#StartAfter defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [488, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [12283, 40] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#Token defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [490, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [12579, 35] Ignoring duplicate but equivalent shape definition: com.amazonaws.s3#BucketLocationConstraint defined at jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [495, 1] and /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [775, 54] ERROR: com.amazonaws.s3#AmazonS3 (Model) @ /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json | 216 | "com.amazonaws.s3#AmazonS3": { | ^ = Conflicting shape definition for `com.amazonaws.s3#AmazonS3` found at `jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [40, 1]` and `/Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [216, 38]` ERROR: com.amazonaws.s3#GetBucketLocationRequest (Model) @ /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json | 4145 | "com.amazonaws.s3#GetBucketLocationRequest": { | ^ = Conflicting shape definition for `com.amazonaws.s3#GetBucketLocationRequest` found at `jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [300, 1]` and `/Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [4145, 54]` ERROR: com.amazonaws.s3#Size (Model) @ /Users/gu/code/smithy-rs/codegen-server-test/model/s3.json | 12224 | "com.amazonaws.s3#Size": { | ^ = Conflicting shape definition for `com.amazonaws.s3#Size` found at `jar:file:/Users/gu/.gradle/caches/modules-2/files-2.1/software.amazon.smithy/smithy-aws-protocol-tests/1.16.1/ce943d9bdd913a0d8788a11a8a28f491d954b3ab/smithy-aws-protocol-tests-1.16.1.jar!/META-INF/smithy/restXml/services/s3.smithy [486, 1]` and `/Users/gu/code/smithy-rs/codegen-server-test/model/s3.json [12224, 34]` Validation result: 3 ERROR(s), 0 DANGER(s), 0 WARNING(s), 0 NOTE(s) Validated 3941 shapes in model The model is invalid: 3 ERROR(s), 0 DANGER(s) > Task :codegen-server-test:smithyBuildJar FAILED Running smithy build The model is invalid: 3 ERROR(s), 0 DANGER(s) FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':codegen-server-test:smithyBuildJar'. > The model is invalid: 3 ERROR(s), 0 DANGER(s) * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.6.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 4s 13 actionable tasks: 2 executed, 11 up-to-date ```