Closed robin-aws closed 1 day ago
Smoke tests aren't intended to test client serialization (or deserialization), they're intended to make connections to live services to be a sanity check that the service and client are interacting properly.
Additionally, clients shouldn't be doing validation of anything but types because client-side validation is forwards-incompatible. For an example of this, look to the difficulty around EC2's increased instance id size several years ago. A number of clients had been making assertions that instance ids were a particular size (which was modeled iirc), and they all broke when EC2 started returning ids that were longer. EC2 had to spend years warning people and migrating.
With regards to binary data, this is more of a limitation in Smithy itself, since there's no way in the IDL or the AST to indicate binary data. Unfortunately, changing the way nodes are parsed to expect binary data to be base64 or similarly encoded would be a breaking change.
@smokeTests
is a great testing trait that should apply to my use case nicely as well (https://github.com/awslabs/smithy-dafny), but I had planned to use it to check that validation is happening correctly, and it turns out I can't do that because the trait validation asserts thatparams
satisfies the traits like@required
and constraint traits like@range
on the operation's input shape, and the raised validation events are at theERROR
severity level so I can't suppress them.I'd propose that the validation be loosened to only assert that the
params
node types are compatible with the input shape of the operation.Two more minor points I'd like to address if possible:
failure: {errorId: ValidationError}
, but validation errors are not modeled so there's no such shape id to reference. Could we have an additionalvalidationFailure: ...
member for theExpectation
union?params
also states "Parameter values that contain binary data MUST be defined using values that can be represented in plain text as the plain text representation". That seems to have carried over from the HTTP protocol tests spec, but it feels less necessary here since we're not inspecting the Base64-encoded payloads, and in my case I'd love to be able to pass arbitrary binary data.I'm willing to cut a PR if we agree on a solution. :)