Closed YOU54F closed 1 year ago
I guess this will require changes in pact-ruby & pact-provider-verifier
Actually, @YOU54F, a similar use case to yours was what originated me asking about being able to modify them dynamically (I was also working with Lambda behind an API Gateway with AWSv4 signing). I (wrongly) assumed that the state handlers they implemented had access to the request object, but it appears not. It means you can test the happy path with what they've implemented, but not the negative scenario!
I can share with you the workaround I had at the time, which would still work for you now! I wrote a blog post about it here: https://medium.com/dazn-tech/pact-contract-testing-dealing-with-authentication-on-the-provider-51fd46fdaa78 . Hopefully that helps!
But TLDR: instead of pulling the pacts directly from the broker, we point it to a local file instead. in a before hook to the pact verification, we pull the pact ourselves from the pact broker into a file, and then use a function to parse through it and add the request headers based on the request information. Since you'll have access to the provider states there too, you should be able to add custom logic based on that too (i.e. if "Is authenticated" -> add the header; otherwise, don't)
I've got this working nicely inside the requestFilter with access to the path
for get requests, but I can't get access to the body
of the pact under test for POST requests.
I'm not too fussed about the unhappy paths at the moment.
here is my req filter that works for GET requests
requestFilter: (req, res, next) => {
const requestUrl = PACT_PROVIDER_URL;
const host = new url.URL(requestUrl).host;
const options = {
host,
path: '/Test' + req.path
headers: {}
};
aws4.sign(options, {
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
sessionToken: process.env.AWS_SESSION_TOKEN
});
authHeaders = options.headers;
req.headers["Host"] = authHeaders["Host"];
req.headers["X-Amz-Security-Token"] = authHeaders["X-Amz-Security-Token"];
req.headers["X-Amz-Date"] = authHeaders["X-Amz-Date"];
req.headers["Authorization"] = authHeaders["Authorization"];
next();
},
For POST requests, I need to something like
requestFilter: (req, res, next) => {
const requestUrl = PACT_PROVIDER_URL;
const host = new url.URL(requestUrl).host;
const options = {
host,
path: '/Test' + req.path,
body: JSON.stringify(req.body),
headers: {}
};
aws4.sign(options, {
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
sessionToken: process.env.AWS_SESSION_TOKEN
});
authHeaders = options.headers;
req.headers["Host"] = authHeaders["Host"];
req.headers["X-Amz-Security-Token"] = authHeaders["X-Amz-Security-Token"];
req.headers["X-Amz-Date"] = authHeaders["X-Amz-Date"];
req.headers["Authorization"] = authHeaders["Authorization"];
next();
},
but req.body
is empty.
Reading up on Stack Overflow, as of Express v4.16, (https://stackoverflow.com/questions/11625519/how-to-access-the-request-body-when-posting-using-node-js-and-express#11631664)
we can use app.use(express.json())
which when placed above this line - https://github.com/pact-foundation/pact-js/blob/9d118a82ae2232448093589a98124491760a7f96/src/dsl/verifier.ts#L148
gives me access to the request body, in the requestFilter, with req.body
, I can create the AWS header, with the body contents, but the verifier times out. This might be due to the changes, or due to my providers endpoint, which is painfully slow at times. (times out after 30 seconds, and tends to take a horrendous ~25 seconds to return)
Will do some more digging tomorrow
Well the data we have encoded in the pact is garbage (our term matchers generate the string "string" which throws a validation error if directly sent to the provider through postman, so I would expect the verifier to throw an error, and not time out)
[2019-06-12T22:33:11.879Z] DEBUG: pact@8.1.2/54019 on YOU54FMAC: Proxing /decision
{ Error: Timeout waiting for verification process to complete (PID: 54029)
at Timeout._onTimeout (/Users/you54f/dev/compassdev/compass/compass-pact-provider/node_modules/q/q.js:1846:21)
at listOnTimeout (internal/timers.js:535:17)
at processTimers (internal/timers.js:479:7) code: 'ETIMEDOUT' }
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
➜ compass-pact-provider git:(awsVerify) ✗
which probably means there is something else amiss.
Time for a beer anyhoo =D
@YOU54F any updates please? since i'm still having the same issues for req.body
as undefined
Not all requests are JSON bodies (could be any type) so that parser hasn't been added. I'll double check, but I'm hopeful it looks at the media type and only attempts to parse a body into req.body if that's true, in which case I'll add that in as it's clearly going to be useful.
In the mean time, you can just read in the body yourself as you would any other node http request. Consult the node docs on how to do that.
See https://stackoverflow.com/questions/62662605/how-to-modify-a-graphql-variable-in-a-contract-during-the-provider-verification for an example. Albeit it seems it won't let you modify the body anyway
Summary of current issue.
req.body
isn't automatically parsed. I think at the least, we should parse the body. Looking at the problem before, it may be an issue with the way the http proxy wires in to the process - right now, changing the body makes it hang. I suspect this is due to an event not being fired/closed on the event loop.
In any case, there are two issues:
req.body
is not prepopulated for JSON bodiesfor now what i have done as a workaround is using an http proxy similar to https://github.com/http-party/node-http-proxy/blob/master/examples/middleware/bodyDecoder-middleware.js where i update the body based on some criteria, of course this is not the ideal solution but for now it's the best option i could think of
Hi there,
You have an awesome product. I really enjoy working with PACT.
Do you have an ETA on this issue? The workaround from @combmag is something we can try as well, but we would rather avoid it because it adds a lot of complexity and hides the implementation from the tests.
Hi Zsolt, thanks for the kind words!
Our v3 branch (the beta release you can find) has support for this. We hope to have it out of beta in the next couple of months, it is only lacking a few features (e.g. message support and Pact Web) and may have a few rough edges API wise.
Hi @mefellows,
Thanks for the quick response. I'll give it a try.
Cheers
for now what i have done as a workaround is using an http proxy similar to https://github.com/http-party/node-http-proxy/blob/master/examples/middleware/bodyDecoder-middleware.js where i update the body based on some criteria, of course this is not the ideal solution but for now it's the best option i could think of
@combmag could you share a short snippet of how you set up the filter and the proxy? I'm having trouble getting it to work properly. I would appreciate it greatly.
Hi Zsolt, thanks for the kind words!
Our v3 branch (the beta release you can find) has support for this. We hope to have it out of beta in the next couple of months, it is only lacking a few features (e.g. message support and Pact Web) and may have a few rough edges API wise.
Hi @mefellows,
Great product you have here and it's the center point of many of our test suites here.
Eagerly waiting for this feature as a big chunk of work is pending due to this :)
Any updated timelines would be appreciated
I think, it would be better if we can access req.body
in stateHandlers
rather than requestFilter
as this will give more control per transaction
Summary of current issue.
req.body
isn't automatically parsed. I think at the least, we should parse the body. Looking at the problem before, it may be an issue with the way the http proxy wires in to the process - right now, changing the body makes it hang. I suspect this is due to an event not being fired/closed on the event loop.In any case, there are two issues:
1. `req.body` is not prepopulated for JSON bodies 2. Even if you parse the body (see https://stackoverflow.com/questions/62662605/how-to-modify-a-graphql-variable-in-a-contract-during-the-provider-verification) you can't modify it.
Hi @mefellows, really appreciate the work you've been doing this is an awesome tool! that being said, any updates on the issue described here? we really need this functionality :(
Thanks @dineshk-qa and @lonely-caat!
So we're currently focussed on our v3 upgrade (which has a new underlying Rust engine, replacing the current Ruby shared core) and several new features, whilst we look to extend Pact via plugins. You could give that a go, the headers and body can be mutated in this implementation: https://github.com/pact-foundation/pact-js#pact-js-v3.
If anyone was interested in revisiting the current code base, a PR would be welcome.
I think the main issue is in the actual http proxy we use currently, so that might need to be replaced (which could be a lot of work to prevent backwards incompatible behaviour).
@mefellows Which version should i install in order to try that? I'm stuck because of the same reason of this issue. I'm at version "@pact-foundation/pact": "^9.15.3",
I just tried the version v10.0.0-beta33 and still the same.
import * as path from 'path';
import { Verifier, VerifierOptions } from '@pact-foundation/pact';
const opts: VerifierOptions = {
providerBaseUrl: process.env.PROVIDER_BASE_URL!,
pactUrls: [path.resolve('packages/shared/pact-testing/contracts/loging-patient-rest-users.json')],
logLevel: 'debug',
requestFilter(req, _, next) {
if (req.url === '/users/auth/login') {
console.log('lol');
req.body = {
email: 'fakeuser@email.com',
password: 'password1234',
} as RestUsers.LoginReqDto;
}
next();
}
};
const verifier = new Verifier(opts);
describe('Login provider verifier', () => {
test('verifier', async () => {
await verifier.verifyProvider();
});
});
The body is not modified
Apologies, I copied the wrong link @alexsotocx - it should be https://github.com/pact-foundation/pact-js#pact-js-v3. Please also note the new package import
Apologies, I copied the wrong link @alexsotocx - it should be https://github.com/pact-foundation/pact-js#pact-js-v3. Please also note the new package import
Tested and working, thanks :)
hey @mefellows, are there any news regarding the ability to modify the request body payload?
Have you read the thread @lonely-caat ?
@mefellows do you mean this message https://github.com/pact-foundation/pact-js/issues/304#issuecomment-803510182 from a month ago where you said that you have other priorities, and to check this out in alpha?
Exactly, our focus is about the next major release. It's looking to be a big piece of work addressing this in the current lib, so it makes more sense on spending the effort on the next major version.
that's cool, thank you. when is the next major release planned for?
We’re hoping to get it out as soon as we can. It’s hard to put a time frame on it, because this is an open source project. Work largely happens in the spare time that maintainers and contributors have available.
Personally, I’m hoping that initiatives like Pactflow (which I’m not involved with, but am pleased to see happen) and the occasional organisation dedicating (or contracting) someone to develop features they need mean that we’ll be able to knock a lot of these longer term goals off sooner than we have in the past. It’s an exciting time for the project- we’re currently seeing the most activity from contributors and community members since I joined.
I can tell you that getting the next major release out is certainly the highest priority for pact-js at the moment.
Sent from my mobile
On 1 May 2021, at 5:17 pm, lonely-caat @.***> wrote:
that's cool, thank you. when is the next major release planned for?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.
FYI I have a spike branch that allows users to modify the request body in the request filters: https://github.com/pact-foundation/pact-js/tree/feat/request-filter-bodies.
I haven't touched it for a few weeks, but should be portable to both the current mainline and also the v3.x.x branch.
Nice one @mefellows I am going to finish the monster that I created with this issue xD have popped this on my list
We’re hoping to get it out as soon as we can. It’s hard to put a time frame on it, because this is an open source project. Work largely happens in the spare time that maintainers and contributors have available.
This! I balanced in between how long would it take me to make this tool do x, versus how long would it take me to roll something out myself.
If anyone reading this, who wants to see this and v3 and other features happen, get involved, we can't do this alone and it's obviously in all of our net interests to partake, so don't hesitate to reach out here or via slack.
Have tested your nice little fix @mefellows in both v2 and v3, it all works as expected. @YOU54F not sure what I can do to help get this moving forward, does it just need a pr raising?
Oh, if you can clean it up and raise a PR I would love that. It's a bit spikey, but if you're keen to tidy / clean it up and add some basic tests we'll get that in and shipped ASAP.
Also, thanks for confirming!
Hi @DaveClissold,
Amazing thanks for helping shift this along my good man!
You can reach out on https://slack.pact.io if you want to chat in more real-time. We could probably create a bit of a task list if that helps 👍🏾
We have a specific pact-js and pact-js-development channel with a fair few members.
Closing this issue down now as complete!,
Conclusion
If anyone wishes to discuss, feel free to reach out to us on here or via our Pact Slack
Just in case anyone lands on this issue
Created another e2e example here
https://github.com/you54f/aws-auth-pact
with the verifier code here
Thanks saf! We should definitely add this to https://docs.pact.io/recipes. I'll take a look Monday :)
@YOU54F
I have looked at your example
const authHeaders = options.headers;
// console.log(authHeaders)
req.headers['Host'] =
authHeaders && authHeaders['Host']
? authHeaders['Host'].toString()
: '';
req.headers['X-Amz-Date'] =
authHeaders && authHeaders['X-Amz-Date']
? authHeaders['X-Amz-Date'].toString()
: '';
req.headers['Authorization'] =
authHeaders && authHeaders['Authorization']
? authHeaders['Authorization'].toString()
: '';
// The following is required if using AWS STS to assume a role
req.headers['X-Amz-Security-Token'] =
authHeaders && authHeaders['X-Amz-Security-Token']
? authHeaders['X-Amz-Security-Token'].toString()
: '';
As it was it was not working for me, I had to add
req.headers['Authorization'] =
authHeaders && authHeaders['Authorization']
? authHeaders['Authorization'].toString()
: '';
But then I have realised that it looks better if instead of setting the headers this way I do
Object.assign(req.headers, signed.headers);
The entire piece looks like this
import * as express from "express";
import { Request, sign } from "aws4";
import { AssumeRoleCommand, STSClient } from "@aws-sdk/client-sts";
import { fromEnv } from "@aws-sdk/credential-provider-env";
import { AwsCredentialIdentity } from "@smithy/types";
import { URL } from "url";
const getRoleCredentials = async (
roleArn: string
): Promise<AwsCredentialIdentity> => {
const sts = new STSClient({});
const { Credentials: credentials } = await sts.send(
new AssumeRoleCommand({
RoleArn: roleArn,
RoleSessionName: "pact-verifier",
})
);
if (
!credentials ||
!credentials.AccessKeyId ||
!credentials.SecretAccessKey
) {
throw new Error("Credentials are not defined");
}
return {
accessKeyId: credentials.AccessKeyId,
secretAccessKey: credentials.SecretAccessKey,
sessionToken: credentials.SessionToken,
};
};
const getCredentials = async (
roleArn?: string
): Promise<AwsCredentialIdentity> =>
roleArn ? getRoleCredentials(roleArn) : fromEnv()();
export const aws4SignAuth = async (
apiEndpoint: string,
requestOptions: Omit<Request, "host" | "path" | "headers">,
roleArn?: string
): Promise<express.RequestHandler> => {
const parsedUrl = new URL(apiEndpoint);
const credentials = await getCredentials(roleArn);
return (req, res, next) => {
const options: Request = {
host: parsedUrl.host,
path: parsedUrl.pathname + req.path,
headers: {},
...requestOptions,
...(req.method === "POST"
? {
body: String(req.body),
headers: { "Content-Type": "application/json" },
}
: {}),
};
const signed = sign(options, credentials);
Object.assign(req.headers, signed.headers);
next();
};
};
// usage
const aws4SignMiddleware = await aws4SignAuth(apiUrl, {
region: "eu-west-1",
service: "appsync",
});
About the usage, if like me you use Appsync Custom Domains
, it does not get the service
correctly (issue with the AWSv4 dependency --> returns appsync-api when using the default one) nor the region.
New update
export const aws4SignAuth = async (
apiEndpoint: string,
requestOptions: Omit<Request, "host" | "path" | "headers">,
roleArn?: string
): Promise<express.RequestHandler> => {
const parsedUrl = new URL(apiEndpoint);
const credentials = await getCredentials(roleArn);
return (req, res, next) => {
const body =
req.body instanceof Object
? JSON.stringify(req.body)
: String(req.body);
const options: Request = {
host: parsedUrl.host,
path: parsedUrl.pathname + req.path,
headers: {},
...requestOptions,
...(req.method === "POST"
? {
body: Buffer.from(body),
headers: {
...req.headers,
host: parsedUrl.host,
},
}
: {}),
};
const signed = sign(options, credentials);
Object.assign(req.headers, signed.headers);
next();
};
};
What am I trying to do?
Trying to verify pacts against an AWS Lambda function, with an API gateway, that requires AWSv4 signed request headers
Problems I need to overcome
Basically we need to pre-sign some headers based on information on our request
We can acheive this with aws4 in a
stateHandler
forisAuthenticated
, however I cannot get access to the pact under test'spath
orrequestBody
, which I need to pass to the pre-signed URLWhat I've tried so far
Using the following Proposal: Allow customProviderHeaders to be dynamically added to different interactions , I have been able to get the verifier successfully working locally & in CI with a hardcoded request path, rather than the request path from the pact under test.
Examples in CI
CircleCI AWS Verification Step AWS-Provider Pact AWS-Provider Pact Verification Results
Steps taken in code
Generate temporary AWS credentials with a bash script and export to bash & run verify script
npx ts-node src/pact/verifier/verify.ts | grep -v Created
let signedHost: string; let signedXAmzSecurityToken: string; let signedXAmzDate: string; let signedAuthorization: string; let authHeaders: any;
const opts: VerifierOptions = { stateHandlers: { "Is authenticated": async () => { const requestUrl = process.env.PACT_PROVIDER_URL; const host = new url.URL(requestUrl).host; const apiroute = new url.URL(requestUrl).pathname; const pathname =
${apiroute}/helloworld
; const options = { host, path: pathname, headers: {} }; await aws4.sign(options); aws4.sign(options, { secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY, accessKeyId: process.env.AWS_ACCESS_KEY_ID, sessionToken: process.env.AWS_SESSION_TOKEN }); authHeaders = options.headers; signedHost = authHeaders.Host; signedXAmzSecurityToken = authHeaders["X-Amz-Security-Token"]; signedXAmzDate = authHeaders["X-Amz-Date"]; signedAuthorization = authHeaders.Authorization; return Promise.resolve(AWS signed headers created
); }, "Is not authenticated": async () => { signedHost = null; signedXAmzSecurityToken = null; signedXAmzDate = null; signedAuthorization = null; return Promise.resolve(Blank aws headers created
); } },requestFilter: (req, res, next) => { // over-riding request headers with AWS credentials if (signedHost != null) { req.headers.Host = signedHost; } if (signedXAmzSecurityToken != null) { req.headers["X-Amz-Security-Token"] = signedXAmzSecurityToken; } if (signedXAmzDate != null) { req.headers["X-Amz-Date"] = signedXAmzDate; } if (signedAuthorization != null) { req.headers.Authorization = signedAuthorization; } next(); },
req.path /_pactSetup req.body { consumer: 'consumer-service', state: 'Is authenticated', states: [ 'Is authenticated' ], params: {} } creating AWS signed headers created AWS signed headers req.path /path/that/the/pact/test/is/calling req.body undefined
{ consumer: 'consumer-service', state: 'Is authenticated', states: [ 'Is authenticated' ], params: {} }
creating AWS signed headers created AWS signed headers
req.path /helloworld req.body undefined
req.path /helloworld req.body {message:"hello world")