aws-amplify / amplify-cli

The AWS Amplify CLI is a toolchain for simplifying serverless web and mobile development.
Apache License 2.0
2.81k stars 821 forks source link

Known issue with Lambda Auth to Graphql API #13235

Closed qwikag closed 7 months ago

qwikag commented 1 year ago

How did you install the Amplify CLI?

Who Knows, because the process is all over the shop.

If applicable, what version of Node.js are you using?

20.3.1

Amplify CLI Version

12.4.0

What operating system are you using?

WIN 10

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

No manual changes

Describe the bug

This document is not helpful in any way. And this is a BUG! https://docs.amplify.aws/guides/functions/graphql-from-lambda/q/platform/js/

Because presently the process and commands to deliver on it do not work. So in rewriting this document and testing the process, the bugs should get removed in the process.

Could we please get it rewritten to accurately describe how to set up a lambda's access to GraphQL API or via node-fetch or direct dynamo or something? So that we can actually get our apps running!!!

The document needs to provide accurate steps, and actually work.

Also update the page with a date so we know when it was last updated/reviewed.

Expected behavior

the documented process takes the user through setting up and provides a few various common situations like Cognito triggers etc.

Reproduction steps

follow the current page: https://docs.amplify.aws/guides/functions/graphql-from-lambda/q/platform/js/

Project Identifier

1aff16e72d6657e6251bd6a212fd28ac

Log output

No response

Additional information

Points of interest and questions: What Amplify Version? What libraries are available (node-fetch only), can API.GraphQL be used? Can it be setup with IAM/API-key/Cognito, and how?, and why? AWS SDK v3 - WHAT? Proper debugging??? Policies to give access - WHAT?

Before submitting, please confirm:

josefaidt commented 12 months ago

Hey @qwikag :wave: thanks for raising this, and thanks for bringing this to our attention! We'll get those samples updated now that Lambda supports Node 18, removing the node-fetch dependency for newly-created Functions.

Please note there is also an active issue we are tracking where Function access is not applied to your GraphQL API. This happens when the API is already deployed and you grant your Function access to the API to query/mutate, and the corresponding resolvers are not updated to include your Function's information for authorization, thus giving you a 401. To workaround this you can add an empty space to the GraphQL Schema to trigger an update and push with amplify push

Could we please get it rewritten to accurately describe how to set up a lambda's access to GraphQL API or via node-fetch or direct dynamo or something?

As a quick callout, it is not recommended to modify records on DynamoDB directly and rather query/mutate through GraphQL. This will ensure any DataStore-connected clients are notified of subscribed updates, and ensure the tables DataStore uses are updated.

What Amplify Version?

Functions created with Amplify CLI >12.2.x are created using the Node 18 runtime, and therefore can be built without the use of node-fetch, however every recent version of Amplify CLI is able to create Functions with sufficient, scoped permissions for GraphQL operations (barring the issue and workaround noted above)

What libraries are available (node-fetch only), can API.GraphQL be used?

Unfortunately the aws-amplify library does not currently support Node environments, however we will evaluate the feasibility of using it during the update of this doc you've linked.

Can it be setup with IAM/API-key/Cognito, and how?, and why?

IAM is our recommendation, however this can also be used with API Key with little updates needed for the code as outlined in the linked doc. For Cognito you will need to configure a REST API to use Cognito to authorize requests, which you can then send along to the GraphQL API request.

AWS SDK v3 - WHAT?

AWS SDK v3 is now included in Lambda's Node 18 runtime, superseding AWS SDK v2 that was available in previous runtime versions.

Proper debugging???

There is currently an active issue for debugging Lambda Functions with amplify mock and using VSCode's debugging tools.

Policies to give access - WHAT?

IAM Policies for access are generated for you automatically when stepping through the amplify add|update function flow and selecting "Grant Resource Access Permissions". No further action should be required from you with this flow.

qwikag commented 12 months ago

Hi @josefaidt,

Fantastic response, Thank you.

I just want to clarify on this point:

This happens when the API is already deployed and you grant your Function access to the API to query/mutate

But the docs do say to do it that way:

The function can only be added when the GraphQL API with IAM authorization exists.

So can you please advise the best order of setting up a new function.

  1. setup new table, with IAM auth, then
  2. do a push --force api, then
  3. setup function with access to API

or

  1. setup function with access to API, then
  2. setup new table, with IAM auth, then
  3. do a push --force api, then

As a side I think this is the problem with most of the guides/turorials, there is very little step by step process. very little "why do we do this" and very little scenario based options (e.g. IAM(SSO), and App Router vs Pages.

The menu structure of the docs is not easy to find stuff. I think there a 2. ways of doing things. 1 via console and 2 via CLI, maybe others like configuration etc. and the console version should show how to do it but also advise that using the CLI guides is best practice. That way people will see the right outcome. e.g. what a VTLs and how do I know when they are setup correct.

Anyway, I hope to oneday be proficient enough to be more helpful/hands on.

josefaidt commented 12 months ago

Hey @qwikag

I just want to clarify on this point:

This happens when the API is already deployed and you grant your Function access to the API to query/mutate But the docs do say to do it that way: The function can only be added when the GraphQL API with IAM authorization exists.

This is a great callout! And I see how the callout in the docs shown after the example causes confusion. In the docs this is a callout specifically for the "AppSync - GraphQL API request (with IAM)" Function template you are prompted for on amplify add function. For this we can move it above the code snippet to be more contextual

image

So can you please advise the best order of setting up a new function.

The second option here is fine. For the active issue I linked the only thing you'll want to ensure is that your API is updated after adding/updating a Function to have access to that API. You can inspect the auth VTL resolvers to verify whether the Function name has been added as an authorized source.

As a side I think this is the problem with most of the guides/turorials, there is very little step by step process. very little "why do we do this" and very little scenario based options (e.g. IAM(SSO), and App Router vs Pages.

This and the following comments are also great callouts, and highlight a gap in our suggested workflows when considering both Amplify Hosting and the CLI. Let's continue this conversation in a docs issue. I see you have one regarding Next.js app router vs pages, but would you mind filing a new issue there with these thoughts?

jerocosio commented 11 months ago

@josefaidt This workaround doesn't work. I just created a new function and added the permissions through the CLI, added a space in my graphQL file and made the deployment with push and I'm still getting this error. The Amplify team should prioritize this type of bugs which have been reported since at least 1 year and 4 months ago, this basically makes all the documentation around 'using GraphQL on lambda' useless as if you follow it up it just doesn't work.

I would also asume by seeing the number of devs with issues on this matter that this affects a great deal of users, so why not prioritize this bugs which affect basic functionality for any type of app instead of adding shiny new things like passkeys? And I actually have a long list of things that the team could prioritize and that the devs have been asking for years like:

These are all things that could improve the developer experience greatly, and things that probably affect or could improve +80% of all projects done in Amplify.

qwikag commented 11 months ago

@jerocosio

This workaround doesn't work. I just created a new function and added the permissions through the CLI, added a space in my graphQL file and made the deployment with push and I'm still getting this error.

>amplify push --force API

have you tried that?

Adding permission: what exactly did you do? > amplify update api

Did you give IAM access to GraphQL API?

jerocosio commented 11 months ago

@qwikag I've tried almost everything found on Github/Stackoverflow by now, this bug has been messing with me now for 2 weeks, I decided to just query the database directly as there's no issues with getting permissions directly from the db, but once more I tried doing it from scratch and again got the same result, here are my detailed step-by-step:

  1. Ran amplify function add and this were my options:
    
    ? Select which capability you want to add: Lambda function (serverless function)
    ? Provide an AWS Lambda function name: getItem
    ? Choose the runtime that you want to use: NodeJS
    ? Choose the function template that you want to use: AppSync - GraphQL API request (with IAM)

✅ Available advanced settings:

? Do you want to configure advanced settings? Yes ? Do you want to access other resources in this project from your Lambda function? Yes ? Select the categories you want this function to have access to. api ? Api has 2 resources in this project. Select the one you would like your Lambda to access xxxx ? Select the operations you want to permit on xxxx Query

You can access the following resource attributes as environment variables from your Lambda function API_XXXX_GRAPHQLAPIENDPOINTOUTPUT API_XXXX_GRAPHQLAPIIDOUTPUT API_XXXX_GRAPHQLAPIKEYOUTPUT ENV REGION ? Do you want to invoke this function on a recurring schedule? No ? Do you want to enable Lambda layers for this function? No ? Do you want to configure environment variables for this function? No ? Do you want to configure secret values this function can access? No ✔ Choose the package manager that you want to use: · NPM

2. Edited the `index.js` file of the generated `getItem` function and just copy/pasted the generated `listItems` query from GraphQL in src/graphlq, this is the file:
```javascript
/* Amplify Params - DO NOT EDIT
    API_XXXX_GRAPHQLAPIENDPOINTOUTPUT
    API_XXXX_GRAPHQLAPIIDOUTPUT
    API_XXXX_GRAPHQLAPIKEYOUTPUT
    ENV
    REGION
Amplify Params - DO NOT EDIT */

import crypto from '@aws-crypto/sha256-js';
import { defaultProvider } from '@aws-sdk/credential-provider-node';
import { SignatureV4 } from '@aws-sdk/signature-v4';
import { HttpRequest } from '@aws-sdk/protocol-http';
import { default as fetch, Request } from 'node-fetch';

const GRAPHQL_ENDPOINT = process.env.API_XXXX_GRAPHQLAPIENDPOINTOUTPUT;
const AWS_REGION = process.env.AWS_REGION || 'us-east-1';
const { Sha256 } = crypto;

export const query = /* GraphQL */ `
  query ListItems($filter: ModelItemFilterInput, $limit: Int, $nextToken: String) {
    listItems(filter: $filter, limit: $limit, nextToken: $nextToken) {
      items {
        id
        owner
        public
      }
      nextToken
    }
  }
`;

/**
 * @type {import('@types/aws-lambda').APIGatewayProxyHandler}
 */

export const handler = async (event) => {
  console.log(`EVENT: ${JSON.stringify(event)}`);

  const endpoint = new URL(GRAPHQL_ENDPOINT);

  const signer = new SignatureV4({
    credentials: defaultProvider(),
    region: AWS_REGION,
    service: 'appsync',
    sha256: Sha256
  });

  const requestToBeSigned = new HttpRequest({
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      host: endpoint.host
    },
    hostname: endpoint.host,
    body: JSON.stringify({ query }),
    path: endpoint.pathname
  });

  const signed = await signer.sign(requestToBeSigned);
  const request = new Request(endpoint, signed);

  let statusCode = 200;
  let body;
  let response;

  try {
    response = await fetch(request);
    body = await response.json();
    if (body.errors) statusCode = 400;
  } catch (error) {
    statusCode = 500;
    body = {
      errors: [
        {
          message: error.message
        }
      ]
    };
  }

  return {
    statusCode,
    //  Uncomment below to enable CORS requests
    // headers: {
    //   "Access-Control-Allow-Origin": "*",
    //   "Access-Control-Allow-Headers": "*"
    // },
    body: JSON.stringify(body)
  };
};
  1. Added one enter/space to the schema.graphql for the Item type so it looks like this:

    type Item
    @model(subscriptions: null)
    @auth(rules: [{ allow: private, provider: iam }, { allow: owner }]) {
    id: ID!
    
    owner: String @auth(rules: [{ allow: owner, operations: [read, delete] }])
    public: String @default(value: "false")
    }
  2. Made sure that both the function and the API are going to be updated so ran amplify status and it looks like so:
    ┌──────────┬────────────────────────────────────┬───────────┬───────────────────┐
    │ Category │ Resource name                      │ Operation │ Provider plugin   │
    ├──────────┼────────────────────────────────────┼───────────┼───────────────────┤
    │ Function │ getItem                            │ Create    │ awscloudformation │
    ├──────────┼────────────────────────────────────┼───────────┼───────────────────┤
    │ Api      │ apiname                       │ Update    │ awscloudformation │
    ├──────────┼────────────────────────────────────┼───────────┼───────────────────┤
    │ Api      │ AdminQueries                       │ No Change │ awscloudformation │
    ├──────────┼────────────────────────────────────┼───────────┼───────────────────┤
    │ Auth     │ authXXXX                   │ No Change │ awscloudformation │
    ├──────────┼────────────────────────────────────┼───────────┼───────────────────┤
    │ Auth     │ userPoolGroups                     │ No Change │ awscloudformation │
    ├──────────┼────────────────────────────────────┼───────────┼───────────────────┤
    │ Function │ AdminQueries935fd287               │ No Change │ awscloudformation │
    ├──────────┼────────────────────────────────────┼───────────┼───────────────────┤
    │ Function │ authXXXXPostAuthentication │ No Change │ awscloudformation │
    ├──────────┼────────────────────────────────────┼───────────┼───────────────────┤
    │ Function │ authXXXXPreSignup          │ No Change │ awscloudformation │
    ├──────────┼────────────────────────────────────┼───────────┼───────────────────┤
    │ Storage  │ XXXX                       │ No Change │ awscloudformation │
    └──────────┴────────────────────────────────────┴───────────┴───────────────────┘
  3. Ran amplify push and accepted all the changes, no issues and deployment succeeds.
  4. Go into the lambda console with my AWS account and ran a test and I got this error:
    {
    "statusCode": 400,
    "body": "{\"data\":{\"listItems\":null},\"errors\":[{\"path\":[\"listItems\"],\"data\":null,\"errorType\":\"Unauthorized\",\"errorInfo\":null,\"locations\":[{\"line\":3,\"column\":5,\"sourceName\":null}],\"message\":\"Not Authorized to access listItems on type Query\"}]}"
    }

    If I go into the details of the lambda I can even see that the role is created and it has the necessary permissions or it looks like it but doesn't work:

    Screenshot 2023-09-26 at 9 08 29 p m

Any suggestion greatly appreciated.

qwikag commented 11 months ago

@jerocosio I went down the same path as you; your experience is not your fault, the documentation needs to be fixed and the tools also need fixing and I believe AWS are on top of that. Hence me helping you out so they can keep working on the issues.

But... You kinda ignored my questions... I am happy to help, but if you don't answer the helpers questions then you are only hurting yourself.

I will reiterate: > amplify update api (see below to include IAM) update schema (add the space if need be or create more/less schema. >amplify push --force api

I also have used >amplify api gql-compile & > amplify codegen

But I think they are included in the push. (very little documentation)

Here is my setup: > amplify update api

? Select from one of the below mentioned services: GraphQL

General information

Authorization modes

  • Default: Amazon Cognito User Pool
  • IAM

Conflict detection (required for DataStore)

  • Disabled

? Select a setting to edit (Use arrow keys)

Authorization modes

qwikag commented 11 months ago

@jerocosio Seemingly we do not need to use the @aws-sdk anymore. changing that to standard GrpahQL api would be interesting. personally I am in no hurry to fux it.

Your code looks exactly like mine and mine is now working due to the a space & --force api So it is doable.

jerocosio commented 11 months ago

Thank you so much @qwikag for looking into this I really appreciate it, I continued following the steps you shared by doing:

  1. Ran amplify update api to double check my settings for the authorization:
    
    ? Select from one of the below mentioned services: GraphQL

General information

Authorization modes

Conflict detection (required for DataStore)

? Select a setting to edit Authorization modes ? Choose the default authorization type for the API Amazon Cognito User Pool Use a Cognito user pool configured as a part of this project. ? Configure additional auth types? Yes ? Choose the additional authorization types you want to configure for the API API key, IAM API key configuration ✔ Enter a description for the API key: · xxxApiKey ✔ After how many days from now the API key should expire (1-365): · 365

2. Added one more line on the graphql file
3. Ran `amplify api gql-compile`
4. Ran `amplify codegen`
5. Updated the function to add a new console.log to debug more
6. Ran `amplify push --force api`
7. Ran a test on the lambda aws console

But still got the same error:
``` javascript
{
  "statusCode": 400,
  "body": "{\"data\":{\"listItems\":null},\"errors\":[{\"path\":[\"listItems\"],\"data\":null,\"errorType\":\"Unauthorized\",\"errorInfo\":null,\"locations\":[{\"line\":3,\"column\":5,\"sourceName\":null}],\"message\":\"Not Authorized to access listItems on type Query\"}]}"
}

I also just found something odd on my resolvers, on all of them in the auth I have this:

#if( $util.authType() == "IAM Authorization" )
  #set( $adminRoles = ["xxxxPostConfirmation-dev"] )
  #foreach( $adminRole in $adminRoles )
    #if( $ctx.identity.userArn.contains($adminRole) && $ctx.identity.userArn != $ctx.stash.authRole && $ctx.identity.userArn != $ctx.stash.unauthRole )
      #return($util.toJson({}))
    #end
  #end
  #if( ($ctx.identity.userArn == $ctx.stash.authRole) || ($ctx.identity.cognitoIdentityPoolId == "us-west-1:xxxxx" && $ctx.identity.cognitoIdentityAuthType == "authenticated") )
    #set( $isAuthorized = true )
  #end
#end

But I actually deleted that function 'xxxxPostConfirmation' a long time ago, but I don't see the actual function that I'm trying to give access to anywhere in the resolvers.

qwikag commented 11 months ago

@jerocosio Sounds like you have done all you can do.

I am on amplify-cli version 12.5.0

Lets discuss alternative platforms shall we.

I have just raised 2 very serious bugs occurring in the schema: https://github.com/aws-amplify/amplify-studio/issues/1044 https://github.com/aws-amplify/amplify-studio/issues/1045

there is no way to get proper attention here, or quick support. I have paid for support and as much as the people I have been dealing with are quality I think AWS have dug themselves into a hole. each time I have been the resolver of the bug (by luck or skill), and they have just played the role of pushing me to investigate more things.

So again what is a better platform outside AWS?

jerocosio commented 11 months ago

@qwikag I really like Amplify ad have used it in a number of projects, but it's true that support is basically non-existent and you have to find 'hacks' around making it work the way you want definately supabase or firebase are great alternatives to it, but I haven't really used them for a project lately.

I believe the issue with Amplify is the way they prioritize stuff, focusing on creating flashy stuff like their Figma integration, the Amplify Studio, etc... which are cool to have, but not really useful for more than 5% of their users vs things such as simple improvements to their transforms, functions, etc... (as mentioned above) which no one really sees but would improve 99% of their users.

josefaidt commented 11 months ago

Hey @jerocosio :wave: the error in the issue you're experiencing seems related to a missing @auth rule on the Items model. If the Function does not have access to call the API you should see a generic 401 or 403 response, where the 400 you're seeing is admittedly a bit confusing since it is also an auth-related error but from within the API. Can you share a snippet of your schema?

For example I've created a sample project using the following steps:

  1. create a new project with amplify init -y
  2. add a GraphQL API with amplify add api > GraphQL > add both API Key and IAM for auth
  3. push with amplify push -y (to simulate having an existing API in the project when we add our function)
    type Todo
      @model
      @auth(rules: [{ allow: public }, { allow: private, provider: iam }]) {
      id: ID!
      name: String!
      description: String
    }
  4. add a function with amplify add function
    1. Choose the AppSync - GraphQL API request (with IAM) template
    2. for the advanced settings prompt select Y
    3. grant access to the API with "Grant resource access permissions"
      ? Which setting do you want to update? Resource access permissions
      ? Select the categories you want this function to have access to. api
      ? Select the operations you want to permit on 13235 Query, Mutation
  5. add a space to the GraphQL schema to trigger an update (and regenerate the resolvers)
  6. push with amplify push -y
  7. navigate to the Lambda console to run the function and observe a successful call

image

A few notes:

josefaidt commented 11 months ago

@qwikag thank you for helping out here! Would you be open to hopping on a quick call to gather additional feedback? If so would you mind sending me a message at amplify-cli@amazon.com?

josefaidt commented 11 months ago

@jerocosio apologies, just noticed the schema snippet a few comments above,

type Item
  @model(subscriptions: null)
  @auth(rules: [{ allow: private, provider: iam }, { allow: owner }]) {
  id: ID!

  owner: String @auth(rules: [{ allow: owner, operations: [read, delete] }])
  public: String @default(value: "false")
}

which looks good. The auth resolver you noted does look a bit odd. Is this an overridden resolver by chance? If you run amplify api gql-compile and inspect the *.auth.* resolver generated in amplify/backend/api/<api-name>/build/resolvers/Query.listItems.auth.1.req.vtl does it still show the deleted function?

qwikag commented 11 months ago

Hi @josefaidt, sending you email now.

jerocosio commented 11 months ago

@josefaidt thank you for looking into this, I just ran amplify api gql-compile once more on my project and went through my resolvers and yeah the deleted function is still there, it's like my resolvers are not re-generating at all, and no it's not an overwritten one they're the ones generated from amplify. Is there anyway to force the re-generation of the resolvers? I can see the deleted function also on the resolvers inside the build directory.

jerocosio commented 11 months ago

@josefaidt I looked a little deeper into the files and they haven't been updated since at least a couple of weeks ago, I tracked down those dates and it looks like they match to when I created a new env for my project, not sure if it's 100% correlated, but looks like so.

josefaidt commented 11 months ago

@jerocosio can you try deleting the build directory and pushing?

jerocosio commented 11 months ago

@josefaidt I finally got it working, for some reason there was a resolver directory on the root of the api directory which had all the resolvers that I had in the past, so on the new build they were being picked as if they were made to override the new resolvers that were created. Thanks for digging into this and for the help, I hope that the overal bug around having to make edits on the schema every time a user wants to add/remove the permissions gets solved soon.

josefaidt commented 11 months ago

Hey @jerocosio glad to hear it!! That sounds like a side effect of an older bug where some mock issue would leave the resolvers/ directory in-tact with the generated resolvers https://github.com/aws-amplify/amplify-category-api/issues/1211

AakashKB commented 8 months ago

Any auth change I make is not reflected until I delete the resolvers folder and re mock. Huge hinderance to dev/debugging workflow.

josefaidt commented 7 months ago

Closing in favor of tracking existing item

github-actions[bot] commented 7 months ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.