503 Error on api call with Apollo Server "The Lambda function associated with the CloudFront distribution is invalid or doesn't have the required permissions."

[grantmontgomery]

I'm deploying an application using the @sls-next/serverless-component plugin. I built out an api route with an Apollo Server, specifically "apollo-server-micro" because this is what's suggested by other Next.js users. All my other api routes work, but I can't access this one in production mode which you can see on my actual site at https://sekndapp.com/api/createNewUser. This is what my createNewUser.ts file looks like.

import { ApolloServer, gql } from "apollo-server-micro";

export const config = {
  api: {
    bodyParser: false,
  },
};

const apolloServer: ApolloServer = new ApolloServer({ typeDefs, resolvers });

const handler = apolloServer.createHandler({ path: `/api/createNewUser` });

export default handler;

And my serverless.yml file.

myNextApplication:
  component: "@sls-next/serverless-component@1.17.0"
  inputs:
    domain: "sekndapp.com"
    timeout: 30
    name:
      defaultLambda: resourceAPILambda
      apiLambda: resourceAPILambda2

I'm also having issues where I can't see logs from my lambda functions in lambda. Is there a way to fix this or should I go with another way to deploy to AWS?

[dphang]

Will need more logs, please check here on how to get logs: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-edge-testing-debugging.html on how to get logs for Lambda@Edge - they are in the region closest to the viewer which is not necessarily in us-east-1 (where the Lambda is created in).

[grantmontgomery]

@dphang Which Lambda should I test because there's a default function and an API lambda function created by the serverless component? I'm also not sure how to run the tests since the code on these functions are 1500+ lines long and I don't know exactly what to input in for tests. Is the default key/value test valid for these functions? I've been testing deployment with Vercel too and I'm not experiencing the issue on there https://sekndproduction.vercel.app/api/handleUser. This is the link for the serverless component to the same API route too https://d34hjxtv8xi8je.cloudfront.net/api/handleUser.

[dphang]

I mean that you need to look at Lambda@Edge logs after you are hitting /api/handleUser and see what the error is on the server side (i.e the Lambda logs). Since this is an API route, it would be using the API lambda (looks like resourceAPILambda2) above.

Go to CloudWatch for the region closest to where you're accessing from, and find the log group with a similar name. It's probably called /aws/lambda/us-east-1.resourceAPILambda2 (ignore the us-east-1, it's named like this in all regions). Then you can browse logs and search for what is happening.

Unfortunately, the 503 error page is intentionally generic and won't tell you much, so I have no idea what is wrong.

For testing, do you mean sending a test event in Lambda console? You can (and it'd provide you the logs in the test console), but you'd need to provide a CloudFront origin request test event to it by following: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-event-structure.html and generating your own event. Probably easier to just find the logs first after making a request to the CloudFront distribution.

[grantmontgomery]

This is a log from the Lambda@Edge test. The logs are from the us-east-1 region because there are no logs or events invoked from the region closest to me. The function name is called something else now because I got rid of old deployments and tried deploying again, but this is the API Lambda from the most recent deployment.

START RequestId: d1fcbff2-e6f7-4b82-833b-79dccebef4a8 Version: $LATEST
2020-11-26T20:29:56.453Z    d1fcbff2-e6f7-4b82-833b-79dccebef4a8    ERROR   Invoke Error    {"errorType":"TypeError","errorMessage":"Cannot read property '0' of undefined","stack":["TypeError: Cannot read property '0' of undefined","    at Runtime.handler$1 [as handler] (/var/task/index.js:893:34)","    at Runtime.handleOnce (/var/runtime/Runtime.js:66:25)"]}
END RequestId: d1fcbff2-e6f7-4b82-833b-79dccebef4a8
REPORT RequestId: d1fcbff2-e6f7-4b82-833b-79dccebef4a8  Duration: 4.45 ms   Billed Duration: 100 ms Memory Size: 512 MB Max Memory Used: 65 MB  Init Duration: 140.23 ms    

This is the isolated error from the actual CloudWatch. Looks like it just gives the same information as the log from the Lambda@Edge test.

2020-11-26T20:31:34.825Z 1443f0d3-64ae-48fb-909e-305c0d87356d ERROR Invoke Error

{
    "errorType": "TypeError",
    "errorMessage": "Cannot read property '0' of undefined",
    "stack": [
        "TypeError: Cannot read property '0' of undefined",
        "    at Runtime.handler$1 [as handler] (/var/task/index.js:893:34)",
        "    at Runtime.handleOnce (/var/runtime/Runtime.js:66:25)"
    ]
}

Let me know if there's any other information you need.

[dphang]

Ok, as per the error message, it says it's trying to access [0] of an undefined value, which will fail. However, checked and this is not part of serverless-nextjs code, as our API handler for version 1.17.0 is only 341 lines long.

I am not familiar with Apollo server and it's not a part of serverless-nextjs - it looks like you are generating a new handler using this line:

const handler = apolloServer.createHandler({ path: `/api/createNewUser` });

You will need to inspect your custom index.js (in .serverless_nextjs/api-lambda/index.js) at line 893:34 to figure out what line may be failing

[grantmontgomery]

Apollo Server is a GraphQl server you use to send GraphQl requests to. There are several methods of calling the server for example this would be one method in a basic Node.js or Express server.

const server = new ApolloServer({ typeDefs, resolvers });

// The `listen` method launches a web server.
server.listen().then(({ url }) => {
  console.log(`🚀  Server ready at ${url}`);
});

The method I and other Next.js users use is the "apollo-server-micro" method which has a "createHandler" function which returns a Promise with a response object. There is also a "apollo-server-lambda" method that also has the same "createHandler" function, but this doesn't work with the API routes of Next.js because it doesn't seem to return a response object. There's actually instructions on how to deploy to Serverless with "apollo-server-lambda" https://www.apollographql.com/docs/apollo-server/deployment/lambda/, but I'm not sure if the instruction would apply to implementing the same thing using the "sls" Next.js serverless component and the Next.js API routes. Assigning the handler to a variable like that was a different method I had tried, but I have it set up like this now and it creates the same error.

export default cors(
  new ApolloServer({
    typeDefs,
    resolvers,
  }).createHandler({
    path: "/api/handleUser",
  })
);

This is the handler function from those lines of code in the index.js file. Line 893 is the second line in this block where the request variable is assigned.

const handler$1 = async (event) => {
    const request = event.Records[0].cf.request;
    const routesManifest = RoutesManifestJson__default['default'];
    const buildManifest = manifest__default['default'];
    // Handle domain redirects e.g www to non-www domain
    const domainRedirect = getDomainRedirectPath(request, buildManifest);
    if (domainRedirect) {
        return createRedirectResponse(domainRedirect, request.querystring, 308);
    }
    // Handle custom redirects
    const customRedirect = getRedirectPath(request.uri, routesManifest);
    if (customRedirect) {
        return createRedirectResponse(customRedirect.redirectPath, request.querystring, customRedirect.statusCode);
    }
    // Handle custom rewrites but not for non-dynamic routes
    let isNonDynamicRoute = buildManifest.apis.nonDynamic[normaliseUri(request.uri)];
    if (!isNonDynamicRoute) {
        const customRewrite = getRewritePath(request.uri, routesManifest);
        if (customRewrite) {
            request.uri = customRewrite;
        }
    }
    const uri = normaliseUri(request.uri);
    const pagePath = router(manifest__default['default'])(uri);
    if (!pagePath) {
        return {
            status: "404"
        };
    }
    // eslint-disable-next-line
    const page = require(`./${pagePath}`);
    const { req, res, responsePromise } = nextAwsCloudfront(event.Records[0].cf, {
        enableHTTPCompression: buildManifest.enableHTTPCompression
    });
    page.default(req, res);
    const response = await responsePromise;
    // Add custom headers before returning response
    addHeadersToResponse(request.uri, response, routesManifest);
    return response;
};

exports.handler = handler$1;

[dphang]

Which line is the one failing? Is it const request = event.Records[0].cf.request; which is line 893?

There was a similar error here (https://github.com/serverless-nextjs/serverless-next.js/issues/711) but it was due to invoking the Lambda through the test console and not having the event.Records set correctly. I guess you mentioned you are invoking it through CloudFront so it would be strange if it's still failing on this line.

[grantmontgomery]

Which line is the one failing? Is it const request = event.Records[0].cf.request; which is line 893?

There was a similar error here (#711) but it was due to invoking the Lambda through the test console and not having the event.Records set correctly.

Yes sorry it's that line you mentioned.

[dphang]

I see, I am not sure why that line would be failing if you are invoking it through CloudFront, CloudFront should be sending event.Records correctly. You can try to print out JSON.stringify(event) by manually adding some log statements to your handler to see what is happening, and then deploy (with build = false to prevent rebuilding / overwriting your index.js handler).

Is Apollo server manipulating event or anything similar before it reaches the serverless-nextjs code? Right now this component's code expects event to be a CloudFront event with Records in its handler.

[grantmontgomery]

I entered in some console.log statements in that handler function for the event object and set build to false, but nothing from these was logged out on Lambda@Edge or Cloudwatch. It's just the same errors as before.

const handler$1 = async (event) => {
  console.log(JSON.stringify(event));
  console.log(event.Records);....

I even tried attaching event to the response which gets returned, but it wasn't logged either. How could I log the statements in a way that would be caught by the logs? Btw I appreciate the help and your responsiveness especially on a day like this. Happy Thanksgiving!

[dphang]

Regular console logging statements should be logged into CloudWatch, could you please check that the updated handler is being uploaded?

In fact we do have logging statements e.g here (https://github.com/serverless-nextjs/serverless-next.js/blob/2427517e093a7f938d22600771b86a6a36c40666/packages/libs/lambda-at-edge/src/default-handler.ts#L45) when you turn perf logging on.

You could also build the handler from scratch and add logging statements following this: https://github.com/serverless-nextjs/serverless-next.js/blob/master/CONTRIBUTING.md. The equivalent line to change would be here: https://github.com/serverless-nextjs/serverless-next.js/blob/2427517e093a7f938d22600771b86a6a36c40666/packages/libs/lambda-at-edge/src/api-handler.ts#L64

[grantmontgomery]

Apollo is something I'm putting to the side for now since there are issues setting cookie tokens for handling authentication in Apollo Server along with these issues for Serverless. The cookie issues are very difficult to solve and many solutions suggested by other users are not working on my end. I'll be creating separate versions of my app and be using a normal API route to handle authentication, but I'll try to build this out in the future.

[grantmontgomery]

@dphang I just want to add that I really want to contribute and help find a fix for getting Apollo Server to work with serverless-nextjs, I just don't have the time right now because I'm trying to finish up this main project and find my first job. It's also that cookie issue I mentioned with Apollo that's making it difficult to incorporate in my project. I promise I'll come to back to this later and help you all.

[brandonfriday]

@grantmontgomery @dphang Any update on this? I tried to deploy a Next.js + Apollo Server project with the serverless nextjs plugin and also got this same 503

[grantmontgomery]

I'm going to have time to help out with this starting next week. Really want to help contribute to getting this to work. @brandonfriday how did you start your Next project? Did you create it from scratch or something like Create Next App like I did?

[brandonfriday]

Hi @grantmontgomery it's started with create next app I believe. Did you want to chat about that sometime this week?

[brandonfriday]

I saw another issue about a 503 coming from the Internationalized Routing so was wondering if this could be more an issue with that.

[zerbinidamata]

I have the same problem with Apollo Cliente, I discovered that it's an issue regarding my private client config, I wasn't able to fix this yet but maybe I can give a clue to someone. Everything works locally but gives 503 on Cloudfront.

[brandoncroberts]

Yea @dphang @zerbinidamata @grantmontgomery I'm quite sure now that this isn't an issue with the serverless-next plugin, but rather something within our own implementations.

I've just deployed this apollo client/ server example from next that also has API routes and it works flawlessly, also on API Routes. https://github.com/vercel/next.js/tree/canary/examples/api-routes-apollo-server-and-client Going to look into my setup later and compare with that example