Building on ARM CPUs causes error on AWS

[unix]

Describe the bug

The example works fine locally, but when deployed to the AWS platform, the endpoint is not accessible. (I also tried using Pulumi or manually creating a lambda in AWS and adding a gateway, etc., and it worked fine.)

When I call the endpoint directly or try to test under AWS' Lambda test service, I see the following error in the log:

START RequestId: b478530d-bdfe-46e8-8006-70473b2f87ff Version: $LATEST
2022/05/19 05:54:58 Registered Gateway Plugin
2022/05/19 05:54:58 Starting Child Process
2022/05/19 05:54:58 Services listening on: 127.0.0.1:50051
2022/05/19 05:54:58 Membrane Error: there was an error starting the child process: fork/exec /usr/local/bin/ts-node: exec format error, exiting
2022/05/19 05:54:58 gateway 'Stop' called, waiting for lambda runtime to finish
END RequestId: b478530d-bdfe-46e8-8006-70473b2f87ff
REPORT RequestId: b478530d-bdfe-46e8-8006-70473b2f87ff  Duration: 15017.11 ms   Billed Duration: 15000 ms   Memory Size: 128 MB Max Memory Used: 6 MB   
2022-05-19T05:55:13.417Z b478530d-bdfe-46e8-8006-70473b2f87ff Task timed out after 15.02 seconds

Expected behavior

Consistent results with local runs after deployment.

To Reproduce

Steps to reproduce the behavior:

1. Run nitric new to create project.
2. Run nitric run to check function on local. It works well here.
3. Run nitric up -s <my-stack> to deploy. No errors were generated and all resources were successfully deployed. In Pulumi's monitoring panel, I can see that every service is working properly.
4. Trigger <endpoint>/hello/world, get Internal Server Error.

I've tried removing all resources and redeploying with the command and got the same result.

Output of nitric info -o yaml:

os: darwin
arch: arm64
goversion: go1.17.9
cliversion: was not built with version info
fabricversion: v0.15.1-rc.8
containerruntime: docker
containerruntimeversion: |
  platform:
    name: Docker Desktop 4.8.1 (78998)
  components:
  - name: Engine
    version: 20.10.14
    details:
      ApiVersion: "1.41"
      Arch: arm64
      BuildTime: "2022-03-24T01:45:44.000000000+00:00"
      Experimental: "false"
      GitCommit: 87a90dc
      GoVersion: go1.16.15
      KernelVersion: 5.10.104-linuxkit
      MinAPIVersion: "1.12"
      Os: linux
  - name: containerd
    version: 1.5.11
    details:
      GitCommit: 3df54a852345ae127d1fa3092b95168e4a88e2f8
  - name: runc
    version: 1.0.3
    details:
      GitCommit: v1.0.3-0-gf46b6ba
  - name: docker-init
    version: 0.19.0
    details:
      GitCommit: de40ad0
  version: 20.10.14
  apiversion: "1.41"
  minapiversion: "1.12"
  gitcommit: 87a90dc
  goversion: go1.16.15
  os: linux
  arch: arm64
  kernelversion: 5.10.104-linuxkit
  experimental: false
  buildtime: "2022-03-24T01:45:44.000000000+00:00"

Brief description of your project

Basic sample, create from nitric new.

Additional context

[unix]

BTW, Nitric is indeed an amazing service, our team spent a lot of time deploying and debugging multiple lambda's, especially on testing AWS with Pulumi, Nitric's out of the box and 0 configuration was a great fit for us.

I guess our team may be early adopters, feel free to contact me for any subsequent updates or user surveys, and it would be even better if there are features or corporate cooperation (or sponsorship), I am always ready to listen to your voice!

[asalkeld]

Hi @unix

I tried to reproduce this on our develop branch and it seemed to work. Here are my steps.

nitric new
? What is the name of the project? basic-sample
? Choose a template: official/TypeScript - Starter
? Glob for the function handlers? functions/*.ts
angus@fedora:~/.../github.com/nitrictech/> cd basic-sample/
angus@fedora:~/.../nitrictech/basic-sample/> ls
functions  nitric.yaml  package.json  README.md  tsconfig.json
angus@fedora:~/.../nitrictech/basic-sample/> cat nitric.yaml 
name: basic-sample
handlers:
- functions/*.ts
angus@fedora:~/.../nitrictech/basic-sample/> ls functions/
hello.ts
angus@fedora:~/.../nitrictech/basic-sample/> cat functions/hello.ts 
import { api } from "@nitric/sdk";

const helloApi = api('main');

helloApi.get("/hello/:name", async (ctx) => {
    const { name } = ctx.req.params;

    ctx.res.body = `Hello ${name}`;

    return ctx;
});angus@fedora:~/.../nitrictech/basic-sample/> nitric stack new
? What do you want to call your new stack? test
? Which Cloud do you wish to deploy to? aws
? select the region us-east-1
angus@fedora:~/.../nitrictech/basic-sample/> yarn install
yarn install v1.22.10
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
warning Your current version of Yarn is out of date. The latest version is "1.22.19", while you're on "1.22.10".
Done in 10.20s.

I then did a run

nitric run
 SUCCESS  Configuration gathered (1s)                                                                                                                                                                                                         
 SUCCESS  Created Dev Image! (0s)                                                                                                                                                                                                             
 SUCCESS  Started Local Services! (17s)                                                                                                                                                                                                       
 SUCCESS  Started Functions! (0s)                                                                                                                                                                                                             
Local running, use ctrl-C to stop

In a different window

curl http://localhost:9001/apis/main/hello/angus
Hello angus

I then did the aws deploy

angus@fedora:~/.../nitrictech/basic-sample/> nitric up -s test
 SUCCESS  Configuration gathered (1s)                                                                                                                                                                                                         
 SUCCESS  Images built (46s)                                                                                                                                                                                                                  
 Deployed  Repository/basic-sample-hello (3s)                                                                                                                                                                                                 
 Deployed  Group/basic-sample-test (5s)                                                                                                                                                                                                       
 Deployed  Role/helloLambdaRole (5s)                                                                                                                                                                                                          
 Deployed  RolePolicy/helloListAccess (2s)                                                                                                                                                                                                    
 Deployed  RolePolicyAttachment/helloLambdaBasicExecution (2s)                                                                                                                                                                                
 Deployed  Image/hello-image (3m18s)                                                                                                                                                                                                          
 Deployed  Image/hello (3m18s)                                                                                                                                                                                                                
 Deployed  Function/hello (30s)                                                                                                                                                                                                               
 Deployed  AWSLambda/hello (3m48s)                                                                                                                                                                                                            
 Deployed  Api/main (9s)                                                                                                                                                                                                                      
 Deployed  Stack/basic-sample-basic-sample-test (4m0s)                                                                                                                                                                                        
 Deployed  Permission/mainhello (2s)                                                                                                                                                                                                          
 Deployed  Stage/mainDefaultStage (5s)                                                                                                                                                                                                        
 Deployed  Stack (4m9s)                                                                                                                                                                                                                       
┌───────────────────────────────────────────────────────────────┐
| API  | Endpoint                                               |
| main | https://2mjon7cqo6.execute-api.us-east-1.amazonaws.com |
└───────────────────────────────────────────────────────────────┘

and the curl to the above url

angus@fedora:~/.../nitrictech/pro-backend (main %)/> curl https://2mjon7cqo6.execute-api.us-east-1.amazonaws.com/hello/unix
Hello unix

It might be that we just need to release what we have in develop. It looks like it's only about 2 weeks old.

[asalkeld]

I have tested v1.3.0 and that also works for me. Note: I am using fedora (linux) and you are using a mac. From a quick google it might be something like this: https://github.com/lovell/sharp/issues/1377#issuecomment-420623747 Basically the node_modules have the wrong OS binaries (buildtime instead of runtime OS).

[jyecusch]

I just tried the same steps on my MacBook without issue using published assets only. That said, I'm using an intel MacBook - I'll see if I can get access to an M1 to test, that's not a platform I've tested on before so it could be where the difference is.

[jyecusch]

I was able to replicate the issue using an M1 MacBook, we'll do some more testing to find a fix.

[unix]

Yes, it does seem to be an OS issue with the device, I just replaced an intel MacBook and followed the same process to deploy lambda to AWS and everything works fine. And thank you for debugging and replies.

[jyecusch]

For anyone else who comes across this issue. We've narrowed it down to an issue with the way we build the container images using docker. Unless we use BuildKit we're unable to build for multiple platforms, it just builds for the host platform. This means ARM-based machines don't currently build amd64 images with nitric up, causing compatibility issues with AWS Lambda.

We'll support this by moving to BuildKit for nitric builds, but it's a bit of work to make the change so may take time. In the meantime, local runs should be fine and we recommend running deployments through a CI server running an Intel or AMD CPU.

[tjholm]

This has been merged into our develop branch of the CLI and will soon be available for release.