Issues with lambda receive function

[bagitte]

I've setup Fraudmarc-ce on 6/23 in AWS and i'm getting the following invocation errors for the fraudmarc-ce-receive lamda function:

START RequestId: f2f63600-7779-11e8-b538-5fcd7d03aece Version: $LATEST fork/exec /var/task/receive: no such file or directory: PathError null END RequestId: f2f63600-7779-11e8-b538-5fcd7d03aece

I ran the Fraudmarc-ce-install Docker file to automate the setup of the lambda functions in AWS. Previously, I attempted the advance installation but encountered issues with 'apex deploy' where it indicates that there's no GO files.

This appears to be the same issue mentioned above. https://stackoverflow.com/questions/49000980/deploying-aws-lambda-with-go-cloudformation

[jeffkbkim]

Hi @bagitte , I will look into it tomorrow morning and write back to you!

[jeffkbkim]

@bagitte for clarification, are you encountering the issue for both the advanced and simple installation? Or did the simple install work but the advanced install fail when you ran apex deploy?

[bagitte]

@kimkb2011 I only encountered the apex deploy issue with the advanced installation whereby there's no GO files hence the lambda functions were never created. The simple installation completed w/out incident; however, the receive lambda function has invocation errors that it can't find a file or directory. Sorry for the confusion.

[jeffkbkim]

@bagitte for the advanced install, did you run apex deploy in the project directory?

[bagitte]

@kimkb2011 yep, i ran it in the project directory where project.json is located.

[jeffkbkim]

@bagitte can you post the output of pwd && ls -l && apex deploy from your project directory?

[bagitte]

@kimkb2011 here's the output:

[ fraudmarc-ce]$ pwd && ls -l && apex deploy /home/ec2-user/go/src/github.com/fraudmarc/fraudmarc-ce total 3964 -rw-rw-r-- 1 ec2-user ec2-user 10613 Jun 23 22:57 25Fraudmarc-CE-Logo-on-Light.png drwxrwxr-x 4 ec2-user ec2-user 4096 Jun 23 22:57 backend drwxrwxr-x 2 ec2-user ec2-user 4096 Jun 23 22:57 database -rw-rw-r-- 1 ec2-user ec2-user 50408 Jun 23 22:57 diagram2.png -rw-rw-r-- 1 ec2-user ec2-user 1248 Jun 23 22:57 Dockerfile -rw-rw-r-- 1 ec2-user ec2-user 296 Jun 24 05:14 env.list -rwxrwxr-x 1 ec2-user ec2-user 211 Jun 23 22:57 env.list.old -rw-rw-r-- 1 ec2-user ec2-user 6646 Jun 23 22:57 fraudmarcce drwxrwxr-x 4 ec2-user ec2-user 4096 Jun 23 22:57 frontend drwxrwxr-x 4 ec2-user ec2-user 4096 Jun 23 22:57 functions -rw-rw-r-- 1 ec2-user ec2-user 10243 Jun 23 22:57 INSTALL-ADVANCED.md drwxrwxr-x 2 ec2-user ec2-user 4096 Jun 23 22:57 installer -rw-rw-r-- 1 ec2-user ec2-user 10853 Jun 23 22:57 INSTALL.md -rw-rw-r-- 1 ec2-user ec2-user 11357 Jun 23 22:57 LICENSE -rw-rw-r-- 1 ec2-user ec2-user 8369 Jun 23 22:57 logo.png -rw-rw-r-- 1 ec2-user ec2-user 3886844 Jun 23 22:57 newgif.gif -rw-rw-r-- 1 ec2-user ec2-user 842 Jun 23 20:13 project.json -rw-rw-r-- 1 ec2-user ec2-user 747 Jun 23 22:57 project.json.old -rw-rw-r-- 1 ec2-user ec2-user 1269 Jun 23 22:57 README.md -rw-rw-r-- 1 ec2-user ec2-user 373 Jun 23 22:57 SES-setup.json ⨯ Error: function process: build hook: ../../../../../gopkg.in/mgutz/dat.v1/init.go:7:2: no Go files in /home/ec2-user/go/src/github.com/mgutz/logxi

[jeffkbkim]

I believe the build failed because you did not install the dependencies correctly before deploying the Lambdas. Could you tell me if running this works (the patch should resolve the build issue)?

   go get github.com/fraudmarc/fraudmarc-ce/backend/lib \
          github.com/fraudmarc/fraudmarc-ce/database
   (go get -d gopkg.in/mgutz/dat.v1 ; exit 0)
   cd $GOPATH/src/gopkg.in/mgutz/dat.v1
   patch -p1 < $GOPATH/src/github.com/fraudmarc/fraudmarc-ce/database/dat.patch
   cd $GOPATH/src/github.com/fraudmarc/fraudmarc-ce
   go get ./...

[bagitte]

@kimkb2011 Here's the output:

[ec2-user@ip ~]$ go get github.com/fraudmarc/fraudmarc-ce/backend/lib \ github.com/fraudmarc/fraudmarc-ce/database patch -p1 < $GOPATH/src/github.com/fraudmarc/fraudmarc-ce/database/dat.patch cd $GOPATH/src/github.com/fraudmarc/fraudmarc-ce go get ./...go/src/gopkg.in/mgutz/dat.v1/init.go:7:2: no Go files in /home/ec2-user/go/src/github.com/mgutz/logxi [ec2-user@ip ~]$ (go get -d gopkg.in/mgutz/dat.v1 ; exit 0) go/src/gopkg.in/mgutz/dat.v1/init.go:7:2: no Go files in /home/ec2-user/go/src/github.com/mgutz/logxi [ec2-user@ip ~]$ cd $GOPATH/src/gopkg.in/mgutz/dat.v1 [ec2-user@ip dat.v1]$ patch -p1 < $GOPATH/src/github.com/fraudmarc/fraudmarc-ce/database/dat.patch [ec2-user@ip dat.v1]$ cd $GOPATH/src/github.com/fraudmarc/fraudmarc-ce [ec2-user@ip fraudmarc-ce]$ go get ./... ../../../gopkg.in/mgutz/dat.v1/init.go:7:2: no Go files in /home/ec2-user/go/src/github.com/mgutz/logxi [ec2-user@ip fraudmarc-ce]$ ^C [ec2-user@ip fraudmarc-ce]$ apex deploy ⨯ Error: function receive: build hook: ../../../../../gopkg.in/mgutz/dat.v1/init.go:7:2: no Go files in /home/ec2-user/go/src/github.com/mgutz/logxi

[jeffkbkim]

Does running apex deploy from the root directory still not work?

[bagitte]

Correct, I get the same message: ⨯ Error: function receive: build hook: ../../../../../gopkg.in/mgutz/dat.v1/init.go:7:2: no Go files in /home/ec2-user/go/src/github.com/mgutz/logxi

[bagitte]

Also, is this related to the simple installation because that went through fine and the lambda functions were created. The only issue is that error I referenced above.

[jeffkbkim]

@bagitte it seems the patch isn't being applied correctly. Maybe starting from a completely clean slate of golang will fix this issue.

Regarding the simple install, we have resolved the zip issue with lambda and I will merge it to master branch. Since you already have deployed your Lambdas, RDS, etc, don't forget to delete all of the fraudmarc-ce related AWS services before re-running the simple install as it will exit if there's an existing instance with the same name.

[bagitte]

@kimkb2011 sounds goods. Thanks.

[jeffkbkim]

@bagitte merged.

[bagitte]

@kimkb2011 will let you know in a few how that went. Thanks again.

[bagitte]

@kimkb2011 so I removed the lambda role, rds, lambda functions and re-ran.

Here's the output:

**Your Database has been launched! Check the AWS RDS Console -> Instances tab (fraudmarcce) Your IAM Role has been created! Check the AWS IAM Console (FraudmarcCE)

An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: The role defined for the function cannot be assumed by Lambda. Your Lambda functions have been created! Check the AWS Lambda Console (fraudmarc-ce-receive/process)

An error occurred (MalformedPolicyDocument) when calling the PutRolePolicy operation: Syntax errors in policy.**

The lambda role that was created looks fine. Not sure what happened there. In any event, I'm going to remove and try again.

[bagitte]

@kimkb2011 re-ran and got the same error. Yeah, it wasn't able to create the inline policy for that role. # #

[bagitte]

@kimkb2011 I replaced the inline-policy and lamdba-assume-policy json with the previous copies and this time it went through successfully:

Your Database has been launched! Check the AWS RDS Console -> Instances tab (fraudmarcce) Your IAM Role has been created! Check the AWS IAM Console (FraudmarcCE) Your Lambda functions have been created! Check the AWS Lambda Console (fraudmarc-ce-receive/process) Your inline-policy has been added to the FraudmarcCE role! Check the IAM Console Your AWS Role, RDS, Lambdas has been setup. Fraudmarc CE installation complete.

[jeffkbkim]

@bagitte Nice! The Dockerfile inside the installer directory looks for the string "ARN" and replaces it with your receive lambda ARN. Since you already have changed the string, the error occurred since it couldn't find the ARN it was looking for.

[bagitte]

@kimkb2011 ah ok. Thanks.

[bagitte]

@kimkb2011 Looks like the issue remains. Here's the output:

fork/exec /var/task/receive: no such file or directory: PathError null

[jeffkbkim]

When does this exactly occur? When you send an email to your SES recipient?

[bagitte]

Yeah, after the email(s) is in the S3 bucket and then the lambda function is triggered.

[jeffkbkim]

@bagitte That issue should have been fixed with the previous merge. I have made some other changes to the /installer/Dockerfile file (resolved InvokeFunction) to the master branch. Could you try cloning the repo and go through the steps?

Also, I've made some changes in the INSTALL.md file (Database Configuration section). Check the diff and see how you can connect the Lambdas to the DB instance.

[bagitte]

@kimkb2011 Ok, will do. Thanks.

[bagitte]

@kimkb2011 looks like the lambda receive function is working properly now. The only issue now is that it's timing out connecting to the database. I'll need to configure the function to be able to communicate with RDS. Thanks for fixing the file/directory issue.

[kcolemangt]

@bagitte Glad to know it's working for you. Next is simply a matter of configuring your environment or DB instance. Use the DMARC forum or Fraudmarc Support for help with any of that and reserve this tracker for code issues.

ps. Did you know we provide a free service tier where it only takes ~1 minute to install DMARC for your domain? There's even an automated tool to protect your domain with p=reject.

[bagitte]

@kcolemangt ok, will do. Will take a look at the free service tier. Thanks.

[jeffkbkim]

Closing issue.