search

zotero / translation-server

A Node.js-based server to run Zotero translators
Other
121 stars 50 forks source link

AWS Lambda, Internal Server Error #103

Closed tyfried closed 5 years ago

tyfried commented 5 years ago

Problem

I am trying to get a working AWS Lambda instance for translation-server. I have the instance up and running (and possibly configured correctly), but every time I make a request via the API Gateway url the translation-server fails with an "Internal Server Error" or something similar.

Examples

Request: curl -d 'https://www.nytimes.com/2018/06/11/technology/net-neutrality-repeal.html' -H 'Content-Type: text/plain' <aws-api-gateway>/web Response Body: Internal Server Error

Request: curl -d 'https://www.ncbi.nlm.nih.gov/pubmed/?term=crispr' -H 'Content-Type: text/plain' <aws-api-gateway>/web Response Body: Internal Server Error

Request: curl -d 10.2307/4486062 -H 'Content-Type: text/plain' <aws-api-gateway>/search Response Body: An error occurred during translation. Please check translation with the Zotero client.

Request: curl -d "'https://www.nytimes.com/2018/06/11/technology/net-neutrality-repeal.html'" -H 'Content-Type: text/plain' <aws-api-gateway>/web Response Body: URL not provided

Background

I am an absolute beginner with AWS and nodejs, but I was able to work through the configuration requirements:

Possible Issues/Solutions

I've spent the better part of the last 36 hours trying to figure this out to no avail. Below are some avenues I have considered.

lambda_template.yaml

Maybe there is an issue with my lambda_template.yaml file? Mine is identical to lambda_template.yaml.j2 with the following substitutions:

I saw that up until a few months ago, there was a lambda_deploy script that automated some of the operations I mentioned above, including creating the required template file for the aws package operation, so I just tried to follow what it did. I have attached a zip file with my lambda_template.yaml and the serverless-output.yaml that it helped generate.

POST, application/x-www-form-urlencoded

Calling curl -d creates a request with the header Content-type: application/x-www-form-urlencoded, and according to here and here the Amazon API Gateway does not play nicely with that type of request header. I can't tell if this applies only to the Method Test feature in the API Gateway console, or any request in general (e.g. from curl). This led me down a rabbit hole of trying make a POST request similar to the examples above without the application/x-www-form-urlencoded header which ended here and without success. That tutorial requires you to make a mapping template which doesn't seem to be possible if the resource you're modifying is a proxy resource, as is the case for translation-server (I think).

production.json

I specified my node_env as production in lambda_template.yaml, but there is no production.json file in the package directory created by ./lambda_package. Do I need to create a production.json file? I'm confused because I think this is something the AWS Lambda infrastructure is suppposed to take care of, but I'm not sure. This is where my inexperience with node is hurting me :)

Returning Status Code

For solving an "Internal Server Error" issue, this post and this post both talk about making sure that the JSON response is formatted correctly. I don't think this is the issue because the translation-server only returns the "Internal Server Error" part of the time, but I haven't been able to rule it out because I'm not sure how to test this.

Conclusion

Any help would be greatly appreciated! If I can't figure Lambda out then I can always run this out of a digital ocean droplet (or something similar), but the former seems cooler and much cheaper.

dstillman commented 5 years ago

Did you clone with submodules? It's possible you're just missing some files, including the translators.

If not that, check your CloudWatch logs for the Lambda function to see what's being logged when the requests fail.

Your requests are all correct, except this one:

Request: curl -d "'https://www.nytimes.com/2018/06/11/technology/net-neutrality-repeal.html'" -H 'Content-Type: text/plain' /web Response Body: URL not provided

Here you just doubled the quotes, so it can't parse the URL. Otherwise, it looks like you're successfully reaching the server code, so you're pretty close.

I saw that up until a few months ago, there was a lambda_deploy script that automated some of the operations I mentioned above

Sorry about that. We've moved to a central script for all our serverless tools to avoid duplicating the deployment code, but we haven't yet committed that.

I specified my node_env as production in lambda_template.yaml, but there is no production.json file in the package directory created by ./lambda_package. Do I need to create a production.json file?

This part is just node-config. It will log a warning if you don't have a matching config file, but it won't throw by default. If you don't want to change anything, you can just put {} in a production.json file.

tyfried commented 5 years ago

It’s working! I was calling sam package and sam deploy from $dir/package instead of $dir 🤦

Also, when I put just {} into production.json I got Error: Configuration property "blacklistedDomains" is not defined, but when I set it to the contents of default.json it works fine.

Thank you for your help. I had convinced myself that the API Gateway was my only source for debug info, but once you mentioned CloudWatch I was able to pinpoint the problem pretty quickly. Also, this project (and Zotero in general) is awesome, thank you for making this available and accessible.