search

teticio / lambda-scraper

Use AWS Lambda functions as a proxy pool to scrape web pages.
BSD 3-Clause "New" or "Revised" License
112 stars 14 forks source link

Can I please ask a few beginner questions? #1

Closed chrisjdixon closed 3 years ago

chrisjdixon commented 3 years ago

Hi @teticio , I've come across your guide for this repo. Looks amazing! I'm trying to follow instructions but have never done anything with Terraform, AWS CLI, or Lambda before so I'm a little confused about a few things. Your instructions seem clear and straightforward but I'm a little lost through inexperience.

Do you mind if I ask a few dumb questions? I don't know where else I could message you so is this the most appropriate place? In the issues section of your repo? Because I wouldn't classify this as an "issue" 😅

teticio commented 3 years ago

Sure. You can email me at @.*** Happy to help

On Sun, 11 Jul 2021 at 07:51, chrisjdixon @.***> wrote:

Hi @teticio https://github.com/teticio , I've come across your guide https://medium.com/nerd-for-tech/web-scraping-with-a-proxy-pool-the-cheap-way-4c7d6fc9f859 for this repo. Looks amazing! I'm trying to follow instructions but have never done anything with Terraform, AWS CLI, or Lambda before so I'm a little confused about a few things. Your instructions seem clear and straightforward but I'm a little lost through inexperience.

Do you mind if I ask a few dumb questions? I don't know where else I could message you so is this the most appropriate place? In the issues section of your repo? Because I wouldn't classify this as an "issue" 😅

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/teticio/lambda-scraper/issues/1, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKRPDBZSSWZXIVBK62P3W2LTXE5NVANCNFSM5AE7YFUA .

ElMurte commented 3 years ago

image Hi man, i'm a beginner in that field, i followed your instructions but when i lunch it, i got the error in the image n times (n number of function tried to deploy)

PS C:\Desktop\Web_scrapper\lambda-scraper> terraform apply -auto-approve ╷ │ Error: Plugin error │ │ with module.lambda_function[3].module.source_zip_file.data.external.archive[0], │ on .terraform\modules\lambda_function.source_zip_file\main.tf line 28, in data "external" "archive": │ 28: data "external" "archive" { │ │ The plugin returned an unexpected error from plugin.(*GRPCProvider).ReadDataSource: rpc error: code = Internal desc = │ grpc: error while marshaling: string field contains invalid UTF-8

teticio commented 3 years ago

What OS are you running on? I just tried running it on Windows and had some issues (different from yours). Unfortunately it looks like an issue with the Lambda function module which is developed by someone else. It works on Linux no problem.

I did remove a file from the repo that might solve your problems. Try running everything from scratch in a fresh directory.

teticio commented 3 years ago

OK, I have changed the module for Lambda functions to another provider and it now works on Windows. There was a small wrinkle when I ran it - it had some problem moving a file as it was in use - but running it twice did teh trick (don't worry, terraform will not create double the resources if you run it twice!). Hope this helps. (Again start from scratch in a new folder. To be on the safe side, run "terraform apply -destroy" in your existing folder in case there are some artefacts left over from a partial deployment.)

ElMurte commented 3 years ago

Thanks a lot it works, i'm using Windows. Really appraciate, thanks to share this template with us. Have a good one! 👍