dispherical / quizlet-fetcher

Fetch quizlet cards by scrapping the quizlet website.
GNU Lesser General Public License v3.0
9 stars 1 forks source link

Not working on AWS Lambda #1

Closed awolad closed 2 years ago

awolad commented 2 years ago

It's working locally but getting an error when using it inside AWS lambda.

Failed to fetch the quizlet cards. Status code: 403

dispherical commented 2 years ago

Quizlet has a anti-bot firewall setup (sort of like CloudFlare). The only way I could really suggest fixing it is getting a proxy.

awolad commented 2 years ago

@aboutDavid Thanks for your reply. Could you please give me an example?

dispherical commented 2 years ago

@aboutDavid Thanks for your reply. Could you please give me an example?

Sorry, I don't really use proxies, but here is something I threw together

const quizlet = require("quizlet-fetcher")
const fetch = require("node-fetch-with-proxy");

fetch("https://quizlet.com/...")
  .then(res => res.text())
  .then(html => {
  var flashcard = quizlet(html)
  // do something with "flashcard"
});

Edit: forgot one thing: You need to set your proxy

# UNIX-based operating systems
export HTTP_PROXY=http://your.proxy:8888
# Windows
set HTTP_PROXY=http://your.proxy:8888
awolad commented 2 years ago

@aboutDavid I'll try it, thanks.

dispherical commented 2 years ago

Let me know if something is not working as intended