How to deal with basic auth?

amoilanen / js-crawler

Web crawler for Node.JS

MIT License

253 stars 55 forks source link

How to deal with basic auth? #57

Open pittersnider opened 5 years ago

pittersnider commented 5 years ago

Dear developers,

I am crafting a tool that let me automatically crawl a few sites.
However, they are protected by a username and password (that I have).

Which is the correct way of passing this information to js-crawler?

Best regards,
Pitter.

sbr2567 commented 4 years ago

From what I understand js-crawler and other http request packages can only manage static html. Meaning you can't dynamically interact with the page. You'd have to visit the url's through a headless browser like puppeteer for auth.