Craweler not fetching all available components

kiliman / tailwindui-crawler

tailwindui-crawler downloads the component HTML files locally

MIT License

763 stars 95 forks source link

Craweler not fetching all available components #3

Closed darsenault closed 4 years ago

darsenault commented 4 years ago

Thank you for developing this tool! Great idea and work for sure.

I noticed that the crawler is not pulling all available snippets/components.

➜  tailwindui-crawler node index.js
Processing /components/marketing/sections/heroes...
* Found 1 snippet
Writing ./output/components/marketing/sections/heroes/with_angled_image_on_right.html...

There are 3 snippets not just 1:

Thank you.

kiliman commented 4 years ago

Did you add your username and password to the .env file? When I see only 1 component, that typically means it failed authentication and is simply showing the free "preview" page which is very limited.

darsenault commented 4 years ago

I did create the .env. But, it appears I had a typo from a bad copy/paste. Recreating the file indeed fixed the problem. Your crawler works beautifully. My apologies for the bother. Thank you!

kiliman commented 4 years ago

I've updated the crawler to check for success/failure on authentication so you won't get false positives