search

stereobooster / react-snap

👻 Zero-configuration framework-agnostic static prerendering for SPAs
MIT License
5.06k stars 392 forks source link

Idea: use rel=nofollow to prevent crawling by ReactSnap #82

Open stereobooster opened 6 years ago

stereobooster commented 6 years ago

Related https://github.com/stereobooster/react-snap/pull/42

strobox commented 6 years ago

Agree partially. Don't want to include 1 to 1000...n....000 (hundreds thousand millions) pages of pagination. So rel=nofollow and/or some attribute of anchor (for example: <a data-skip-snap={true} ) will be good options.

strobox commented 6 years ago

@stereobooster Looks like one line need to be changed

    Array.from(document.querySelectorAll("a:not([data-skip-snap]):not([rel=nofollow])")).map(anchor => anchor.href)

Have sense to do PR?

stereobooster commented 6 years ago

I also thought about rel="noindex". I'm not sure if this a good idea or not, so if you will send PR it can stuck