How do you add or exclude routes?

[whydna]

1) Is it possible to define a list of routes? This is useful for routes which are not "linked" from any page - for example, landing pages for ad campaigns, etc.

2) Is it possible to exclude certain routes? This is useful for routes which have a lot of dynamic or private content, such as user account pages.

[juanse254]

1.- Just add them to the include, "Include : ["/". "/dir/"] 2.-There is a PR for that but AFAIK @stereobooster won't merge it to master, i forked and merged the exclude routes and the allow third party domain PR. You can look it up on my profile.

The real question is, how can we crawl a page that is behind authentication? whenever i visit a dynamic page which wasn't crawled and i hit refresh i get a non found error.

[stereobooster]

yes @juanse254 is right.

1. there is include option, to include pages. You need to use it only if crawler didn't catch some pages
2. there is a PR for exclude options, but I have doubts about it so it is not merged. You can use fork (it is possible to install using github link)

[FateRiddle]

Just add them to the include, "Include : ["/". "/dir/"]

@stereobooster @juanse254 to what include? in package.json? This is so important yet nowhere found in readme.

[stereobooster]

in the package.json

"reactSnap": {
  "include" : ["/". "/dir/"]
}

[vemundeldegard]

I have a few follow up questions on how to include routes if that is ok :)

In my app it crawls the "/" (index) path, but does not crawl other paths in my Router. My app gets it's paths from a CMS. So on page load it loads the routes from the CMS Menu component like this:

{menu.map(route => (
              <Route key={route._id} path={route.path} exact render={(props) => <Page {...props} isAuthed={true} />} />
))}. 

Is there a way without using the "Include" option manually to make sure all is crawled?

How can I make it crawl my blog posts for example, that appears in my routes with a pathslug? <Route path="/blogg/:slug" exact component={Post} />. Some posts are crawled, but not all. I am not sure why.

[stereobooster]

If links are present in generated HTML then crawl should catch it, if it is not present it means that prerenderer didn't catch all the content. Maybe some fetch requests to your CMS failed. How do you handle failed requests? Put console log there or throw an error and it will be visible in logs.

You can use react-snap programatically

const { run } = require("react-snap");
run({
  include: getAllMyRouteFromCMS()
})

[vemundeldegard]

Will cheak that out. Thanks!

I am noticing that even though the console logs on build that it has crawled a path, doesn't mean that it is actually crawled? I am not getting any static html.

crawled 3 out of 8 (/blogg/test-post)

[stereobooster]

If there is crawled 3 out of 8 (/blogg/test-post) and you don't have /blogg/test-post/index.html it means there is a bug, you can open issue

[vemundeldegard]

I don't have a direct /blogg/test-post/index.html file, but I do have /blogg/post.js which is my component for the blog posts which is accessed at /blogg:slug. Can that be the issue?

[stereobooster]

I'm not sure. You mean an issue with mkdir? Maybe. Do you have reproducible example?

[vemundeldegard]

You can check out this repo: https://github.com/vemundeldegard/gauc

Live example of post that should be crawled: https://vizible.surge.sh/blogg/dette-er-en-test

[vemundeldegard]

I also tried this, but it fails to compile:

import { run } from "react-snap"

const RoutesFromSanity = () => {
  return ['/', '/kontakt-oss', '/blogg']
}

run({
  include: RoutesFromSanity()
})

Failed to compile.

./node_modules/puppeteer/lib/WebSocketTransport.js Module not found: Can't resolve 'ws' in '/Users/vemundeldegard/gauc/node_modules/puppeteer/lib'

[constantin-melniciuc]

@stereobooster @juanse254 anybody could provide the link to the forked version with exclude? Would be super handy to have that option. Thanks.

[juanse254]

The original exclude was from zephyr but I merged that and the thirdPartyDomain branch into my branch https://github.com/juanse254/react-snap. Cheers dude @JoraCardan