Serve robots.txt

[nishtacular]

Hey guys how do i serve robots.txt on my site. I currently have it saved under the static directory.

[nishtacular]

@fridays any tips? Thanks

[breeny]

@nishtacular I'm not sure this is a routing issue - serving static assets is part of nextjs itself.

Given you need to implement a server.js file, I'd say you should add a route in express directly that serves the robots.txt file (or redirects to the static directory).

[nishtacular]

Thanks for the reply @breeny . As I am using next-routes for routing of the app, I am unsure on how to go about setting that up. Appreciate any tips. This is my server.js file

// server.js
const next = require('next')
const routes = require('./routes')
const app = next({dev: process.env.NODE_ENV !== 'production'})
const handler = routes.getRequestHandler(app)

// With express
const express = require('express')
app.prepare().then(() => {
  express().use(handler).listen(3000)
})

[breeny]

So what you need to do is add a route at the Express layer, before adding the next-routes middleware. Then you can use that route to serve your file:

const next = require('next')
const routes = require('./routes')
const app = next({dev: process.env.NODE_ENV !== 'production'})
const handler = routes.getRequestHandler(app)

// With express
const express = require('express')
app.prepare().then(() => {
  const server = express();

  //You can tell express to serve all files from the static file from route
  //(just make sure that there's no name conflicts as the file would override your react routes)
  server.use(express.static('/static'));

  //Or if you want to just serve robots.txt and nothing else from root
  server.use('/robots.txt', express.static('/static/robots.txt'));

  server.use(handler).listen(3000)
})

There's two different types of router at play - next-routes which is the routing from within the React application, and then Express as the server and server side router.

For clarity, what the line server.use(handler) does is provide middleware to Express that tells it to pass all routes that haven't already been matched above that to it. The handler then passes those routes into next-routes to be handled in React. By providing an Express route above that middleware, routes that match in Express will never be passed to next-routes, so is handled purely in the Node application.

Hope that helps clarify.

[nishtacular]

Hi @breeny, appreciate the detailed reply and insight, it really helped. I made a small adjustment to the code as it did not work without it for me.

server.use('/robots.txt', express.static(join(__dirname, '/static/robots.txt')));

Thanks again 😃

[ColinPNtenga]

Hello @nishtacular i came across this issue when searching for a way to serve static files. I am also using next-routes, my static file is showing under this url "/static/myStaticFile", was wondering is their a way exclude "/static" from showing in the browser url?

Thnaks in advance!

[nishtacular]

@ColinPNtenga did you try the following? It does not show the /static/ path in the url

server.use('/robots.txt', express.static(join(__dirname, '/static/robots.txt')));

[ColinPNtenga]

@nishtacular i tried using the same setup as you suggested but am getting a 404 error (page doesn't exist) in the url if i exclude the /static/ but works fine when including it. Do i need to define those routes inside my routes.js file ?

[ColinPNtenga]

@nishtacular for some weird reason the following setup works, but am not sure why? are you able to explain this for me please?

server.get('/sitemap.xml', (req, res) => {
      app.serveStatic(req, res, path.resolve('./static/sitemap.xml'))
    })