Deal with robots with React UI

[nworden]

We're currently keeping bots away from certain pages (namely search results and person record pages) with a meta tag, implemented here, but it's not going to be that easy with the React app. We can't use a robots.txt file because PF lives in a subdirectory (google.org/personfinder).

I think there's a couple options:

• Don't deal with it with React, serve Lite/AMP UI to robots so we can just stick with what we've been doing. The disadvantage of this is that there's a lot of bot user agents so this might not be feasible or practical.
• Use react-helmet. Advantages are the NY Times says it's (mostly) fantastic and its name is a pun. Disadvantage is it wouldn't work if we wanted to do server-side rendering.The project doesn't get a lot of activity, but it does get some (it's not totally forgotten), so I think/hope the infrequency of commits is because it's stable rather than because it's neglected.
• Use react-helmet-async (article here), a fork of react-helmet from an engineer at the NY Times. The advantage of react-helmet-async is that it'll work if we want to use server-side rendering. A disadvantage is that it's not clear that it has the support of the NYT itself: it's under the engineer's personal GH account, so we'd be relying on him personally, instead of an organization, to keep it alive.
• Some other library. I haven't fully dug into this, but there's probably more than just the two above.

[gimite]

• Don't deal with it with React, serve Lite/AMP UI to robots so we can just stick with what we've been doing. The disadvantage of this is that there's a lot of bot user agents so this might not be feasible or practical.

Another disadvantage here is that search engines may consider that our site is cloaking, because we return different pages to bots and real browsers, which (I believe) they discourage.

At a glance, use of react-helment sounds good to me, since (I believe) we do not have immediate plan of server side rendering.

Another potential option is to implement the logic in the server side (i.e., we do the same thing as we do today only for the meta tag part). It should work because (I believe) search engines always directly fetch pages, not following links by running JavaScript (which causes pushState).