[EPIC] SEO: Make datasets findable on search engines

[meh9]

At the moment if you search for some specific dataset that you are interested in using some internet search engine, no links to that dataset on any TerriaJS instances will be found unless maybe if somebody has made a share URL and put that into a static document which was then crawled by a search engine. This is a shame because this would probably lead to a lot more users finding useful data.

The problem is that search engine crawlers can probably not "run" the TerriaJS instance JavaScript in order to meaningfully extract the data.

We should implement some feature that makes it possible for search engines to deep link to the layers provided on TerriaJS instances.

This might take a number of forms:

• A dynamically generated page that lists all datasets with deep links (share URLs?) into the TerriaJS instance that a search engine can find and crawl.
• Some form of sitemap? I know Google can accept that, but even if we made one with the share URL's of all the available layers I don't think it would be able to extract the name/metadata.
• Detect when the client is a crawler and then present a "HTML only" version of the catalogue with all layers available together with their metadata.
• Some other as-yet-unthought-of clever thing.

[stevage]

Yeah, nice one. Hard, but important.

• "dynamically generated page that lists all datasets with deep links (share URLs?)": That sounds like a pretty good outcome, and isn't harder to achieve than the others.
• "Some form of sitemap": You'd think this would be the right answer, but yeah, without being able to provide any metadata, I can't see how it would work. It's just a list of raw URLs - where would they go to? Share URLs? GetCapabilities etc endpoints?
• "Detect when the client is a crawler and then present a HTML-only version of the catalogue": This is essentially the same as the first option, with just a minor difference in when that HTML page is presented.
• "as-yet-unthought-of clever thing": sure. :)

Btw, I think your basic premise that Google can't run JavaScript isn't quite right: http://searchengineland.com/tested-googlebot-crawls-javascript-heres-learned-220157

I suspect that even once the page is rendered, it's still not at all search engine friendly though.

[soyarsauce]

I am still working through this, have a demo up & hope to get a set of 3 PRs up for it (for TerriaMap, terriajs & terriajs-server)

[soyarsauce]

droughtmap is in Oct decommission, moving to next sprint

[AnaBelgun]

Update 15 March:

• start with a write up of current state and future plan
• break it down into smaller tickets
• think about site map
• consider using a server side approach

[AnaBelgun]

Update 12 Apr:

• to be progressed this sprint

[steve9164]

For SEO we should generate a sitemaps of catalogue routes (i.e. map.terria.io/catalog/d89fe routes) and ensure that these pages can be crawled.

Options:

• Server Side Rendering
  • Fetch metadata from our server, render TerriaMap and return to client
  • Not feasible because metadata requests can be long
• Client Side Rendering
  • Potentially not as good for SEO
  • Crawlers may leave our pages before metadata has loaded
  • Could be very easy to test this one using our existing infrastructure, fixed meta tags and a sitemap
• Pre-rendering
  • Change indexer to also pre-render every catalogue page, and change infrastructure as needed to serve these
• Server Side Rendering using catalog index
  • A mix of pre-rendering and server-side rendering
  • Use catalog index details to server-side render a page
  • Probably the hardest to maintain, and no real advantage to pre-rendering

Next steps: Try improving client side rendering, then look at pre-rendering.

[AnaBelgun]

Update 7 Jun 23:

• 1st step - get home pages indexed