vision on geonode and search engine access

[pvgenuchten]

Is your feature request related to a problem?

Compared to v3, the v4 version has some challenges for search engine crawlers to identify content in a geonode portal

For example when you use a crawler tester like https://totheweb.com/learning_center/tools-search-engine-simulator/ and try both urls

• https://development.demo.geonode.org/catalogue/#/dataset/548
• https://stable.demo.geonode.org/layers/httpspro-ags1dfsunorgarcgisrestservicesclearmap_webplainmapserverwmts100:httpspro-ags1dfsunorgarcgisrestservicesclearmap_webplainmapserverwmts100:12

You will notice that in the first case (v4) only the single page web application is accessed and not the dataset related information. In the second case (v3) the dataset information is available.

Describe a solution

If the geonode community is interested to allow geonode be crawled by search engines, there are some solutions:

• enable in the single page application the ability to be crawled by search engines, see this link. for the options
• pycsw these days has ogcapi-records support, ogcapi-records can be crawled by search engine by design

[giohappy]

hi @pvgenuchten this is a known issue. I have a branch with a WIP fix, but it's not ready yet (just a matter of time to complete it). My solution was quite simple: create a new "share" view, which only renders the required meta tags. This view contains an automatic redirection to the real app. Web crawlers generally don't follow redirections, and many of them don't execute js, so this solution seems effective and simple at the same time.

Did you already implement an alternative solution?

[pvgenuchten]

thanx @giohappy, no i have not implemented any solution yet, just evaluating the options

On another project i used the forward method using onload(location.href='/#/foo'), but google didn't like it, it stopped indexing our site...

[giohappy]

Oh really?! That's what I was implementing... Only Google stopped indexing?

[pvgenuchten]

please give it a try, behaviour may be different these days. google crawler does parse some javascript, so if you forward users, the crawler may notice it and you will run in the same single-page-web challenge.

we only checked google, yandex/bing little to no experience here