Is the gazebosim.org main page content not accessible to search engine crawlers?

j-rivero commented 1 year ago

Looking into why the gazebosim.org page is mostly ignores for the Google searches I found some hints that can point out that the gazebosim.org main page is totally invisible to the search crawlers:

wget http://gazebosim.org/ is not able to see any real content:

  <meta name="viewport" content="width=device-width, initial-scale=1">           
  <link rel="icon" type="image/x-icon" href="/assets/icon/favicon.ico">          
  <title>Gazebo</title>                                                          
  <meta name="description" content="Simulation Driven Robotics. Iterate quickly on design concepts and control strategies with Gazebo's rich suite of tools, libraries, and cloud services.">

  <base href="/">                                                                
  <style type="text/css">@font-face{font-family:'Roboto';font-style:normal;font-weight:300;src:url(https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmSU5fCRc4AMP6lbBP.woff2) format('woff2');unic  <style type="text/css">@font-face{font-family:'Montserrat';font-style:normal;font-weight:300;src:url(https://fonts.gstatic.com/s/montserrat/v25/JTUSjIg1_i6t8kCHKm459WRhyyTh89ZNpQ.woff2) format('wo  <style type="text/css">@font-face{font-family:'Material Icons';font-style:normal;font-weight:400;src:url(https://fonts.gstatic.com/s/materialicons/v139/flUhRq6tzZclQEJ-Vdg-IuiaDsNcIhQ8tQ.woff2) fo mat('                                                                           
  <style>                                                                        
  body, html {                                                                   
    background-color: #ffffff;                                                   
    height: 100%;                                                                
  }                                                                              
  </style>                                                                       
<style>html,body{margin:0;padding:0;width:100%;height:100%;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;text-rendering:optimizeLegibility}</style><link rel="stylesheet" href= style                                                                           
<body>                                                                           
  <gz-app-root></gz-app-root>                                                    
<script src="runtime.1423ce2e84ce5652.js" type="module"></script><script src="polyfills.0b2dc2934fc02624.js" type="module"></script><script src="scripts.417efceffc01cbd5.js" defer></script><script s c="ma                                                                           
</body></html>

seobility.net is reporting that there are no h1 elements in the page, while if you check the web in your browser, there are:

seobility.net is reporting that there is no content in the web

screamingfrog seo tool crawls no subpages from the main page, just scripts

azeey commented 1 year ago

It might not be as dire as it seems from the wget command because this is an Angular app, which needs javascript to run to load the page. I don't think wget runs javascript, but I think search engines do, or at least this article makes it sound like they do. But yeah, since most of gazebosim.org is static, it would be nice to serve simple html content via pre-rendereding or changing the frontend completely. I would love to use something like jekyll, but the engineering effort to convert is probably not well justified.

On a related note, running wget https://gazebosim.org/docs gives a 404 error. /cc @nkoenig

j-rivero commented 1 year ago

I think search engines do, or at least this article makes it sound like they do.

Reading the article (note that is an article from upstream code, which could suggest a bias), it says:

Understanding the SEO implications of a single page app There is one major reason why single page apps are not used everywhere so far (with two separate causes): Single page apps don't perform well on search engines

About Google it says:

The good news is that none of these two reasons are 100% accurate anymore! Google has started to index better single page apps.

mention that "none of these two reasons are 100% accurate anymore!" Google "has started to index better". More on other search engines:

This is great news, but there are other search engines that drive significant traffic. Baidu for example drives more than half of the traffic in China (curr. 1.3 billion people).

so the problem is still there for other search engines and we don't really know how bad is affecting the Google crawler.

Note that the article suggest the use of something called "Angular Universal"

What is Angular Universal, what does it enable? Simply put, Angular Universal enables us to build apps that have both the performance and user engagement advantages of single page apps combined with the SEO friendliness of static pages.

Which is the same that other SEO pages suggest when using Angular. I totally brainstorming here and excuse me if this is totally wrong. just found some pointers in the same direction to solve a problem that I think it is real.

gazebo-web / gazebosim-web-frontend

Is the gazebosim.org main page content not accessible to search engine crawlers? #20