Evaluate crawling capability of Sage websites, portals and apps

[tschaffter]

The idea is to crawl these sites to identify whether they are optimized for SEO. We will use the crawler recently added to this repo.

Main:

• https://synapse.org
• https://sagebionetworks.org/

Portals:

• https://agora.adknowledgeportal.org
• https://adknowledgeportal.synapse.org/
• https://nf.synapse.org/
• https://csbc-pson.synapse.org/
• https://www.cri-iatlas.org/

Shiny apps:

• https://isb-cgc.shinyapps.io/iatlas

[tschaffter]

Configuration

const generator = SitemapGenerator(siteUrl, {
  filepath: sitemapFilepath,
  maxDepth: 2,
  maxEntriesPerFile: 50000,
  stripQuerystring: false
});

https://sagebionetworks.org

node tools/generate-sitemap.js http://www.sagebionetworks.org sitemap-sagebionetworks.xml

Number of <url>: 29

https://synapse.org

node tools/generate-sitemap.js http://www.synapse.org sitemap-synapse.xml

Number of <url>: 1

https://agora.adknowledgeportal.org

node tools/generate-sitemap.js https://agora.adknowledgeportal.org sitemap-agora.xml

Number of <url>: 1

https://adknowledgeportal.synapse.org

node tools/generate-sitemap.js https://adknowledgeportal.synapse.org sitemap-adknowledgeportal.xml

Number of <url>: 1

https://nf.synapse.org

node tools/generate-sitemap.js https://nf.synapse.org sitemap-nf.xml

Number of <url>: 1

https://csbc-pson.synapse.org

Note https://csbc-pson.synapse.org redirects to https://cancercomplexity.synapse.org/

node tools/generate-sitemap.js https://cancercomplexity.synapse.org sitemap-cancercomplexity.xml

Warning The crawler does not save the sitemap file when using https://csbc-pson.synapse.org (because of the redirection?).

Number of <url>: 1

https://www.cri-iatlas.org

node tools/generate-sitemap.js https://www.cri-iatlas.org sitemap-iatlas.xml

Number of <url>: 6

https://isb-cgc.shinyapps.io/iatlas

node tools/generate-sitemap.js https://isb-cgc.shinyapps.io/iatlas sitemap-iatlas-shiny.xml

Warning The crawler does not save the sitemap file (here there is no redirection).

[tschaffter]

About the crawler not saving sitemap file

• The crawler failed to generate a sitemap.xml file for https://csbc-pson.synapse.org but worked fine when targeting the redirected URL, https://cancercomplexity.synapse.org.
• The crawler failed to generate a sitemap.xml file for https://isb-cgc.shinyapps.io/iatlas. It seems that the generator fails to generate a sitemap for Shiny apps as the same result was observed for the following Shiny apps:
  • https://danielrivera1.shinyapps.io/Regression2
  • https://gallery.shinyapps.io/nz-trade-dash
  • https://vac-lshtm.shinyapps.io/ncov_tracker

[tschaffter]

Terminology

• CSR: Client-side rendering
• SSR: Server-side rendering
• SE: Search engine
• SEO: Search engine optimization

Results

Valid on 2022/08/04.

Site	Pages found	SSR enabled	sitemap.xml	Technology	Contact
https://synapse.org	1	❌	❌	Google Web Toolkit	Jay Hodgson
https://sagebionetworks.org	29	✅	✅	WordPress
https://www.cri-iatlas.org	6	✅	✅	WordPress
https://challenge-registry.org	3	✅	✅	Angular	Thomas Schaffter
https://agora.adknowledgeportal.org	1	❌	❌	Angular	Anna Greenwood
https://adknowledgeportal.synapse.org	1	❌	✅	React	Jay Hodgson
https://nf.synapse.org	1	❌	✅ (empty)	React	Jay Hodgson
https://cancercomplexity.synapse.org	1	❌	✅	React	Jay Hodgson
https://isb-cgc.shinyapps.io/iatlas	NA*	✅	❌	Shiny	Andrew Lamb

*The crawler used seems to be unable to crawl Shiny apps.

Conclusions

• Google search engine is capable of crawling CSR apps but most of the other browsers can't (Bing, DuckDuckGo, etc.).
• All SEs are capable of crawling SSR apps.
• Serving an app with SSR instead of CSR offers the following benefits:
  • Make our application more SE friendly.
  • Improve the social media presence of our application.
  • Server-side rendered/cached pages are generally displayed quickly while the JavaScript app bundle is being downloaded. See Rendering on the Web for a comprehensive description of the benefits and drawbacks.
  • Generate the sitemap.xml file automatically, e.g. using the script tools/generate-sitemap.js by this project.

References

• Rendering on the Web
• Server-side rendering (SSR) with Angular Universal