Redirect javadoc URLs for the package iframe to the full javadoc page

[kinueng]

Problem

One of the iframe content from the javadocs are being indexed by search engines.

Recreation steps

I subset of URLs that should not be indexed by search engines.

• https://openliberty.io/docs/modules/reference/microprofile-4.0-javadoc/org/eclipse/microprofile/context/package-frame.html
• https://openliberty.io/docs/modules/reference/liberty-jakartaee9.1-javadoc/jakarta/enterprise/inject/package-frame.html

Possible Solutions 1

• Mark these URLs as no index

Possible Solution 2

Redirect these URLs to the appropriate page. The decision needs to be made which page is considered the "appropriate". Here are some choices

1. https://openliberty.io/docs/latest/reference/javadoc/liberty-jakartaee9.1-javadoc.html?package=jakarta/enterprise/inject/package-frame.html&class=overview-summary.html
   • Drawback is only the bottom left iframe changes to jakarta.enterprise.inject

2. https://openliberty.io/docs/latest/reference/javadoc/liberty-jakartaee9.1-javadoc.html?class=jakarta/enterprise/inject/package-summary.html&package=allclasses-frame.html
   • Drawback is only the right iframe changes to jakarta.enterprise.inject

3. https://openliberty.io/docs/latest/reference/javadoc/liberty-jakartaee9.1-javadoc.html?package=jakarta/enterprise/inject/package-frame.html&class=jakarta/enterprise/inject/package-summary.html
   • This looks to be the best end result solution where all frames are in context for jakarta.enterprise.inject package.

[kinueng]

Found similar URLs that users should not be using but they are showing up in search results

Search example https://www.google.com/search?q=Package+jakarta.batch.runtime.context+-+Open+Liberty

• https://openliberty.io/docs/modules/reference/liberty-jakartaee9.1-javadoc/index.html?jakarta/enterprise/context/spi/package-summary.html

[kinueng]

Our thoughts are that the code that handles a URL like https://openliberty.io/docs/modules/reference/liberty-jakartaee9.1-javadoc/jakarta/batch/api/AbstractBatchlet.html and redirects it to https://openliberty.io/docs/latest/reference/javadoc/liberty-jakartaee9.1-javadoc.html?package=jakarta/batch/api/package-frame.html&class=jakarta/batch/api/AbstractBatchlet.html is unable to handle the two broken URL examples in the issue description.

Start with looking at the code that handles transforming the URLs and redirecting.

[Sreejith-Websphere]

Hi @kinueng noticed that for this issue for all the jakarta/javaee/microprofile which ends frame.html is not redirecting eg: https://openliberty.io/docs/modules/reference/liberty-javaee8-javadoc/overview-frame.html https://openliberty.io/docs/modules/reference/liberty-javaee8-javadoc/allclasses-frame.html https://openliberty.io/docs/modules/reference/liberty-javaee8-javadoc/allclasses-noframe.html https://openliberty.io/docs/modules/reference/liberty-javaee8-javadoc/overview-frame.html https://openliberty.io/docs/modules/reference/liberty-javaee8-javadoc/javax/annotation/package-frame.html

https://openliberty.io/docs/modules/reference/microprofile-5.0-javadoc/overview-frame.html https://openliberty.io/docs/modules/reference/microprofile-5.0-javadoc/allclasses-frame.html

https://openliberty.io/docs/modules/reference/liberty-jakartaee9.1-javadoc/overview-frame.html https://openliberty.io/docs/modules/reference/liberty-jakartaee9.1-javadoc/jakarta/activation/package-frame.html

[kinueng]

The last remaining piece is to decide how to mark the javadoc files used for iframes as noindex to avoid search engines indexing the iframes. Example of the iframe files are in comment https://github.com/OpenLiberty/openliberty.io/issues/2665#issuecomment-1170319592. We cannot put the HTML element noindex into the files because the files are generated by a script command javadoc.

[natalie-bernhard]

We may be able to use rules in our robots.txt to prevent crawlers from seeing the iframes by setting rules specific to these files still causing issues (overview-frame.html, package-frame.html, allclasses-frame.html). We can also target the bot that indexes for Google Search specifically if using the * user agent prevents the iframes from loading on the site.

More info here: https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt#create_rules