gadenbuie / metathis

:information_source: <meta> tags and social media cards for R-made web things
https://pkg.garrickadenbuie.com/metathis
Other
67 stars 3 forks source link

Deploying on shinyapps.io when crawlers are blocked via robots.txt #11

Open gadenbuie opened 4 years ago

gadenbuie commented 4 years ago

shinyapps.io uses a root-level, non-configurable robots.txt file to block all web crawlers. Any web crawlers that respect the robots.txt specification will not show a preview for your app. This is why preview cards won't appear as expected on Twitter.

There are good reasons for this default behavior. Primarily, blocking web crawlers keeps ensures that your app isn't started up (incurring costs and hours against your quota) just to render the preview card. On the other hand, most social media sites implement some sort of caching mechanism.

At the moment, I am not aware of any way to configure the settings in the robots.txt file. It is served globally at the root level of <user_name>.shinyapps.io, so an app-specific robots.txt stored at www/robots.txt won't have an effect or change the setting.

I'll update this issue if these settings become configurable, or if anyone reports a work-around.