search

rust-lang / crates.io

The Rust package registry
https://crates.io
Apache License 2.0
2.91k stars 595 forks source link

robots.txt is too restrictive, preventing Discord from generating embeds #7947

Closed ds84182 closed 4 months ago

ds84182 commented 7 months ago

Current Behavior

When linking a crate on Discord, the link does not generate an embed. Discord specifically checks for the Discordbot user agent, since embeds are done server-side. This does not affect Twitter because Twitter embeds do not check robots.txt.

Expected Behavior

Linking a crate on Discord should display the crate name and the description, similar to sites like Twitter.

Steps To Reproduce

  1. Open Discord, any server (with embed permissions) or Direct Message.
  2. Post a link to a crate on crates.io
  3. No embed :(

Environment

Anything else?

No response

LawnGnome commented 7 months ago

I, for one, am shocked — shocked — that Twitter might not respect a standard to prevent misuse of the internet. :astonished:

Do you know if Discord documents this anywhere? I'm not opposed to changing this (and adding similar robots.txt rules for the Facebook and Twitter bots), but I would like to add a cite to our robots.txt in case it changes further in the future so we don't have to go figure this out again next time.

ds84182 commented 7 months ago

Discord doesn't exactly document the user agent they use, but it's been referenced in their API doc issues (https://github.com/discord/discord-api-docs/issues/1600) as well in Twitter's robots.txt (because they, annoyingly, disabled embeds from Discord)

Turbo87 commented 7 months ago

Linking a crate on Discord should display the crate name and the description, similar to sites like Twitter.

worth noting that our og:image for Twitter does not display a crate name or description (yet) either. this is just a static image and without server side rendering the rest of the metadata will also not be dynamic, unless Discord runs JS on their scrapers.