search

haskell-infra / hl

Haskell web site (old -- see new repo at https://github.com/haskell-infra/www.haskell.org)
http://haskell.org/
BSD 3-Clause "New" or "Revised" License
158 stars 92 forks source link

robots.txt (or sitemap) for downloads.haskell.org #214

Open chris-martin opened 7 years ago

chris-martin commented 7 years ago

Hi, I'm not sure this is the right place to open this issue, but I'm not sure where the code is for downloads.haskell.org, so I hope it's an acceptable discussion place or that somebody can direct me to the proper venue.

The SEO situation for GHC seems a bit hairy at the moment, and I was wondering if just adding a robots.txt to that site could yield some improvement.

Say, for example, you Google for "haskell type families". One of the first results I get is

https://downloads.haskell.org/~ghc/7.8.2/docs/html/users_guide/type-families.html

whereas the better result from haskell.org would be

https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/glasgow_exts.html#ghc-flag--XTypeFamilies

It seems unlikely that one would want to search old versions of the manual, so could we clear this up with a robots file with Disallow entries for the older GHC versions' docs?

gbaz commented 7 years ago

Not a bad idea. @thoughtpolice what do you think?

gbaz commented 7 years ago

also @bgamari

bgamari commented 7 years ago

I agree that our SEO situation is quite bad. Sadly this is not an area where I have a lot of knowledge. Outright disallowing old versions seems like a rather large hammer, but perhaps the benefit to effort ratio makes it worthwhile.

gbaz commented 7 years ago

Would a sitemap be more fine-grained?

On Fri, Sep 1, 2017 at 12:27 PM Ben Gamari notifications@github.com wrote:

I agree that our SEO situation is quite bad. Sadly this is not an area where I have a lot of knowledge. Outright disallowing old versions seems like a rather large hammer, but perhaps the benefit to effort ratio makes it worthwhile.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/haskell-infra/hl/issues/214#issuecomment-326624878, or mute the thread https://github.com/notifications/unsubscribe-auth/ABEt2e8IauKRfgLghRbKPGyiZET1l9Acks5seDBdgaJpZM4ObzBX .

bgamari commented 7 years ago

gbaz notifications@github.com writes:

Would a sitemap be more fine-grained?

This is actually the first I've heard of sitemaps. It sounds like composing a sitemap would be the right solution here.