A set of useful alternative package index mirrors?

[HuStmpHrrr]

Similar issue has been around for long and I cannot find a place where it shows what can be the alternative mirrors when s3 is not working well. Specifically, I was trying to download package index for the last 24 hours without managing it.

A couple of related issues are following:

https://github.com/commercialhaskell/stack/issues/1223 https://github.com/commercialhaskell/stack/issues/774 https://github.com/commercialhaskell/stack/issues/3088

So I am wondering if there is already a list of stable alternative mirrors which can be used already? I can only find such alternatives in github issues. Wouldn't it be a good idea to make it in documentation?

I cannot find such list in the documentation, my apologies if I looked over.

[hvr]

So I am wondering if there is already a list of stable alternative mirrors which can be used already?

As a matter of fact there is! The work on Hackage Security (funded by the Industrial Haskell Group (IHG)) made this possible, and there's even a DNS-based discovery protocol available which allows cabal and other Hackage clients to discover mirrors (as you obviously wouldn't want to hardcode them) when being bootstrapped in case the primary upstream is unreachable. But you can also take a look at http://hackage.haskell.org/mirrors.json to see a list of currently registered fallback mirrors; but generally from a UX perspective this is something that IMO should ideally be handled automatically by the tooling w/o requiring user intervention - maybe Stack could steal some ideas from Cabal in this context.

[mgsloan]

Looks like mirrors.json only has 2 things in it, and one is the default ` https://s3.amazonaws.com/hackage.fpcomplete.com/". The following seems to work for the fpco mirror, standard hackage and the dream.io mirror:

package-indices:
- name: fpco-hackage-mirror
  download-prefix: https://s3.amazonaws.com/hackage.fpcomplete.com/packages/
  http: https://s3.amazonaws.com/hackage.fpcomplete.com/
  hackage-security:
    keyids:
      [ "0a5c7ea47cd1b15f01f5f51a33adda7e655bc0f0b0615baa8e271f4c3351e21d"
      , "1ea9ba32c526d1cc91ab5e5bd364ec5e9e8cb67179a471872f6e26f0ae773d42"
      , "280b10153a522681163658cb49f632cde3f38d768b736ddbc901d99a1a772833"
      , "2a96b1889dc221c17296fcc2bb34b908ca9734376f0f361660200935916ef201"
      , "2c6c3627bd6c982990239487f1abd02e08a02e6cf16edb105a8012d444d870c3"
      , "51f0161b906011b52c6613376b1ae937670da69322113a246a09f807c62f6921"
      , "772e9f4c7db33d251d5c6e357199c819e569d130857dc225549b40845ff0890d"
      , "aa315286e6ad281ad61182235533c41e806e5a787e0b6d1e7eef3f09d137d2e9"
      , "fe331502606802feac15e514d9b9ea83fee8b6ffef71335479a2e68d84adc6b0"
      ]
    key-threshold: 3
- name: standard-hackage
  download-prefix: https://hackage.haskell.org/packages/
  http: https://hackage.haskell.org/
  hackage-security:
    keyids:
      [ "0a5c7ea47cd1b15f01f5f51a33adda7e655bc0f0b0615baa8e271f4c3351e21d"
      , "1ea9ba32c526d1cc91ab5e5bd364ec5e9e8cb67179a471872f6e26f0ae773d42"
      , "280b10153a522681163658cb49f632cde3f38d768b736ddbc901d99a1a772833"
      , "2a96b1889dc221c17296fcc2bb34b908ca9734376f0f361660200935916ef201"
      , "2c6c3627bd6c982990239487f1abd02e08a02e6cf16edb105a8012d444d870c3"
      , "51f0161b906011b52c6613376b1ae937670da69322113a246a09f807c62f6921"
      , "772e9f4c7db33d251d5c6e357199c819e569d130857dc225549b40845ff0890d"
      , "aa315286e6ad281ad61182235533c41e806e5a787e0b6d1e7eef3f09d137d2e9"
      , "fe331502606802feac15e514d9b9ea83fee8b6ffef71335479a2e68d84adc6b0"
      ]
    key-threshold: 3
- name: dream-hackage-mirror
  download-prefix: http://objects-us-west-1.dream.io/hackage-mirror/package/
  http: http://objects-us-west-1.dream.io/hackage-mirror/
  hackage-security:
    keyids:
      [ "0a5c7ea47cd1b15f01f5f51a33adda7e655bc0f0b0615baa8e271f4c3351e21d"
      , "1ea9ba32c526d1cc91ab5e5bd364ec5e9e8cb67179a471872f6e26f0ae773d42"
      , "280b10153a522681163658cb49f632cde3f38d768b736ddbc901d99a1a772833"
      , "2a96b1889dc221c17296fcc2bb34b908ca9734376f0f361660200935916ef201"
      , "2c6c3627bd6c982990239487f1abd02e08a02e6cf16edb105a8012d444d870c3"
      , "51f0161b906011b52c6613376b1ae937670da69322113a246a09f807c62f6921"
      , "772e9f4c7db33d251d5c6e357199c819e569d130857dc225549b40845ff0890d"
      , "aa315286e6ad281ad61182235533c41e806e5a787e0b6d1e7eef3f09d137d2e9"
      , "fe331502606802feac15e514d9b9ea83fee8b6ffef71335479a2e68d84adc6b0"
      ]
    key-threshold: 3

Not 100% sure what the value "keyids" should be. These are the keyids that are used for the fpco mirror, and they seem to work fine with hackage / dream.io.

This should probably be added to the documentation on haskellstack.org . It shouldn't be directly inlined in the configuration docs, but definitely the docs for "package-indices" should link to it - https://docs.haskellstack.org/en/stable/yaml_configuration/#package-indices

We could also consider having stack default to using a set of mirrors like this, for reliability.

[HuStmpHrrr]

We could also consider having stack default to using a set of mirrors like this, for reliability.

yeah, that will definitely be a wonderful idea. it's lots of suffering every time I setup haskell I will need to flip a coin to test my luck.

[mgsloan]

Hmm, I wonder what makes it unreliable. It has been a very long time since I've encountered any difficulties due to downtime of the S3 mirror. Far more often I've encountered hackage-security bugs (for now, it is best to avoid ctrl-C in the middle of an index update!)

On Fri, Feb 16, 2018 at 3:03 PM, Jason Hu notifications@github.com wrote:

We could also consider having stack default to using a set of mirrors like this, for reliability.

yeah, that will definitely be a wonderful idea. it's lots of suffering every time I setup haskell I will need to flip a coin to test my luck.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/commercialhaskell/stack/issues/3870#issuecomment-366383734, or mute the thread https://github.com/notifications/unsubscribe-auth/AABYKjJGfwddkqcnxrF2qCJYX8cksMsRks5tVgk7gaJpZM4SHyr- .

[HuStmpHrrr]

for now, it is best to avoid ctrl-C in the middle of an index update!

I am not sure it's just me or not cuz I cannot find any related issues, very often, stack download things at a much slower speed than what I would make in a browser, or even freezes. For instance, quite oftenly, downloading ghc binary might freeze, and it also happens for downloading indices as well(but I cannot tell since there is no progress indication). anyways, i don't know what can be done, but it's true that things do not seem quite stable.

[mgsloan]

Note that the mirrors json above had an error which I've corrected via an edit. The download-prefix field of fpco-hackage-mirror was missing a /.

[cblp]

Currently, Russia is blocking AWS and S3 from its own citizens. We can't use stack because of it.

[doofin]

some chinese mirror is working https://mirror.tuna.tsinghua.edu.cn/help/stackage/

[Yu-zh]

some chinese mirror is working https://mirror.tuna.tsinghua.edu.cn/help/stackage/

Thanks! This mirror is awesome!