search

haskell / hackage-security

Hackage security framework based on TUF (The Update Framework)
http://hackage.haskell.org/package/hackage-security
56 stars 47 forks source link

Missing package.json files for some packages? #183

Closed snoyberg closed 6 years ago

snoyberg commented 7 years ago

When I turn on "require package hashes" in Stack on the Hackage Security branch, I get the following error message:

Package index Hackage is configured to require package hashes, but no hash is available for amazonka-codedeploy-1.4.0

Sure enough, if I look in the 01-index.tar file, there is no package.json file for that release of amazonka-codedeploy:

screen shot 2017-01-25 at 4 04 32 pm

Is there some reason for this file to be missing? From a security standpoint, it would be nice to be able to depend on the existence of a hash for every package.

Also, I'm not sure if this is the appropriate repo for this question. I can move it to the Hackage Server tracker if that's better.

hvr commented 7 years ago

We have known about this for quite some time already (c.f. https://github.com/haskell/hackage-server/issues/488) but this glitch is very hard to reproduce and hasn't re-occured ever since then. This is also the reason why the hackage-mirror-tool needs this temporary hack. At some point we're gonna supply those missing package.json files in the index tarball, we just didn't get to it yet.

snoyberg commented 7 years ago

Is there no method available to:

  1. Manually add the missing package.json files, and
  2. Add a sanity check (like each .cabal file has a package.json file) before the 01-index.tar.gz file is generated?

This does seem to significantly impede the ability of tools to provide security guarantees.

hvr commented 7 years ago

This does seem to significantly impede the ability of tools to provide security guarantees.

That's not really true though because as a kind of happy accident this unintentional situation currently exercises cabal's code-paths for when the package hashes are missing; and in fact cabal refuses to download those couple of packages whose cryptographic checksums are missing. This just forces us to be able to cope with this exceptional case which we would have to anyway.

tfausak commented 7 years ago

Is there any way for me to help with this issue? It has been a known problem for a year and a half. It continues to happen as recently as a week ago. It does not seem like something that tools should have to deal with; if a .cabal file is in the index, a package.json file should be there too.

tfausak commented 7 years ago

It's hard to tell how often this happens because the original report (https://github.com/haskell/hackage-server/issues/488) has been edited, but these are the packages that are currently affected by this problem:

Generated with https://gist.github.com/tfausak/ed463c32638b8bd876df1a0712fc88bd.

snoyberg commented 7 years ago

@tfausak That looks like to lines up with the list from hackage-mirror-tool: https://github.com/snoyberg/hackage-mirror-tool/blob/213064e4dd5983d01b6c7daa1c7eb37bed5255fd/src/IndexShaSum.hs#L165

snoyberg commented 7 years ago

Looks like aivika-transformers-5.3.1.tar.gz needs to be added to the list.

tfausak commented 6 years ago

As @snoyberg mentioned, aivika-transformers-5.3.1 is missing its package.json. It was uploaded at 2017-10-27T04:35:54Z.

Also this just happened again with llvm-hs-5.1.1 (https://github.com/haskell/hackage-server/issues/643). That one was uploaded at 2017-12-16T14:00:07Z.

gbaz commented 6 years ago

This is resolved with https://github.com/haskell/hackage-server/issues/488

snoyberg commented 6 years ago

Can you clarify how this resolved the issue? It seems like this is a one time fix-up, but doesn't solve the underlying cause, meaning this may recur in the future. I'd be happy to hear that the underlying cause was fixed too (and frankly would be really interested in learning what the problem is).

On Mar 19, 2018 4:30 AM, "gbaz" notifications@github.com wrote:

This is resolved with haskell/hackage-server#488 https://github.com/haskell/hackage-server/issues/488

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/haskell/hackage-security/issues/183#issuecomment-374083781, or mute the thread https://github.com/notifications/unsubscribe-auth/AADBB7n70lyzteie4789-hsYCIDWLQ6bks5tfxhBgaJpZM4LthXA .

gbaz commented 6 years ago

The underlying cause was fixed in https://github.com/haskell/hackage-server/pull/644

Before, insertion of the package into the packagedb occurred and then hackage security data was added to the db, in two steps. So it was possible a thread could die or the server could fall-over between the steps. That change made sure that either both changes happened or neither change happened.

snoyberg commented 6 years ago

Awesome, thanks!

On Mon, Mar 19, 2018 at 8:12 AM, gbaz notifications@github.com wrote:

The underlying cause was fixed in haskell/hackage-server#644 https://github.com/haskell/hackage-server/pull/644

Before, insertion of the package into the packagedb occurred and then hackage security data was added to the db, in two steps. So it was possible a thread could die or the server could fall-over between the steps. That change made sure that either both changes happened or neither change happened.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/haskell/hackage-security/issues/183#issuecomment-374110991, or mute the thread https://github.com/notifications/unsubscribe-auth/AADBB94KD5QWTfREIHbj4NjpygFHEHELks5tf0w5gaJpZM4LthXA .