search

haskell-works / cabal-cache

CI assistant
BSD 3-Clause "New" or "Revised" License
44 stars 11 forks source link

How to ensure proprietary packages are not pushed to a possibly public S3 bucket? #108

Open hasufell opened 4 years ago

hasufell commented 4 years ago

Afais there is no explicit check or option for this. Implicitly it seems that cabal-cache ignores all local packages, which, IMO, might actually be a bug that has gone unnoticed?

https://github.com/haskell-works/cabal-cache/blob/1241aaeabab514df473cfa50d1e47baf5db16d4a/src/App/Commands/SyncToArchive.hs#L125

This will return False for local packages, because their storepath points to something like ghc-8.6.5/foo-inplace-3.0.conf (but that exists inside dist-newstyle). The function then appends this path to the store path (not dist-newstyle) and there it doesn't exist.

So two questions:

  1. Is this expected or a bug?
  2. How to ensure only hackage available deps are ever pushed?

Edit: afais, Package.style is set to "local" for local packages: https://github.com/haskell-works/cabal-cache/blob/master/src/HaskellWorks/CabalCache/Types.hs#L27 Maybe this should be used for distinguishing?

AlexeyRaga commented 4 years ago

As far as I remember we do ignore local packages and do not archive them intentionally. The intent of cabal-cache is to preserve/reuse the "global" cache, that is usually in ~/.cabal/store. dist-newstyle is considered a "project business", when you do cabal v2-clean it is all gone, etc.

hasufell commented 4 years ago

cabal-3.4.0.0 won't treat things defined in source-repository-package as local anymore, so this has to be re-evaluated and tested. (ghcup has cabal-3.4.0.0-rc1 if you want to try it out)

newhoggy commented 4 years ago

I haven't had much experience with private packages.

Do you know of a good way to tell private and public packages apart.

The local/hackage package distinction is a not good way to tell if a package should be excluded from the cache because it's fairly common to local packages public projects too.

newhoggy commented 4 years ago

We handle proprietary code by using a private S3 bucket and avoid syncing to public buckets from private builds.

It is possible to sync down from multiple buckets so you can sync down both public and private S3 buckets in the same build.

Admittedly there is no way to prevent accidental syncing, so it would be good to have a solution for that.