search

haskell / cabal

Official upstream development repository for Cabal and cabal-install
https://haskell.org/cabal
Other
1.61k stars 691 forks source link

Distinguish package index location from packages #9352

Open j-mie6 opened 10 months ago

j-mie6 commented 10 months ago

Describe the feature request At the moment, it is possible to configure where cabal puts its store of packages/index files for Hackage/other providers. It would be nice, however, if these indexes could be stored in a separate location from the packages themselves.

Additional context The package index files for Hackage are starting to reach the 1GB mark. In a scenario where we have many many users of cabal floating around, this adds up rapidly if everyone has the same copy of the package index. This, for instance, is the case for Universities, which may have approaching 1TB of Hackage indices spread across all their students! In practice, the cabal update command, does not need to be run by every student, though each student may have their own unique package requirements; as such, it would be nice if they could have packages cached in their ~/.cabal/packages folder, but we can keep the package index itself in some shared read-only directory which staff can refresh regularly -- with no risk of race-conditions from concurrent cabal updates performed by different users, or providing global write access to a folder.

If I understand correctly, the ability to set the directory path to store package indexes separate from the packages would resolve this problem. By default, these two paths could be set to be the same thing, of course.

andreabedini commented 10 months ago

Just leaving a note in case someone wants to pick this up.

The value of cache-dir gets passed to withRepoContext assharedCacheDir and attached to each Repo. The function packageFile in FetchUtils then decides the location of each package.

This might get a bit tricky because in case of RepoLocalNoIndex I think we just skip the download step so what packageFile returns becomes where the package is in the local repo, rather than where it has to be saved. I might be wrong though.

Mikolaj commented 10 months ago

@andreabedini: I gather, this is not a newcomer ticket (otherwise, I'd set one of the newcomer labels)?

andreabedini commented 10 months ago

@Mikolaj I think it can be a newcomer ticket. I will add the appropriate labels (unless there any opposition to the idea itself).

Mikolaj commented 10 months ago

Oh, that's even better. I guess I got scared by your "might get a bit tricky" and the general feel the design is not finalised yet (asking a newcomer to come up with the design gradually, via code-diving and experimentation, might be a high call, but OTOH, the functionality sounds pretty straightforward).