Plutus on Hackage/Stackage

[bjing]

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the feature you'd like

Request

Put Plutus and its dependencies on Hackage and Stackage, so that our life as a dapp developer can be a lot easier.

An ideal Plutus dev experience in my mind is that: we choose a GHC snapshot from Stackage, specify Plutus and various dependencies in our package.yml, and the project should just build, no more worrying about setting up nix both locally and in our build pipelines, and no more manual managing of release tag and commit hashes. What we get is guaranteed to build.

Benefits

• as mentioned above, reproducible builds,
• easier industry adoption. Most people I know who work with Haskell professionally use Stackage,
• we get less frustrated newcomers and retain more of them,
• easier to set up build pipelines without the involvement of Nix which is pretty complex and not widely used in the industry (from my experiences)

Describe alternatives you've considered

Not sure what would be an alternative.

Additional context / screenshots

I've personally spent roughly 10 hours to get the Plutus Starter project to build after trying building all the nix build targets possible, clearing caches and clearing built packages, building packages with different flags, switching Plutus release tags/hashes, and tweaking my global ~/.cabal/config config.

Most other people I've chatted to also had spent hours trying to get their first project built.

[bhoudebert]

Sure it would be great!

Still plutus-apps, in my opinion, is not stable, still a beta, at most prerelease, so under development (whatever terms suits better).

I do not know about Hackage policy but as a Haskeller I want to find release and not on going/beta development there, just an opinion so. Plus Hackage have clear rules for deployment/version which, for now, may not be aligned with the current state project.

If you follow a bit plutus-apps, at least this public repo, a lot of things are changing so it may take times to settle a final release (heard something about February somewhere)

For the time being I would suggest people (plutus developers/community) to update the plutus-starter for each plutus-apps pre-release, that should help a lot of people.

NB: Anyway as soon as you did align everything you know now how to upgrade. nothing solve for newcomers so.

[kk-hainq]

I don't know a solution yet but simplifying the development setup would be tremendous. For now, one would need to maintain a very scary cabal.project even if all he needs is the schema of the chain index DB and a few serialization instances to deserialize data from it. Fetching cardano-node and ouroboros-network, among others, to build from source is very time-consuming. It risks many conflicts bumping the commit hash of one thing too.

A typical dApp consists of on-chain validators (which are supposed to be simple and "limited" functions) and off-chain interfaces to the PAB. In an ideal setup, developers should only maintain plutus-apps, or perhaps plutus as well as dependencies, and do not need to worry about the underlying infrastructure of the blockchain. Building the node, consensus protocol, ledger specs (and their tests!) from source to deserialize a bunch of datums, for example, is very counter-intuitive.

To be fair, this is not just a Plutus problem but also relates to the Haskell/Nix ecosystem and other repositories in the Cardano ecosystem. For example, if we push for Stackage as suggested here, we would have to convince other Cardano projects that are Plutus dependencies to go there first. An ideal setup would require a large scale package refactoring across the board too. Complicated stuff.

[bjing]

A typical dApp consists of on-chain validators (which are supposed to be simple and "limited" functions) and off-chain interfaces to the PAB. In an ideal setup, developers should only maintain plutus-apps, or perhaps plutus as well as dependencies, and do not need to worry about the underlying infrastructure of the blockchain. Building the node, consensus protocol, ledger specs (and their tests!) from source to deserialize a bunch of datums, for example, is very counter-intuitive.

Yes that big cabal file with all the specific commit hashes is something that needs to go away (or at least hidden away from dapp devs) if we wanna see adoption.

To be fair, this is not just a Plutus problem but also relates to the Haskell/Nix ecosystem and other repositories in the Cardano ecosystem. For example, if we push for Stackage as suggested here, we would have to convince other Cardano projects that are Plutus dependencies to go there first. An ideal setup would require a large scale package refactoring across the board too. Complicated stuff.

I just don't see how having this nix + cabal setup can entice enterprises to try it out. It's a huge pain to set up both locally and in CICD pipelines. Also since we are in the early days of Plutus, it's a lot easier to make changes compared to doing it later on.

[kk-hainq]

Yes that big cabal file with all the specific commit hashes is something that needs to go away (or at least hidden away from dapp devs) if we wanna see adoption.

This is only for plutus, not plutus-apps, but will something like this work for you? https://github.com/HachiSecurity/plutus-resolver

I just don't see how having this nix + cabal setup can entice enterprises to try it out. It's a huge pain to set up both locally and in CICD pipelines. Also since we are in the early days of Plutus, it's a lot easier to make changes compared to doing it later on.

True. We started with Nix but abandoned it because it was so slow. Even raw cabal builds can freeze decent PCs building all these Cardano repositories from the source. A few commit bumps did take 1-2 hours to build even on good machines too.

I think we've gone far enough to trivially change the dependency management of Plutus or any major codebase in Cardano for the matter. It can gradually be improved though. I guess we can start by trimming unnecessary dependencies from any cabal.project among the major repositories.

For example, while experimenting with https://github.com/input-output-hk/cardano-wallet/pull/3038 I could simplify many, and delete 22 unused Nix files simply by removing unused sub dirs from cabal.project and stack.yml there.

[bhoudebert]

True. We started with Nix but abandoned it because it was so slow

Very weird ... Nix is the fastest way to compile and have a binary, really no debate. Did you use the caching mechanism like cachix, hydra etc? I am also wondering how did you fall into 2 hours of compilation. My guess is you recompile everything including external and ghc without the benefit of any mentioned remote cache which is really required when using Nix.

To give you an idea on my CI, building our custom PAB is no more than 10min (in reality more as we do a lot of extra stuff).

So please do no throw away Nix and give it some love.

Again, plutus-starter, in my mind, is the right temporary solution. Creating pre-release/release aligned with plutus framework is sufficient. People will only have to do some small copy/paste to keep it updated, this is a quick win until Plutus reach a release version which could be far from now.

[kk-hainq]

Nix is the fastest way to compile and have a binary, really no debate.

I think Nix's main strength is reproducibility, not build time. It's not hard to build something faster without Nix.

Did you use the caching mechanism like cachix, hydra etc?

Yes, but downloading and unpacking caches also contributes to building time. Raw cabal builds have caches too, it's hard to imagine a setup without one. Nix just tends to cache more for reproducibility and requires more code and maintenance to achieve so.

I am also wondering how did you fall into 2 hours of compilation. My guess is you recompile everything including external and ghc without the benefit of any mentioned remote cache which is really required when using Nix.

For several experiments, it's common (at least for us) to bump dependencies even before upstream caches are available, especially for our own forks where we fix breaking API changes among the dependencies ourselves. It's easy to get 1-2 hours of build time when one gets adventurous enough. And no, one would need a war machine to also include GHC in that time frame.

To give you an idea on my CI, building our custom PAB is no more than 10min (in reality more as we do a lot of extra stuff).

I don't know, building something like PAB with caches in 10 minutes sounds slow to me. Also, it would be nicer if we could have most CI minutes on running tests, even as far as setting up a reproducible private testnet to run tests on.

So please do no throw away Nix and give it some love.

I've been a Nix fan for almost 3 years now and set everything up with Nix at the beginning. However, it was still trivial for the others to convince me to drop it, due to build time, disk usage, macOS support, and more. For what it's worth, we're still using it in a few projects where the maintainers are happy with the setup. I believe we did indirectly fund the Nixification of several things already. Regardless, it's still an arguably unfriendly setup to many people, especially newcomers to the Cardano ecosystem.