Use of source-repository-packages/cabal.project in CI

I recently committed some fixes to another branch in an attempt to fix the CI for bifunctors. I haven't pushed to master yet, however, because it does a couple of opinionated things:

Instead of using submodules, like in the master branch of distributive, it vendors in the comonad library using a source-repository-package in the cabal.project file. See here.
Speaking of which, I'm using a cabal.project file. bifunctors used to have one, but @ekmett appears to have removed it in bc5d8e88f7a944aff598889e39259e111b33f0f3 for reasons that aren't clear to me.

In any case, let me briefly explain why I favor (1) and (2) above:

Using source-repository-package allows vendored libraries to be saved in cabal's store (using cabal-instal-3.4 or later), which means that you don't have to constantly rebuild them. More critically, it also means that vendored libraries can be cached in CI, so it will save quite a bit of CI time in the long run.
Having a cabal.project file makes it so that I can check out the bifunctors repo within the context of another project with a cabal.project file and not have the latter cabal.project file clobber it. This is important for my workflows, as I manage bifunctors within another repo.

@ekmett: I'm raising this issue because I don't want to step on your toes when you're developing your libraries. Let me know your preferences and I can find a way to make both of us happy.

Long term I think what you want to do here is very much the right thing to do.

I've been wrestling with trying to find a workflow to juggle several packages at once with consistent versioning, during a stint while I'm much more actively shuffling modules between them than usual.

The submodule with no cabal.project approach means I can just dip into each sub-dir, spin up ghcid, fix something, and then see the impact on all the downstream packages I'm working on at the same time in the outer 'distributive' project, committing the submodule then locks it in making it easy to share with Aaron Vargo who has been working with me during this bit. Unless, and I've never tested it, I can do a source-repository link that doesn't tag the hash code and just names the branch?

Moving to vendored links as I currently use them means that I need to commit the sub-project, go grab the version number, punch it into the source-repository package link for each of the downstream packages' cabal files that have their own cabal.project files and kind of blows up the work to O(n^2).

I temporarily removed the cabal.project file because it actively gets in the way of the above dip in, ghcid, work, dip out workflow, as then it selects the contents of the local cabal.project file, so I have to run ghcid from outside the project dir manually with a massive pile of flags to tell it what library I want and to keep all of its no-code-gen optimizations intact or risk screwing up and building with inconsistent versions.

I'm about to do ~50 of these, so n here is going to get pretty large. I'm inclined to slowly do this transform once I have stabilized what is moving in/out, but knowingly break this in the interim.

-Edward

On Wed, Apr 7, 2021 at 5:29 AM Ryan Scott @.***> wrote:

I recently committed some fixes to another branch https://github.com/ekmett/bifunctors/tree/fix-ci in an attempt to fix the CI for bifunctors. I haven't pushed to master yet, however, because it does a couple of opinionated things:

Instead of using submodules, like in the master branch of distributive, it vendors in the comonad library using a source-repository-package in the cabal.project file. See here https://github.com/ekmett/bifunctors/commit/e023b36fdae0a5f6973a316acc4a5d60a939c70c#diff-b8ed06757045fac949c89f2139a862498ad2b6d1f82c61a31e7c91d6cf0eaa70R3 .

Speaking of which, I'm using a cabal.project file. bifunctors used to have one, but @ekmett https://github.com/ekmett appears to have removed it in bc5d8e8 https://github.com/ekmett/bifunctors/commit/bc5d8e88f7a944aff598889e39259e111b33f0f3 for reasons that aren't clear to me.

In any case, let me briefly explain why I favor (1) and (2) above:

Using source-repository-package allows vendored libraries to be saved in cabal's store (using cabal-instal-3.4 or later), which means that you don't have to constantly rebuild them. More critically, it also means that vendored libraries can be cached in CI, so it will save quite a bit of CI time in the long run.

Having a cabal.project file makes it so that I can check out the bifunctors repo within the context of another project with a cabal.project file and not have the latter cabal.project file clobber it. This is important for my workflows, as I manage bifunctors within another repo https://github.com/RyanGlScott/ci-maintenance.

@ekmett https://github.com/ekmett: I'm raising this issue because I don't want to step on your toes when you're developing your libraries. Let me know your preferences and I can find a way to make both of us happy.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ekmett/bifunctors/issues/91, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKMELPDZMMGREXRRICHRTTHRFZLANCNFSM42QWHEJQ .

And of course right after I send that, I found that the source-repository-package section of the cabal.project documentation is incomplete and there is a branch option.

If that works with git I'd be fine with a switch back to cabal.project files with source-repository-package links.

On Wed, Apr 7, 2021 at 8:27 AM Edward Kmett @.***> wrote:

Long term I think what you want to do here is very much the right thing to do.

I've been wrestling with trying to find a workflow to juggle several packages at once with consistent versioning, during a stint while I'm much more actively shuffling modules between them than usual.

The submodule with no cabal.project approach means I can just dip into each sub-dir, spin up ghcid, fix something, and then see the impact on all the downstream packages I'm working on at the same time in the outer 'distributive' project, committing the submodule then locks it in making it easy to share with Aaron Vargo who has been working with me during this bit. Unless, and I've never tested it, I can do a source-repository link that doesn't tag the hash code and just names the branch?

Moving to vendored links as I currently use them means that I need to commit the sub-project, go grab the version number, punch it into the source-repository package link for each of the downstream packages' cabal files that have their own cabal.project files and kind of blows up the work to O(n^2).

I temporarily removed the cabal.project file because it actively gets in the way of the above dip in, ghcid, work, dip out workflow, as then it selects the contents of the local cabal.project file, so I have to run ghcid from outside the project dir manually with a massive pile of flags to tell it what library I want and to keep all of its no-code-gen optimizations intact or risk screwing up and building with inconsistent versions.

I'm about to do ~50 of these, so n here is going to get pretty large. I'm inclined to slowly do this transform once I have stabilized what is moving in/out, but knowingly break this in the interim.

-Edward

On Wed, Apr 7, 2021 at 5:29 AM Ryan Scott @.***> wrote:

I recently committed some fixes to another branch https://github.com/ekmett/bifunctors/tree/fix-ci in an attempt to fix the CI for bifunctors. I haven't pushed to master yet, however, because it does a couple of opinionated things:

Instead of using submodules, like in the master branch of distributive, it vendors in the comonad library using a source-repository-package in the cabal.project file. See here https://github.com/ekmett/bifunctors/commit/e023b36fdae0a5f6973a316acc4a5d60a939c70c#diff-b8ed06757045fac949c89f2139a862498ad2b6d1f82c61a31e7c91d6cf0eaa70R3 .

Speaking of which, I'm using a cabal.project file. bifunctors used to have one, but @ekmett https://github.com/ekmett appears to have removed it in bc5d8e8 https://github.com/ekmett/bifunctors/commit/bc5d8e88f7a944aff598889e39259e111b33f0f3 for reasons that aren't clear to me.

In any case, let me briefly explain why I favor (1) and (2) above:

Using source-repository-package allows vendored libraries to be saved in cabal's store (using cabal-instal-3.4 or later), which means that you don't have to constantly rebuild them. More critically, it also means that vendored libraries can be cached in CI, so it will save quite a bit of CI time in the long run.

Having a cabal.project file makes it so that I can check out the bifunctors repo within the context of another project with a cabal.project file and not have the latter cabal.project file clobber it. This is important for my workflows, as I manage bifunctors within another repo https://github.com/RyanGlScott/ci-maintenance.

@ekmett https://github.com/ekmett: I'm raising this issue because I don't want to step on your toes when you're developing your libraries. Let me know your preferences and I can find a way to make both of us happy.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ekmett/bifunctors/issues/91, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKMELPDZMMGREXRRICHRTTHRFZLANCNFSM42QWHEJQ .

Trying it with cabal.project using branch based source links seems to work and should potentially fix ci without undue tag # maintenance.

Thanks for the very thorough response, @ekmett! I also wish that the ergonomics of cabal.project allowed switching between source-repository-packages and submodules more easily. That being said, if using tag: <branch-name> works for you, then perhaps that's the best approach for now.

Using the bifunctors repo as a particular example, should I vendor in comonad's master branch as its tag?

I've been switching each from master to main for consistency. In theory bifunctors should already be flipped over.

Let me know if I fudged it up.

One concern I'm noting is that this workflow seems pretty brittle. When I go to do rebuilds after changes to the upstream branch, cabal gets confused, and fails to pull the updated branch correctly, and because each of these is getting a mini cabal.project file, i'm getting lots of dist-newstyle folders, meaning lots of redundant build work in my dev cycle. Neither of these is great for me, so I may recant using this style, but I'll at least try it for a few days first.

If it is untenable in practice, I could maybe see about using some tooling to handle the source-repository link management, by doing something like generating cabal.project.locals via tooling or something. There should be something in this corner of the design space that works without breaking CI at any rate.

-Edward

On Wed, Apr 7, 2021 at 3:44 PM Ryan Scott @.***> wrote:

Thanks for the very thorough response, @ekmett https://github.com/ekmett! I also wish that the ergonomics of cabal.project allowed switching between source-repository-packages and submodules more easily. That being said, if using tag: works for you, then perhaps that's the best approach for now.

Using the bifunctors repo as a particular example, should I vendor in comonad's master branch as its tag?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ekmett/bifunctors/issues/91#issuecomment-815309144, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKMEOU724SZ43TY4NWORLTHTN53ANCNFSM42QWHEJQ .

One concern I'm noting is that this workflow seems pretty brittle. When I go to do rebuilds after changes to the upstream branch, cabal gets confused, and fails to pull the updated branch correctly, and because each of these is getting a mini cabal.project file, i'm getting lots of dist-newstyle folders, meaning lots of redundant build work in my dev cycle. Neither of these is great for me, so I may recant using this style, but I'll at least try it for a few days first.

Good to know. If it is untenable, I'm not opposed to using submodules. The only catch is that I don't think haskell-ci's GitHub Actions supports automatically cloning submodules upon checkout at the moment, so I'd need to implement that before going further.

If it is untenable in practice, I could maybe see about using some tooling to handle the source-repository link management, by doing something like generating cabal.project.locals via tooling or something. There should be something in this corner of the design space that works without breaking CI at any rate.

That is an interesting idea. We could certainly check in two cabal.project files, one of which using submodules and the other using source-repository-packages, and we could use them as is appropriate for the task at hand.

Ah, I didn't notice that you already switched to using a source-repository-package in main. In that case, I went ahead and pushed a commit leveraging that to fix CI in 269d156f47ce9896d801572d1fc3a286fbf92c4e. Do note that if you want to switch to submodules, I have just landed a patch in haskell-ci which permits this, so it would be straightforward to change how we vendor in other libraries.

ekmett / bifunctors

Use of source-repository-packages/cabal.project in CI #91