Closed Anton-Latukha closed 3 years ago
I address this report in https://github.com/NixOS/nixpkgs/pull/91312.
At this point in time it would be a couple of days before it would be merged.
Maybe we can add an override like in https://github.com/haskell-nix/hnix/blob/master/default.nix#L190 - probably not the right way but I'm not sure how these (librarySystemDepends
, testSystemDepends
,..) are discovered for Haskell package set in NixOS but only few have such overrides defined, rest seem to be automatically populated somehow or maybe I'm just missing a source for that.
For me it seems reverse to logic, since those would be are our own efforts put to dodge our own project bugs downstream, while those efforts effectively to be put into the project itself addressing the real cause of it.
Also, I do not think the PR with such workarounds is a cdepillabout
permission bar. It would be quite hard to get his allowance to drag-in something like this.
Since we have tests and completion testing in CI that tests inside current Nixpkgs - it is OK to have it like this in Nixpkgs for a couple of month. The most important thing that we test it, and that working library is provided to the community and dependent projects.
We just need to get HNix into a great shape - that is our goal. At first, HNix should not depend on Nix, it is an oxymoron currently to have it this way. The project would be able to call itself reaching some initially stated goal when it at least is able to self-boot, before that - it is basically a library to other projects in which we not sure, because project can not self-test with own tool with Nixpkgs in the current state.
Also the Nix should be provided in the proper ./data/nix
path place for its source to be loaded by the project.
I do not know the all angles of the scope of this issue. But since I see the main cause and its resolution that is aligned with major line of the project - that is sufficient for me. If we missed something, and there is a proper good way to workaround this so it works in Nixpkgs before we address it properly, happy to listen, hear and understand it better.
Ok, yes, we missed something, and so I was wrong about the direct cause of this one.
In fact - the reason is that we use new Haskelline, so as you would now remember - we disabled the executable builds for older versions.
malahorn
s https://github.com/NixOS/nixpkgs/pull/91251 addresses it better.
But overall. We are those who know the full picture. All these are partially working solutions to the problem, the executable from the Nixpkgs still would not evaluate the Nix code properly, just like tests suite fails inside Nixpkgs due to the absence of Nix submodule files.
For me it seems reverse to logic, since those would be are our own efforts put to dodge our own project bugs downstream, while those efforts effectively to be put into the project itself addressing the real cause of it.
Also, I do not think the PR with such workarounds is a
cdepillabout
permission bar. It would be quite hard to get his allowance to drag-in something like this.
Yup that's why I want to figure out how these native (system) deps are populated automatically - will ask peti
during the Fridays stream I guess.
Since we have tests and completion testing in CI that tests inside current Nixpkgs - it is OK to have it like this in Nixpkgs for a couple of month. The most important thing that we test it, and that working library is provided to the community and dependent projects.
I agree!
We just need to get HNix into a great shape - that is our goal. At first, HNix should not depend on Nix, it is an oxymoron currently to have it this way. The project would be able to call itself reaching some initially stated goal when it at least is able to self-boot, before that - it is basically a library to other projects in which we not sure, because project can not self-test with own tool with Nixpkgs in the current state.
+1 - being able to bootstrap and build hnix with hnix will be a major step forward.
Thanks for fixing these, I'm going to fix hnix-store-remote
and update-nix-fetchgit
next.
To follow up on later comments there's also a problem with running the testsuite using systems Nix due to the requirement of the /nix/store
- I work around this in hnix-store-remote
by creating ad-hoc stores for testing in /tmp
and running tests in namespaces so they won't pollute your systems store - I will try to refactor the harness a bit and make it a separate package so it can be re-used outside of -remote
. It currently also suffers from the same issue as the original report as it needs nix-daemon
to be able to test remote protocol against it.
Well, the Nix installer had rudimentary hardcode abilities (should be able to) to switch setup from /nix/store
into other path user wants. I do not know if Nix accepts those abilities.
I doubt that it works, Nix version and installer in URL was not updated for 1-2 years or so.
Now I know that macOS blocks the creation of /*
paths, so they probably would be quick to address that ability. It is funny how much people care about complying proprietary macOS demands, but not about their own Free Software ecosystem and Linux distro relatives.
Nix store can not be polluted, it is the idea of it to be a central package build storage on the system, and it is easy to garbage collect it.
The default approach is the other way around - through configuration to setup fixed points, like to link projects and so their GHCs with lists of packages, so they detected as needed closures and not garbage collected, and everything else is cleaned-up.
I was looking into a crystal ball in https://github.com/haskell-nix/hnix/issues/641#issuecomment-648053261
Since Nix upstream made a new release or Nix installer, at last. And it seems that it uses macOS originated installer code.
Well, while they would debug the new Nix installer and rebuild all closures, it broke our CI for the nearest days while Hydra CI would do rebuilds of everything and installer release and changes would be sorted-out.
https://github.com/NixOS/nixpkgs/pull/91251 that addresses the report is merged.
Case closed.
But lets keep it open till haskell-updates
merges into master
and build arrives. It is somewhere over the weekend.
Was looking at this and it is possible to add
build-tool-depends: nix:nix
to cabal file (for tests) which results in testToolDepends = [ nix ];
in cabal2nix
output.
But it doesn't seem to change anything for me as tests are passing even without it (via nix-build
and enabled sandbox).
For the reference https://cabal.readthedocs.io/en/stable/developing-packages.html?highlight=build-tool-depends#pkg-field-build-tool-depends
Yes.
This Report with our current progress become direct discussion towards the https://github.com/haskell-nix/hnix/issues/618.
Don't forget that out default.nix
the derivation and internal Nixpkgs
machinery Haskell derivations are in different infrastructures and are different.
We need reproduce and host the Hydra pipeline and its Haskell pipeline to test for things like this, but that currently seems to be too much effort and cost for too small gain, so I simply directly monitor and PR into Nixpkgs after our releases and between them from time to time.
I got an idea, that this integration test is not hard to do, we do not need Hydra for it, so opened https://github.com/haskell-nix/hnix/issues/670
HNix really did not passes own test suite in Nixpkgs. Both I and maralorn
left dontCheck
for this reason: https://github.com/haskell-nix/hnix/issues/618#issuecomment-656297867
Yes, you found the right solution, we can see that it solves this syndrome (provide nix-instantiate
exec).
Was looking at this and it is possible to add
build-tool-depends: nix:nix
to cabal file (for tests) which results in
testToolDepends = [ nix ];
incabal2nix
output.
Can plain cabal
handle this too? My understanding was that build-tool-depends
are expected to refer to executables from Haskell packages, which build tools like cabal
and stack
will then attempt to install.
I think an alternative solution might be to move the tests that depend on external executables behind Cabal flags or env vars, so they are disabled by default. That should also allow running the test suite in Stackage (https://github.com/commercialhaskell/stackage/issues/5469).
Well, if the tests are of optional features-yes, move them back - if tests test essential features, we can not put them as optional, obviously.
I think we can form executable and split the tests.
Then, I know John told that there is not too much work to untie the dependence on the Nix.
Better to untie us from Nix before we would get in all sorts of software stores. Because if we dodge the Nix dependency too long - we would get constant situations popping-up because of the need to drag it around all the time.
I surprised we did not create the Please, untie HNix from Nix submodule before.
Was looking at this and it is possible to add
build-tool-depends: nix:nix
to cabal file (for tests) which results in
testToolDepends = [ nix ];
incabal2nix
output.Can plain
cabal
handle this too? My understanding was thatbuild-tool-depends
are expected to refer to executables from Haskell packages, which build tools likecabal
andstack
will then attempt to install.
Yes since 2.0: https://cabal.readthedocs.io/en/stable/developing-packages.html?highlight=build-tool-depends#pkg-field-build-tool-depends
I'm not sure how well this works with stack
though.
I think an alternative solution might be to move the tests that depend on external executables behind Cabal flags or env vars, so they are disabled by default. That should also allow running the test suite in Stackage (commercialhaskell/stackage#5469).
The use of nix-instantiate
and nix-store --add
is currently a hack until hnix-store
is used (like in #554) so we shouldn't spend too much time on it as it will become a non-issue soon (it actually moves to hnix-store
which needs nix-daemon
for its tests but the same fix(es) can be used.)
Was looking at this and it is possible to add
build-tool-depends: nix:nix
to cabal file (for tests) which results in
testToolDepends = [ nix ];
incabal2nix
output.Can plain
cabal
handle this too? My understanding was thatbuild-tool-depends
are expected to refer to executables from Haskell packages, which build tools likecabal
andstack
will then attempt to install.Yes since 2.0: https://cabal.readthedocs.io/en/stable/developing-packages.html?highlight=build-tool-depends#pkg-field-build-tool-depends
Hmm. I tried adding build-tool-depends: nix:nix
and got the following error:
$ cabal test --enable-tests
Resolving dependencies...
cabal: Could not resolve dependencies:
[__0] trying: hnix-0.9.0 (user goal)
[__1] rejecting: hnix:!test (constraint from config file, command line flag,
or user target requires opposite flag selection)
[__1] trying: hnix:*test
[__2] unknown package: hnix:nix:exe.nix (dependency of hnix *test)
[__2] fail (backjumping, conflict set: hnix, hnix:nix:exe.nix, hnix:test)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: hnix, hnix:test, hnix:nix:exe.nix
I think cabal
gets confused because nix
doesn't refer to a known Haskell package.
I think an alternative solution might be to move the tests that depend on external executables behind Cabal flags or env vars, so they are disabled by default. That should also allow running the test suite in Stackage (commercialhaskell/stackage#5469).
The use of
nix-instantiate
andnix-store --add
is currently a hack untilhnix-store
is used (like in #554) so we shouldn't spend too much time on it as it will become a non-issue soon (it actually moves tohnix-store
which needsnix-daemon
for its tests but the same fix(es) can be used.)
Oh! That's good to know. Do we have an issue that describes what's planned, or could you possibly file one?
I believe this was resolved..?
It was / is :)
Monitored for package,
0.9.0
hit the Nixpkgs, and in broken state: https://hydra.nixos.org/job/nixpkgs/trunk/haskellPackages.hnix.x86_64-linux#tabs-statusAt my last HNix-related PR into Nixpkgs, I did not wanted
generateOptparseApplicativeCompletion
to be enabled in the Nixpkgs (because we were not testing for completion support in our CI back then), but for me and maintainer there it seemed as it works.In fact, after the merge, it is a fact that it does not work: https://hydra.nixos.org/build/122819514/nixlog/1
The reason is just, and in fact is the same reason why project can not pass its own tests in Nixpkgs:
Because in Nixpkgs there is no Nix submodule. Executable would not be built until the project would become independent from Nix submodule use, so currently it would be just a library in Nixpkgs.