Open Mic92 opened 4 years ago
Would agree to rewrite node2nix to use async/await builtins instead of slasp?
It is not necessarily blocking, but it currently only performs one requests at the time and was not made to do any downloads a parallel.
It could be done with promises/async/await, but there also other ways.
I have not done any experiments yet, but having the ability to have multiple downloads in parallel could be beneficial to the generation speed.
Is the api client that only performs one request at the time? The rest looks like it should be asynchronous.
nixpkgs generate.sh
takes quite a while, and seems like it could potentially benefit from this. Do you have a particular direction in mind how you would like to see this implemented?
@raboof Although having parallel downloads might shave off a bit of processing time, I believe what we really want is the ability to partially regenerate the expression for only the package that needs to be added or changed.
I'm currently investigating options to make that possible.
Not this feature is not useful, but what I expect to happen is that if we make the regeneration process faster, people will add more packages eventually nullifying its effect.
With partial regenerations, we can have faster updates, and better commits on a per package basis.
@raboof Although having parallel downloads might shave off a bit of processing time, I believe what we really want is the ability to partially regenerate the expression for only the package that needs to be added or changed.
That would be really great. Also to enable backports
With partial regenerations, we can have faster updates, and better commits on a per package basis.
That sounds great. I didn't see an issue for this yet, so I filed #192.
I tried to replace slasp.fromEach
with a parallel version. (See repo)
It has speed up fetching about 2x in good network environment but makes the output non-deterministic, which are differences like A-0.2, B // { A-0.1 }, C
vs A-0.1, B, C // { A-0.2 }
while A
is a common transparent dependency. This is related to the order of package traversal, since we currently choose the first version seen to be in outer layer and mark other version as override.
Related code:
https://github.com/svanderburg/node2nix/blob/bfd517c1774ddb4ae1a459b8d96f1bc653a72bd1/lib/Package.js#L148
Is there a way to force newest versions instead of the first version in the outer layer, assuming that newer versions are better? That should make it deterministic as well.
@Mic92
Is there a way to force newest versions instead of the first version in the outer layer, assuming that newer versions are better?
I think the order affects not only the output layout, but also the version selection result.
Consider that A
requires C@^1.0
and B
requires C@=1.5
, assuming C
has version 1.5 and 1.9. Resolving order A, B
results in two package, C@1.9
for A and C@1.5
for B, while resolving order B, A
results in a single package C@1.5
for both A and B.
The version resolution algorithm seems to be quite naive now.
@Mic92
Is there a way to force newest versions instead of the first version in the outer layer, assuming that newer versions are better?
I think the order affects not only the output layout, but also the version selection result. Consider that
A
requiresC@^1.0
andB
requiresC@=1.5
, assumingC
has version 1.5 and 1.9. Resolving orderA, B
results in two package,C@1.9
for A andC@1.5
for B, while resolving orderB, A
results in a single packageC@1.5
for both A and B.The version resolution algorithm seems to be quite naive now.
Also there determinism could be achieved by going for the newest possible version for every constraint, but I am not sure if this is what the users want.
@Mic92 Always picking the newest version is not going to work -- the fact that this does not always happen is intentional. If we would choose to alter this behaviour, then compatibility with certain packages is most likely going to break.
Sadly, NPM always tries to "reuse" dependencies that fit within a certain version range in a node_modules/
folders that reside in the parent directories. For certain packages, this means that a version range will resolve to an older version than the latest. I don't like this behaviour either, but it is how NPM works and instrumental to ensure compatibility.
I had several commercial projects that would fail to deploy if I would not replicate this very odd and illogical behaviour. To discover this, I did quite a few intensive comparisons between node2nix and the vanilla NPM.
Furthermore, this is also NPM's means to break out of cyclic dependencies. If we always resolve to the latest version, then certain deployments will remain stuck in an infinite loop.
@oxalica Thanks for trying to optimize this. Although we have to give it some more thought, it is good to hear that there is a speed up possible.
Sorry to bump this old issue , but has there been any advancements in solving this issue ?
It seems that node2nix currently performs blocking requests:
Maybe you can explain why this happens as you also seem to maintain the asynchronous abstraction behind it: https://github.com/svanderburg/slasp