search

fsprojects / Paket

A dependency manager for .NET with support for NuGet packages and Git repositories.
https://fsprojects.github.io/Paket/
MIT License
2.02k stars 524 forks source link

make a group inherit the dependencies of another group #2608

Closed pchalamet closed 1 year ago

pchalamet commented 7 years ago

Just an idea that could make a lot of sense with groups.

Paket does support groups - but as far as I understand this - they have no relationships for the moment. For example:

group A
    source ...
    nuget FSharp.Core

group B
    source ...
    nuget FsUnit

implies A and B can have different version of FSharp.Core. This could be a problem for example for 2 projects (each using a specific Paket group) with dependency on a base library (with specific nuget). Using groups is complicated because base nugets could end up being different for base library.

This could be interesting to support group dependencies to simplify groups management. I mean, groups could then be used as prerequisites for other groups and ensuring other groups are strictly compatible with inherited dependencies/versions.

For example:

group A
    source ...
    nuget FSharp.Core

group C
    source ...
    nuget XXX

group B <- A, C
    source ...
    nuget FsUnit

would force B to be compatible with versions resolved within group A and C.

What do you think about this ? Could that be interesting for Paket ? Thanks.

forki commented 7 years ago

We introduced groups as resolution boundaries so that you can have different versions. So in some sense it's the original feature. Your proposal sounds interesting. Do you have real use cases for this? It would introduce a bit of complexity and I'm not sure yet if that's justified.

Am 11.08.2017 10:06 nachm. schrieb "Pierre Chalamet" < notifications@github.com>:

Just an idea that could make a lot of sense with groups.

Paket does support groups - but as far as I understand this - they have no relationships for the moment. For example:

group A source ... nuget FSharp.Core

group B source ... nuget FsUnit

implies A and B can have different version of FSharp.Core. This could be a problem for example for 2 projects (each using a specific Paket group) with dependency on a base library (with specific nuget). Using groups is complicated because base nugets could end up being different for base library.

This could be interesting to support group dependencies to simplify groups management. I mean, groups could then be used as prerequisites for other groups and ensuring other groups are strictly compatible with inherited dependencies/versions.

For example:

group A source ... nuget FSharp.Core

group C source ... nuget XXX

group B <- A, C source ... nuget FsUnit

would force B to be compatible with versions resolved within group A and C.

What do you think about this ? Could that be interesting for Paket ? Thanks.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/fsprojects/Paket/issues/2608, or mute the thread https://github.com/notifications/unsubscribe-auth/AADgNFXcYpxvGwGaO8JxRzJdMCgMx7wEks5sXLQxgaJpZM4O1FCL .

pchalamet commented 7 years ago

Yes sure.

We have a microservices architecture with some common assemblies. Services can freely use whatever nugets they want (until it's compatible with the one in common assemblies). Everything is built from sources only even implementations span on multiple repositories (but applies equally in single repository).

Managing nugets for common assemblies is easy - but managing individually each service is complicated because we have to ensure nugets referenced by services are compatible with the one used in common assemblies.

That's where dependencies for groups could be interesting. Common assemblies would specify nugets for common assemblies and services could have their own group taking dependency on common group. Services could upgrade at their own rhythm and would not require to be kept strictly in sync.

I do agree upgrading common group implies upgrading dependent groups. But the reverse is not true and services could be upgraded independently. I imagine this could be just new constraints when solving version for a group (like framework constraint).

Hope it's clearer! Thanks.

forki commented 7 years ago

So if a group B depends on group A what would we do in the resolver?

Add all packages from group A as dependencies with pinned versions?

Am 12.08.2017 9:09 vorm. schrieb "Pierre Chalamet" <notifications@github.com

:

Yes sure.

We have a microservices architecture with some common assemblies. Services can freely use whatever nugets they want (until it's compatible with the one in common assemblies). Everything is built from sources only even implementations span on multiple repositories (but applies equally in single repository).

Managing nugets for common assemblies is easy - but managing individually each service is complicated because we have to ensure nugets referenced by services are compatible with the one used in common assemblies.

That's where dependencies for groups could be interesting. Common assemblies would specify nugets for common assemblies and services could have their own group taking dependency on common group. Services could upgrade at their own rhythm and would not require to be kept strictly in sync.

I do agree upgrading common group implies upgrading dependent groups. But the reverse is not true and services could be upgraded independently. I imagine this could be just new constraints when solving version for a group (like framework constraint).

Hope it's clearer! Thanks.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/fsprojects/Paket/issues/2608#issuecomment-321962877, or mute the thread https://github.com/notifications/unsubscribe-auth/AADgNItF1WPiUOI805Z0khybT5MNXf3Oks5sXU-6gaJpZM4O1FCL .

pchalamet commented 7 years ago

Basically, yes.

matthid commented 7 years ago

So if I understand correctly group B would be forced to use FSharp.Core from group A?

What about transitives? Can group B upgrade transitives as long as they are compatible with direct deps of group A?

@forki If transitives are locked this is quite similar to what we do with the runtime resolution. I'm pretty sure we could unify those.

matthid commented 7 years ago

Off topic: It is a bit spurious to have microservices but make them depend on a common set of packages. Shouldn't they be independent of each other...

forki commented 7 years ago

Yeah I think if we add the whole resolution from A as pins to B then this could work easily.

So IF we want to do this then I think biggest issues are:

Resolution part is probably the easier part

Am 12.08.2017 9:17 vorm. schrieb "Matthias Dittrich" < notifications@github.com>:

So if I understand correctly group B would be forced to use FSharp.Core from group A?

What about transitives? Can group B upgrade transitives as long as they are compatible with direct deps of group A?

@forki https://github.com/forki If transitives are locked this is quite similar to what we do with the runtime resolution. I'm pretty sure we could unify those.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fsprojects/Paket/issues/2608#issuecomment-321963205, or mute the thread https://github.com/notifications/unsubscribe-auth/AADgNFfmO3tMt0AmrJ5POBC0NQ12diMZks5sXVGlgaJpZM4O1FCL .

pchalamet commented 7 years ago

Not sure at all about upgrading inherited dependencies - idea is probably to pin whatever is inherited from other groups and then resolve for the given group.

For the OT comment - yeah but that's life and copy/paste is not the best way to handle this.

pchalamet commented 7 years ago

framework like syntax could be used. "Inherits" is probably not that good but it's the idea.

group C inherits: A, B

matthid commented 7 years ago

Also I'm not sure about the Main group, we probably should not allow it to participate in this feature (neither actively nor passively) as the name of the main group is an implementation detail..

matthid commented 7 years ago

Also we should only allow to reference groups defined above to prevent cycles?

pchalamet commented 7 years ago

Main group could still be referenced by name "Main" explicitly if required. Probably there are already checks in place to avoid naming a group like this if the default one is implicitly created. There is probably no good reason to exclude it but it's maybe part of an unknown compatibility to me.

For the order, yes this would help with implementation tremendously (not necessarily a good reason) and ensure clear readability at the same time. Like f#.

forki commented 7 years ago

Yes Main group is explicitly used in many places. Changing the name of the main group would be really breaking for many users. So I think it's fine.

Am 12.08.2017 16:40 schrieb "Pierre Chalamet" notifications@github.com:

Main group could still be referenced by name "Main" explicitly if required. Probably there are already checks in place to avoid naming a group like this if the default one is implicitly created. There is probably no good reason to exclude it but it's maybe part of an unknown compatibility to me.

For the order, yes this would help with implementation tremendously (not necessarily a good reason) and ensure clear readability at the same time. Like f#.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fsprojects/Paket/issues/2608#issuecomment-321985424, or mute the thread https://github.com/notifications/unsubscribe-auth/AADgNOS_vobwdm-sqvMLq-q_qQ9bXeAwks5sXblMgaJpZM4O1FCL .

pmiossec commented 7 years ago

I think that this proposal is interesting. Correct me if I'm wrong but this feature could be useful to improve performance in a specific case that we have at our company.

Most of our nuget packages are provided from one source (most of the time nuget.org). But we have (depending on the project), around 6 to 10 packages provided by as much as sources (internal sources) .

So we end up with a lot of sources (which are also slower than the main source) queried for each dependencies.

With this feature, we will be able to create some small groups for these specific dependencies with just their sources. And with the main group containing only the main source.

This preventing to query all the internal slow sources for all the packages provided by the main source.

matthid commented 7 years ago

Most of our nuget packages are provided from one source (most of the time nuget.org). But we have (depending on the project), around 6 to 10 packages provided by as much as sources (internal sources) .

So we end up with a lot of sources (which are also slower than the main source) queried for each dependencies.

With this feature, we will be able to create some small groups for these specific dependencies with just their sources. And with the main group containing only the main source.

@pmiossec I don't think this is how this feature should be used. In fact we should just fix any existing performance problems...

Yes Main group is explicitly used in many places.

@forki But not jet by the user itself in the packages-files? I still think it's a implementation detail. People should not use it.

forki commented 7 years ago

Yes the main groups are explicitly used in dependencies files and references files by many users.

Am 13.08.2017 10:38 nachm. schrieb "Matthias Dittrich" < notifications@github.com>:

Most of our nuget packages are provided from one source (most of the time nuget.org). But we have (depending on the project), around 6 to 10 packages provided by as much as sources (internal sources) .

So we end up with a lot of sources (which are also slower than the main source) queried for each dependencies.

With this feature, we will be able to create some small groups for these specific dependencies with just their sources. And with the main group containing only the main source.

@pmiossec https://github.com/pmiossec I don't think this is how this feature should be used. In fact we should just fix any existing performance problems...

Yes Main group is explicitly used in many places.

@forki https://github.com/forki But not jet by the user itself in the packages-files? I still think it's a implementation detail. People should not use it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fsprojects/Paket/issues/2608#issuecomment-322066037, or mute the thread https://github.com/notifications/unsubscribe-auth/AADgNEt1CMIMkFB2xYwjlH-IeXuGLuQVks5sX17dgaJpZM4O1FCL .

pchalamet commented 7 years ago

Ok to sum up. We discussed:

Last point raises interesting questions btw: this could lead to conflicts if 2 groups brings same artifacts but with different versions. This shall end as a resolver error probably. What's your take on this?

Other point: groups/nugets/files are stored to their own packages/ folder on file system. Shall all inherited groups be included as well in folder?

forki commented 7 years ago

Pretty sure this will already end up being resolver error.

Am 14.08.2017 10:55 vorm. schrieb "Pierre Chalamet" < notifications@github.com>:

Ok to sum up. We discussed:

  • dependencies between groups could be implemented
  • syntax has to be defined to reference others groups (proposed is "inherits" but has to be challenged)
  • Main group can be referenced by name "Main"
  • a group can only reference groups which have been defined previously in Paket.dependencies
  • "inherited" group dependencies from resolver point of view is just a bunch of pinned versions.

Last point raises interesting questions btw: this could lead to conflicts if 2 groups brings same artifacts but with different versions. This shall end as a resolver error probably. What's your take on this?

Other point: groups/nugets/files are stored to their own packages/ folder on file system. Shall all inherited groups be included as well in folder?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fsprojects/Paket/issues/2608#issuecomment-322134682, or mute the thread https://github.com/notifications/unsubscribe-auth/AADgNO9kx0fXfBZ5TYZdZ4sdj4QhKIBkks5sYAtqgaJpZM4O1FCL .

forki commented 7 years ago

Btw the github deps feature is very similar. It consumes remote paket.dependencies files and merges them into your local groups.

Am 14.08.2017 11:04 vorm. schrieb "Steffen Forkmann" sforkmann@gmail.com:

Pretty sure this will already end up being resolver error.

Am 14.08.2017 10:55 vorm. schrieb "Pierre Chalamet" < notifications@github.com>:

Ok to sum up. We discussed:

  • dependencies between groups could be implemented
  • syntax has to be defined to reference others groups (proposed is "inherits" but has to be challenged)
  • Main group can be referenced by name "Main"
  • a group can only reference groups which have been defined previously in Paket.dependencies
  • "inherited" group dependencies from resolver point of view is just a bunch of pinned versions.

Last point raises interesting questions btw: this could lead to conflicts if 2 groups brings same artifacts but with different versions. This shall end as a resolver error probably. What's your take on this?

Other point: groups/nugets/files are stored to their own packages/ folder on file system. Shall all inherited groups be included as well in folder?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fsprojects/Paket/issues/2608#issuecomment-322134682, or mute the thread https://github.com/notifications/unsubscribe-auth/AADgNO9kx0fXfBZ5TYZdZ4sdj4QhKIBkks5sYAtqgaJpZM4O1FCL .

matthid commented 7 years ago

Other point: groups/nugets/files are stored to their own packages/ folder on file system. Shall all inherited groups be included as well in folder?

This is indeed a good question considering we are trying to slowly get rid of the packages folder. We could either say yes they are duplicated until we figured it out (easier to implement). Or we say don't duplicate and do some part of the "flexible" packages location as part of this feature...

I'm pretty sure part of this logic can be reused later.

pchalamet commented 7 years ago

So are you ok to go forward with this proposition? Is there something to investigate before going on ? Do you see value implementing this?

If we agree on this, is there someone looking at it or should I try to look at codebase and see how to implement this ?