Allow libraries to reference other libraries

[Game4Freak]

For?

S&Box

What can't you do?

Currently, we are unable to reference other libraries.

Dependency and version management can very easily get out of hand and I get why it was not done. You mostly use references for having less duplicate code or for compatibility. While less duplicate code is more of a optional thing, compatibility is simply impossible without references (at least I haven't found a workaround yet).

Personally, I like to create libraries for others to use, but not just simple tools. I prefer creating generic and modular frameworks with compatibility in mind but without references between libraries, that is very difficult / almost impossible to achieve.

How would you like it to work?

The classic way would be to download the referenced libraries as well, but this will result in chaos and version management doesn't make it easier (just think about the good old node_modules folder). Libraries should be a simple way to add prebuild functionality to your project that you can modify if needed. So the classic way would not really fit this approach.

This is probably not the perfect solution, and it only takes the code into account, but it should be a starting point.

To keep libraries simple to use, some more effort from library dev is probably required, but this is fine as it should be more of an advanced thing anyway. So my idea is that libraries should be kind of self-sufficient even if you have references (sounds kind of dumb, but you will get it).

So, if you reference another library, you will need to add the code that you use from that other library to your own library. This way, your library can be used without the need of the other library.

But why would you need referenced libraries if the code is already in the library itself?

Well, this will only work if you have one library that references the other library. If you have multiple libraries that reference the same library, you will need to have the referenced library else there will be conflicts.

Short recap

General

• Your library contains the code that you would use from another library.

  In Projects

• If only 1 library references another library, the referenced library doesn't need to be added.
• If multiple libraries reference another library, the referenced library needs to be added and the library code contained in the referencing libraries needs to be ignored.

Example

flowchart LR

B[Library 2] -->|references| A[Library 1]
C[Library 4] -->|references| A[Library 1]
C[Library 4] -->|contains| D[Library 3]

MyGame
│
└───┬─Libraries
    │
    ├───┬─Library1
    │   ├───Code
    │   └───...
    │
    ├───┬─Library2
    │   ├───Code
    │   ├───...
    │   └───┬─References
    │       └───Library1 (Not used as Library1 is present)
    │
    ├───┬─Library4
    │   ├───Code
    │   ├───...
    │   └───┬─References
    │       ├───Library1 (Not used as Library1 is present)
    │       └───Library3

What have you tried?

There is not much you could try as it's simple not supported right now.

Additional context

No response

[Retroeer]

I believe Facepunch has said they're against Libraries referencing other Libraries

[garrynewman]

Not totally against it, but I think this stuff complicates libraries and makes them a lot less useful

[MD485]

I'm not really in favour of it myself, it seems every time you allow dependency trees it more often than not becomes soup. https://www.npmjs.com/package/is-even?activeTab=dependencies Is-Even is dependent on Is-Odd which is dependenant on Is-Number!

The libraries often doing what they say on the tin and being self enclosed makes them a lot easier to reason about. As opposed to the "SUPER AMAZING LIBRARY" that combines SDF tools, the player controller and opium post process and just slaps a lackluster ui on top. Then you'll have to sit around wondering what mod pack is ideal for your use case, when often times a lot of these tools are meant to be used in isolation and with foresight in terms of their purpose in your game mode.

[Metapyziks]

As it is, you can't have a math extension library that other libraries use. They'd all have to include their own internal copies of that code. If there's a bug to be fixed, all those libraries would need to be updated to include the fix. What's worse, they'd all have their own versions of whatever instance types are declared by the common library, so they can't interoperate.

I'm not really in favour of it myself, it seems every time you allow dependency trees it more often than not becomes soup. https://www.npmjs.com/package/is-even?activeTab=dependencies Is-Even is dependent on Is-Odd which is dependenant on Is-Number!

Kind of a weak argument if you use a parody package as an example! You can just partition things off in a non-ridiculous way.

[yuberee]

I can see myself using or releasing small libraries in the future. An example that came to mind is a "Weighted List" Library, which is useful for picking random items off a list with a bias towards some over the other. As it stands now you wouldn't be able to make, for example, an NPC library that implements the weighted list for spawning NPC types.

Examples aside, Facepunch itself released a useful but barebones library known as Polygon Mesh Library which I can see being the backbone of many other libraries wrapped around it.

[MD485]

The problem is that is-even is specifically not a parody package, it's a package which achieves it's function and commonly used packages depend on it, it gets 120k weekly downloads and that's a low compared to it's history.

Ubre similarly is planning to release a relatively small package, the weighted list library could be a single method.

public T GetWeighted<T>( List<(T item, int weight)> values )
{
    if ( values.Count == 0 )
    {
        Log.Error( "Input List Empty." );
        return default;
    }

    var totalWeight = values.Select( x => x.weight ).Sum();

    var index = Random.Shared.Next() * totalWeight;
    foreach(var x in values)
    {
        index -= x.weight;
        if ( index < 0 ) return x.item;
    }

    //Code should never reach here but if it does I assume it's due to floating point rounding errors.
    return values[values.Count - 1].item;
}

If hypothetically one day Ubre says "I think this method should return null when the list is empty, and only allow nullable types, because a defaulted struct might be a valid entry in a list".

Which can be somewhat valid reasoning, but might break hundreds of libraries dependent on this one, so you'd have to add version targetting to libraries. Interoperability ends up being shot anyways.

In a vaccum it's a great idea, and if fp decides to do it I'm sure it'll be well reasoned, but it's always gonna be a can of worms.

[yuberee]

Isn't Library versioning already possible? Every package on Sbox.Game has every version uploaded, you can even pick which version is the live one. Library references could always default to that specific version and you'd have to maually update if you want to use the newest version.

[matekdev]

The problem is that is-even is specifically not a parody package, it's a package which achieves it's function and commonly used packages depend on it, it gets 120k weekly downloads and that's a low compared to it's history.

Ubre similarly is planning to release a relatively small package, the weighted list library could be a single method.

public T GetWeighted<T>( List<(T item, int weight)> values )
{
  if ( values.Count == 0 )
  {
      Log.Error( "Input List Empty." );
      return default;
  }

  var totalWeight = values.Select( x => x.weight ).Sum();

  var index = Random.Shared.Next() * totalWeight;
  foreach(var x in values)
  {
      index -= x.weight;
      if ( index < 0 ) return x.item;
  }

  //Code should never reach here but if it does I assume it's due to floating point rounding errors.
  return values[values.Count - 1].item;
}

If hypothetically one day Ubre says "I think this method should return null when the list is empty, and only allow nullable types, because a defaulted struct might be a valid entry in a list".

Which can be somewhat valid reasoning, but might break hundreds of libraries dependent on this one, so you'd have to add version targetting to libraries. Interoperability ends up being shot anyways.

In a vaccum it's a great idea, and if fp decides to do it I'm sure it'll be well reasoned, but it's always gonna be a can of worms.

I'm confused what this example has to do with libraries being able to reference each other? It makes it seem like you are arguing against library support completely since you think modifying some base library will propagate to every single project that uses it automatically.

[MD485]

Yes, I was referring to the benefits ziks outlined.

They'd all have to include their own internal copies of that code. If there's a bug to be fixed, all those libraries would need to be updated to include the fix. What's worse, they'd all have their own versions of whatever instance types are declared by the common library, so they can't interoperate.

If you're targetting a specific version of a library, it's quite reasonable to end up in circumstances where all the libraries you want to use target unique versions of a common library, causing the same issues with interoperability and code duplication.

The only time you'd always have the benefits listed is if all changes immediately propagate.

[trundlr]

I like library references, as long as the source for each is available to the end user.

The current versioning system allows for explicit updates when the consumer needs it. If facepunch.libsdf references facepunch.libpolygon, and libpolygon gets updated, then libsdf must manually choose to update to the new version. Then, libsdf would be a new version. The consumer of libsdf would then choose to update, knowing full well what has been changed. They can choose NOT to update, or even manually fork the library to suit their own needs.

Regarding bugfix propagation, you can manually fix it yourself without having to rely on a chain of libraries updating one by one.

[Game4Freak]

Before we end up in an endless debate for pros and cons of referencing libraries I would like to come back to the main point why I created the feature request.

Dependency trees can be hell, there is nothing you can say against that but they are very important if you want to have compatability and extendability. So instead of arguing about that, we should try to figure out how we can have dependency trees but keep it simple so it doesn't create a hell.

Not having library references is waisting so much potential for s&box that makes me kind of sad. If you don't have library references you are limited to very simple libraries. If you can have library references you can create whole frameworks, addons etc.

For sake of simplicity there is no real benefit in having is-even and is-odd libraries for references. But having libraries that could build on top of e.g. the Polygon Mesh Library would be a huge benefit.

I already posted one approach in the feature request itself how you could allow for references but keep it simple. So if you have better ideas, I missed something or you don't understand what my idea was about, feel free to post it ;)

[garrynewman]

While I get the desire to write and ship code, and then use it from multiple difference places, and build on that, and build on that.. I think ultimately we're getting ourselves into a situation where you download a library to add some viewbob or something and it downloads 2 other libraries and they download 2 other libraries and they download 2 other libraries. And then I feel like all that could be avoided if they copied and pasted the single file from the library.

I get the desire, but lets try to avoid it if we can.

[PeteBroccoli]

Yes, I was referring to the benefits ziks outlined.

They'd all have to include their own internal copies of that code. If there's a bug to be fixed, all those libraries would need to be updated to include the fix. What's worse, they'd all have their own versions of whatever instance types are declared by the common library, so they can't interoperate.

If you're targetting a specific version of a library, it's quite reasonable to end up in circumstances where all the libraries you want to use target unique versions of a common library, causing the same issues with interoperability and code duplication.

The only time you'd always have the benefits listed is if all changes immediately propagate.

I think this is important to consider because, as time goes on and libraries mature, certain libraries will stop being updated and therefore use old versions of other libraries. Eventually, with a big enough library set, a developer will have many duplicated versions of the same library in different libraries. This is no different from how the current system works.

I understand that, in the short run, developers will be eager and constantly update their libraries or dependencies and thats where proper dependency management will be ideal. But in the long run, the overheads that such a system have on the end user and the people who have to maintain it (Facepunch) are much more considerable then the benefits imo.

[Game4Freak]

While I get the desire to write and ship code, and then use it from multiple difference places, and build on that, and build on that.. I think ultimately we're getting ourselves into a situation where you download a library to add some viewbob or something and it downloads 2 other libraries and they download 2 other libraries and they download 2 other libraries. And then I feel like all that could be avoided if they copied and pasted the single file from the library.

I get the desire, but lets try to avoid it if we can.

Totally, that why a traditional approach for dependency management will not fit with the way how libraries are supposed to be.

Personally I don't care about having duplicate code if it makes things simpler. I mostly care about the compatability part.

Lets look at an example

If I create following 2 libraries:

• Inventory System
• Weapon System

I want both libraries to be independent on their own as it's up to the user to decide what they want to use. But I also want both libraries to be directly compatible / integrated.

In the case of the weapon system, it should be based on items so it can be easily used with the inventory system. But as it is independent, the user doesn't need to have the inventory system to use it.

This can only be done if I can share the item definition (e.g. interface / class) between both libraries. You can't just copy paste the item definition in both libraries as this will not compile due to conflicts and even if it would, it's still 2 different definitions from 2 different assemblies.

[PeteBroccoli]

While I get the desire to write and ship code, and then use it from multiple difference places, and build on that, and build on that.. I think ultimately we're getting ourselves into a situation where you download a library to add some viewbob or something and it downloads 2 other libraries and they download 2 other libraries and they download 2 other libraries. And then I feel like all that could be avoided if they copied and pasted the single file from the library. I get the desire, but lets try to avoid it if we can.

Totally, that why a traditional approach for dependency management will not fit with the way how libraries are supposed to be.

Personally I don't care about having duplicate code if it makes things simpler. I mostly care about the compatability part.

Lets look at an example

If I create following 2 libraries:

• Inventory System
• Weapon System

I want both libraries to be independent on their own as it's up to the user to decide what they want to use. But I also want both libraries to be directly compatible / integrated.

In the case of the weapon system, it should be based on items so it can be easily used with the inventory system. But as it is independent, the user doesn't need to have the inventory system to use it.

This can only be done if I can share the item definition (e.g. interface / class) between both libraries. You can't just copy paste the item definition in both libraries as this will not compile due to conflicts and even if it would, it's still 2 different definitions from 2 different assemblies.

To play the devil's advocate here, couldn't you just have a "base" library that both of these use? It'll share the item definitions and whatnot and then the user choses to install A and/or B.

This means that you can copy/paste the libraries with just the one extra step of installing the shared library.

This isn't a new/novel way of doing things, other library systems do this as well and they work just fine.

[Game4Freak]

Yes that's how you would do it but you can't right now (atleast within the library system) as you can not reference anything except the s&box base within your library

[youarereadingthis]

Personally I wouldn't mind using git to import libraries where dependency trees are relevant, so I'm fine with sbox not supporting them. We'll still have many useful standalone libraries.

At the same time, are dependency trees in sbox really so bad? The sub-libraries are typically lightweight. You could also have a built-in limit of how many branches the dependency tree has to discourage this behavior/limit API load, directly solving the main concern.

[bub-bl]

S&box's aim is to offer a simple solution for creating games and tools in an easy, intuitive way. However, the current system is counter-intuitive.

There are advantages/disadvantages on both sides. I think all ideas are welcome. But I think it would be necessary to find a compromise that would allow developers to use other libraries in their own library. I don't think that copying and pasting code is a good solution, because several libraries might have the same duplicated code.

Imagine a game using dozens of libraries using the same duplicate code for a GameObjectSystem. This is simply not viable.

Finally, I'd like to describe a problem we'll all encounter at some point. Let's assume that a library exists to create inventories. Its code might look something like this:

Umb.Inventory Library

public interface IInventoryItem
{
    string Name { get; }
    string Icon { get; }
    // ...
}

// This class controls the inventory system (adding items, deleting items, displaying inventory, etc.).
public class Inventory : Component
{
    // This list is actually of type List<Umb.Inventory.IInventoryItem>.
    private readonly List<IInventoryItem> _items = new();

    public void Add(IInventoryItem item)
    {
        _items.Add(item);
    }

    // ...
}

Umb.EconomySystem Library

public class DollarItem : IInventoryItem
{
    // ...
}

Umb.WeaponSystem Library

public class WinchesterWeaponItem : IInventoryItem
{
    // ...
}

If I copy/paste the IInventoryItem interface into the Umb.EconomySystem and Umb.WeaponSystem libraries, I simply won't be able to add items from the above libraries to my inventory.

The Inventory class (Umb.Inventory) has a List<Umb.Inventory.IInventoryItem> list. This makes it impossible to add the DollarItem and WinchesterWeaponItem items to this list.

• DollarItemimplementsUmb.EconomySystem.IInventoryItem`.
• WinchesterWeaponItemimplementsUmb.WeaponSystem.IInventoryItem`.

What am I supposed to do in this situation? Copy and paste my entire inventory into each of the libraries (Umb.EconomySystem, Umb.WeaponSystem)?

Clearly, we find ourselves in a complicated situation. Dealing with library dependencies surely requires a lot of work on the Facepunch side. But let's be honest, we can't really do otherwise...

[PeteBroccoli]

S&box's aim is to offer a simple solution for creating games and tools in an easy, intuitive way. However, the current system is counter-intuitive...

To play the devil's advocate here, couldn't a developer construct a tree of libraries with the shared library being the root?

Say, for your example, the library creator would create their inventory library. Other users could then place their extending libraries within this folder, so that multiple libraries could use this base library.

It does raise questions about how easy it would be to update these libraries (and potentially having very deep nesting of folders), but I think that could be managed by developers themselves.