search

deplinenoise / tundra

Tundra is a code build system that tries to be accurate and fast for incremental builds
MIT License
438 stars 75 forks source link

A proposal for a package manager #342

Closed leidegre closed 11 months ago

leidegre commented 11 months ago

disclaimer: this is not a feature request nor a bug report I just wanted to share my two cents.

Right now I just build all my external dependencies by putting the source in a folder and writing a unit for it.

For example, to use zlib:

StaticLibrary {
    Name = "zlib",
    Sources = {
        "vendor/zlib-1.2.13/adler32.c",
        "vendor/zlib-1.2.13/compress.c",
        "vendor/zlib-1.2.13/crc32.c",
        "vendor/zlib-1.2.13/deflate.c",
        "vendor/zlib-1.2.13/gzclose.c",
        "vendor/zlib-1.2.13/gzlib.c",
        "vendor/zlib-1.2.13/gzread.c",
        "vendor/zlib-1.2.13/gzwrite.c",
        "vendor/zlib-1.2.13/infback.c",
        "vendor/zlib-1.2.13/inffast.c",
        "vendor/zlib-1.2.13/inflate.c",
        "vendor/zlib-1.2.13/inftrees.c",
        "vendor/zlib-1.2.13/trees.c",
        "vendor/zlib-1.2.13/uncompr.c",
        "vendor/zlib-1.2.13/zutil.c"
    },
    Includes = {
        "vendor/zlib-1.2.13"
    },
    Propagate = {
        Includes = {
            "vendor/zlib-1.2.13"
        }
    }
}

But what if I could do this?

Program {
  Name = "awesome-zip",
  Depends = { 
    "https://github.com/madler/zlib#zlib"
  }
}

As part of the build or as part of some package manager:

And you should be to build it like so:

Build {
  Units = {
    "units-packages.lua", -- managed by package manager
    function()
      Program {
        Name = "awesome-zip",
        Depends = {
          "t2_pkg_zlib_zlib"
        }
      }
    end
  }
}

Stuff like this is like opening the biggest can of worms ever! but there are some good ideas out there on what to do and what not to do. I'm really impressed by what the Go authors did in this space and I think they had many good ideas.

If there's any interests in this from the community I would like to try and hash out some kind of spec for how you could package and share code that will be built with Tundra. For example, it is unlikely that the zlib GitHub repo will contain a units.lua file but one could be provided from a repository if there was a simple package manifest format that could combine source location and Tundra unit definitions. I don't suggest hosting a repository but maybe we could dedicate a github repo for this purpose.

This way, looking up a package is simply a fetch from this github repo which may reside on github.com or locally (i.e. it's just a local file system). Installing "zlib" is just looking up the name "zlib.json" in this repo. Here's an example of a package manifest JSON:

{
  "source": {
    "type": "git",
    "url": "github.com/madler/zlib#15c45ad"
  },
  "units": [
    {
      "name": "zlib",
      "sources": [
        "adler32.c",
        "compress.c",
        "crc32.c",
        "deflate.c",
        "gzclose.c",
        "gzlib.c",
        "gzread.c",
        "gzwrite.c",
        "infback.c",
        "inffast.c",
        "inflate.c",
        "inftrees.c",
        "trees.c",
        "uncompr.c",
        "zutil.c"
      ],
      "propagate": {
        "includes": ["."]
      }
    }
  ]
}

"source" could be changed to something like:

{
  "type": "fs",
  "path": "D:\\dev\\github\\zlib"
},

In which case you just symlink this path into something like a <workspace>/t2-packages/zlib.

I don't want to go crazy with this but what I want to do is to spread tundra and make it more valuable for people to adopt. And I strongly believe in keeping things simple. To me, a Tundra package is ideally just a way for me to obtain the source and some unit definitions that I can then reuse in my Tundra projects. Not all projects in the world wide web will lend itself to this approach and I think that's fine. I also think it might be possible to do this without changing a single line of Tundra source code. Maybe add a special unit type for some name remapping/namespace feature but that's it.

If the interest in something like this is low I will close this issue otherwise I might make a few experiments and see what sticks.

deplinenoise commented 11 months ago

I appreciate what you're trying to do, but I don't want to put a package manager in this build system. It seems to me you could get 99% of what you're trying to here with a pre-processing tool that would load the build scripts and run the checkouts as needed, perhaps making additional source list files. That way there's less stuff that can break in Tundra itself.

leidegre commented 11 months ago

Yeah, no I get that. No worries.

I think I was just looking for a place to talk about this. Hoping that additional people might find it useful or interesting.

emoon commented 11 months ago

I think it's an interesting idea, but yeah, it comes with lots of issues.

leidegre commented 11 months ago

I think I'll revisit this in the future. See what can be done. I'll ping you guys on Twitter if/when I have something to share.