Separation of cogs

[bddvlpr]

To add a bit of context to this proposal I want to clarify that I'm currently working on a set of Nix modules and packages to make Steel environments reproducible along with supporting the helix plugin system later down the road.

Currently the state of cogs are a bit unclear in my opinion, some of them being located in cogs/ while others (that require one or more dylibs) are located in lib/. Most of them follow the steel/<name> name structure, while others use steel-<name> which seems rather illogical to me. In order to achieve a bit of structure I propose these should be structured either in their own repositories or put in a 'standard library-like' monorepo. Alternatively, they could remain in this repository, but have both /lib and /cogs merged.

The mono-repo, standard library-like, approach explained above. Optionally if a cog requires a dylib, its source and Cargo.toml is nested in their folder respectively.

.
├── Cargo.lock
├── Cargo.toml
└── cogs
    ├── lists
    │   └── cog.scm
    ├── markdown
    │   ├── Cargo.toml
    │   └── cog.scm
    ├── sorting
    │   └── cog.scm
    ├── sys-info
    │   ├── Cargo.toml
    │   └── cog.scm
    ├── threads
    │   └── cog.scm
    ├── time
    │   └── cog.scm
    └── websockets
        ├── Cargo.toml
        └── cog.scm

Separating every cog into its own repository seems like the most work but the easiest to maintain in the future.

Let me know what you think, or if there is a reasoning behind the current folder structure. Great work on Steel so far! :smile:

[mattwparas]

Thanks for the kind words.

So first, any lack of clarity is pretty much due to there being no "official" policy on how things are organized, and as a result there may be things that just don't make since, simply because I didn't notice the irregularity or made some mistakes along the way.

You're right, anything under libs is a library requiring a dylib. The idea there with the naming structure was more to name the crate something specific to steel, that being said I don't think the associated cog needs to have the name, so that can probably be adjusted to keep everything in like. For example, steel-sqlite probably could just be steel/sqlite to keep it in line with everything else (that applies to the other ones as well)

I considered a separate repository for the standard library, and still may go that way, however given that I actively develop this it was nice to have all of the tests run and keep everything nice and tidy under a workspace so that I can make sure things build correctly. I'm all ears for other options that still make it easy to develop. When working on the FFI layer, its nice to effectively have a test suite with just cargo build --all.

The proposed structure isn't a bad idea, and I'm happy to explore it. The reason I separated dylibs from cogs is mostly due to how I build and install things with the installer (see cogs/install.scm - it made things simpler at a time when there weren't many eyes looking at steel :smiley: . I'll explore how easy it would be to just merge libs and cogs and look at renaming the libraries.

Pretty much everything is open to be changed though, and I'd happily accept contributions in this area as well if you're interested in hacking on it

[bddvlpr]

I'm willing to take a shot at it, transitioning to a separate standard library repository could be done while keeping the cogs in this repository for the time being. As soon as the repository gets too heavy to maintain with all included cogs, we could start separating them officially (and giving the installer support for it).

As for communication, have you though about setting up a matrix room, Discord guild or slack for community related talks?

[mattwparas]

As long as the installer can sort of bootstrap itself by getting the standard library via git, I don't see an issue with that at the moment.

And yes, I'd like to set up a matrix room but haven't yet. I have a discord for now that is linked in the README. Discussions on github are also fine

[bddvlpr]

Great, I'll be trying to get a PoC working soon.