[Discuss] CLI library scope and vision

[joshgoebel]

From the discussion in #81:

That module is huge - almost 800 lines. I'm meaning to type up an issue for discussion on the philosophy of the CLI. If it tries to adhere somewhat to the minimalism of Wren Core then I'd think that would rule a large class like that out. After all one can always just include that Wren code in their project easily - there is no need that it must be compiled into the CLI.

The CLI library by necessity needs to do things that are impossible to do from Wren alone - such as load a file, get the system time, open a network socket... but timezone math, date parsing, lunar calendar vs solar calendar vs gregorian calendar are things that could be all easily be added by an external library (like Purefox's). [snip]

---

I think we need to begin a high level discussion about what belongs in the CLI library and perhaps have a bit of a plan/vision moving forward. Perhaps we want it to be sprawling like the Ruby standard libs. More likely the Wren philosophy will have it uber minimal and supplemented by a huge variety community driven libraries, but time will tell. Things like the proposal for user-defined native modules would make what I said above matter far less. I think shortly we'll have some ability to install and use libraries (non-native) from a WREN_HOME and that greatly reduces pressure to "add everything to the CLI".

For example I got excited about adding networking (and that certainly depends on native support) but once we can have native libraries then someone could just:

wren install networking

...and then:

import "networking" for HTTP

And boom, they have access to all the classes networking provides from wren. So I'm leaning towards a "smaller and more minimal" built-in API with non-native (and later native libraries) being easy to install into WREN_HOME. But I wanted to start a topic so everyone could contribute their thoughts.

[clsource]

I like the following ideas, that I think are shared from Wren projects like DOME:

• Minimal Core. Just enough building bricks for creating more complex external components
• Easy to Extend. Provide a way to easily implement "extension", "plugins" or "libs" composed on only Wren code or native bridges to libraries from other languages like C, Go, Rust.

By having a simple yet robust core we can have "Progress over perfection". Since the ecosystem can flourish using a descentralized component library where each user can select which package to implement in their script.

About the dependencies I think we can learn from the GOPATH and Go Modules story 👍

• https://www.digitalocean.com/community/tutorials/understanding-the-gopath
• https://divan.dev/posts/gopath/
• https://blog.golang.org/using-go-modules

So in conclusion my taste would be:

• Slim and simple Wren core
• An external package like wren-essentials with most used libraries and extensions like networking or date utils

[ChayimFriedman2]

The Wren core should definitely be minimal, but I don't think the CLI should be. Plugins are great, but for the CLI I think it's better to add than to not. It is meant to be usable to writing programs (and scripts), so I think we should do something more like Python's standard library, and less like C's one.

[PureFox48]

I agree with @ChayimFriedman2 that there is no need for the CLI to be minimal but, OTOH, I wouldn't want to see very bloated 'standard' modules either.

What I think we need to aim for is that sweet spot where the modules make it easy and intuitive to perform basic tasks but expose enough of the underlying functionality to enable folks to build more comprehensive modules in Wren itself. Ideally, as much as possible of the infrastructure should be written in C, or use suitably licensed C libraries, to maximize performance.

Although anybody is welcome to use them for any purpose, I don't consider my own modules (such as Wren-date) to be candidates for the standard library. These modules were written with the sole purpose of enabling me (and anyone else who's interested) to more easily complete Rosetta Code tasks where you soon learn that you need to rival the coverage of the major languages' standard libraries to keep the size of the scripts to reasonable lengths. Consequently these modules are large and complex and written entirely in Wren itself since it's important that they can be run under 'vanilla' Wren-cli without the need for any patching.

Rosetta Code is a very active site which gets plenty of visitors including from colleges and schools. As such it's a good opportunity for a small or nascent language to show what it can do and get itself noticed by the programming community at large.

Clearly, as well as a decent standard library, we also need the ability to be able to use native libraries from Wren and perhaps a somewhat more streamlined module system than we have at present. I like the ideas in #52 and #78 here and DOME-style plug-ins are also worth considering.

[clsource]

If we need some blueprints for what can be inside the CLI as Standard libs, we can look into other projects for their standard libs.

Here is the standard libs for https://nekovm.org/doc/libs/ and Wren implements most of these. There are a only a few ones that the CLI can supply its implementation.

Buffer, Date, File, Int32, Math, MD5, Memory, Module, Random, Serialize, Socket, String, System, UTF8, Xml, Thread, Ui, Process, Misc, Regexp, Mysql, Mod_neko, Sqlite, ZLib and the builtins.

[ChayimFriedman2]

I suggest to look at Python. No wonders they have very broad standard library.

[clsource]

Python has a wonderful extensive library. But they got 30+ years of development and millions of dollars of funding by corporate overlords xd.

So my proposal is start small with a bunch of useful modules and start scaling from there.

Python would be a good inspiration, but implementing that vast amount of libraries it's a task not for the faint of heart :)

[ChayimFriedman2]

I did not mean we should have the library Python does. I meant we should aim there.

[clsource]

Yes thats a good long term objective.

But I doubt we want a wren_cli with too much weight on its binary. So most of those future libs can be delivered as a dynamic extension or plugin.

My idea is focusing on what can be available without any so, dylib or dll. for the core wren_cli everthing else can be an extension.

[clsource]

One idea is to limit the size of the binary of wren_cli. That way it cannot grow more than that to be manageable. A good maximum sice I think would be a floppy disk.

• 3 1/2-inch floppy disk (1.44MB)

For the code itself I think the CLI should aim the same as Wren

• Under 4,000 semicolons. You can skim the whole thing in an afternoon. It’s small, but not dense. It is readable and lovingly-commented.

Another project can be delivered to complement basic wren_cli with modules that needs an extension to work. Like wren-essentials that you can download separately and does not have those constraints.

• The binary is small and have a bunch of building bricks.
• The extended library can be as huge as python's one 👍

[joshgoebel]

But I doubt we want a wren_cli with too much weight on its binary.

Agree at a high level, just I think the real issue is size/complexity/maintenance of a sprawling main library, not the size of the binary (which is likely to still be quite small).

My Ruby binary is only 9kb (wow!) but then libruby.dylib is 4.5MB.

A good maximum sice I think would be a floppy disk.

Arbitrary, but I love a good limit. ❣️

For the code itself I think the CLI should aim the same as Wren > Under 4,000 semicolons.

Agree in spirit... but we can't count JUST ; since a large portion of API can be written in Wren itself.

We currently have:

~ 300 ; (in Wren C module code) ~ 1100 lines of Wren code

[clsource]

That limit was based when I mix static libs from other langs to Wren CLI

• Golang standard static lib weights around 2 MB
• Haskell standard static lib weights around 1.5 MB

My Ruby binary is only 9kb (wow!) but then libruby.dylib is 4.5MB.

So that's my point 👍 Wren CLI can be super small and easy to understand like Wren. But its standard library is a project that can be developed in another repo without such limits.

The only thing required here is that Wren CLI allows extensions so, dylib and dll files and a few quality of life helpers.

💯

[joshgoebel]

So that's my point 👍 Wren CLI can be super small and easy to understand like Wren. But its standard library is a project that can be developed in another repo without such limits.

Exactly. For naming I'm personally starting to like the following nomenclature:

• Core Wren (the actual Wren repository)
• CLI Core or perhaps just CLI
• Standard Library (though it will take some real community effort before it's much of standard) :-)

The real question then becomes does CLI core need networking... is it a requirement that the CLI can talk to the internet out of the box...? I think Bob originally weighed in once upon a time that eventually that was the goal - but I don't know if he was imagining these larger distinctions and thinking in terms of CLI Core vs Standard Lib.

I think the big reason to say yes would be if it needed to download/fetch packages on it's own (serving as a package manager)... but I'm learning towards no. If we truly need a network enabled package manager (vs just using git) I think that should be a separate project which would have it's own binary and could compile in networking, etc... or simple be a library one installed like any other library:

cd .wren/libs
git clone package_manager
git clone networking

[clsource]

I think Wren CLI should follow the unix UNIX philosophy of do one thing and do it well

So we can separate those ideas in different projects

• A Wren version manager (I think there is a Wren config for https://asdf-vm.com/)
• A Package manager (That can be installed from the Wren Version manager)
• The Wren CLI
• The Wren CLI Extended Library
• The Wren Language

So my proposed nomenclature:

• Core or just Wren. Refers to the language itself
• CLI. Refers to the official application that embeds Wren and provides basic classes like File, Process and the ones already available in the current version.
• EXLIB or EXTENDED_LIBRARY. Refers to a dynamic set of community extensions to the CLI like networking, json, hash, date and other utils and bring more power to the standard CLI.
• NEST or PACKAGE MANAGER. Refers to the package manager that enables you to install EXLIB or other extensions to CLI.
• WVM. Or Wren Version Manager or WUM for friends. An app that enables you to install different versions of CLI, and NEST.

The package manager and wren version manager does not need to be written in Wren. Maybe they could be written in Golang or Rust. An option would be forking the cli and embed the Go runtime. But that could be a little off i guess.

[clsource]

So. Which modules should be inside wren_cli?.

Modules that would not require a dynamic libraries to function. And will be statically linked to the binary, so it will add some weight to it.

Some ideas:

• The already available ones.
• Basic unix timestamp. This is needed for more elaborate date classes in the future. The minimum requirement is just the unix timestamp.
• Simple Env. For reading environment variables.
• Bare metal networking. (Although my opinion is better leave this to the extended library dylib). But a good option would be included libcurl with a wren wrapper.
• Extension Loader. (For loading dynamic libraries and Wren extensions)
• Version checker. So libraries can tell which Wren version they will support.
• Regular expresions wrapper around Perl Regular expressions? (maybe this is better of the extended lib)

I don't know which other modules are the building bricks needed for creating the extended library.

[ChayimFriedman2]

I don't agree with @clsource.

The CLI does not have to be small. It's not the Wren core and you are not going to run it on embedded either. It should have batteries-included for executing programs. Although Node.js can be a good candidate for small-but-useful std, I still prefer Python, and don't like the idea of having to include dozens of dependencies just to execute some shell script.

[clsource]

While I agree having a "batteries included" binary would be nice to have. Wren CLI have a bus factor of 1. This means that we depend on Ruby's availability to review PRs and implement changes.

To allow implementing new features and libraries in the fastest way possible, the best approach in my opinion is to separate Wren CLI and the "Extended library".

Having Wren CLI solely maintained by Ruby and the Extended Library as a separated project maintained by the community is the best cost - opportunity.

• We can rely on Ruby's judgement for having the best Wren CLI possible
• We can rely on the community for creating wonderful extensions to the Wren CLI

Also, nothing is stopping that the wren version manager installs the extended-library by default when you install a wren-cli. So at the end the batteries can be included, only they will be separated from the wren-cli executable.

Take note on @joshgoebel about Ruby Lang

My Ruby binary is only 9kb (wow!) but then libruby.dylib is 4.5MB.

All I am advocating for is creating a libwren.dylib that has all the batteries that Python and Ruby lang have. So it can grow as much as it needs 👍

[PureFox48]

When considering adding functionality to Wren, I think you need to distinguish between 3 cases:

1. Stuff that would be useful for both embedded and CLI.

2. As 1. but, in the case of embedded, the functionality in question is best left to the host to provide.

3. Stuff that is only useful to the CLI.

In Box 1, I'd place regular expressions which in view of its likely size I think would have to be an optional module.

Basic string formatting (NOT my monster 800+ lines Wren-fmt) is something else that naturally falls into Box 1.

In Box 2, I'd place date/time support though, as far as the CLI is concerned, I'd like to see this go a liitle further than just the Unix timestamp.

Another possible candidate for Box 2 would be common cryptographic algorithms such as SHA-256/512.

In Box 3, I'd put stuff such as networking, file compression, env, ability to use devices, support for running external processes and accessing external libraries.

So vis-à-vis Wren-cli, we only need to concern ourselves with Boxes 2 and 3.

As far as networking is concerned, I don't know whether it would be best to use libcurl or see how far we can get with libuv. The former is subject to a modified MIT license but, although I'm no lawyer, I don't think that would necessarily be a problem for us. However, it would of course mean that the standard library needed an additional dependency.

I think @clsource makes a good point about the demands on @ruby0x1's time in overseeing all this. That's one reason why I'd like to keep the standard modules fairly lean which the community itself can then enhance as required.

[bedax]

I've only scanned the discussion so far, so apologies if I'm repeating or missing anything.

I really like wren's simplicity and I think it would be a good replacement for what I've previously used shell scripts and python scripts for. The main thing that it appears to be lacking for this is the ability to spawn processes, and then to read from and write to their stdin/stdout/stderr. If that feature is added and made easy to use, then it will be able to replace a POSIX shell interpreter like dash, and then, to my mind, it doesn't need much more than that.

I think that wren-cli should consider itself to be in a similar league to dash, and not a batteries-included interpreter like python, as that would inevitably lead to clutter. So I think you're right to lean towards a very small core and then make it easy for users to specify a location where other modules can be imported from.

That said though, since wren-cli already has access to libuv, it seems like a waste to have its various capabilities sitting there and not expose them to the user, expecting the user to pull in an external library for functionality wren-cli already has. So to my mind, wren-cli should expose the following from libuv: non-blocking sockets, non-blocking file/directory operations, process spawning with stdin/stdout/stderr handles, probably ctrl+c signal receiving, and if you're still feeling ambitious, then file system events. After that, it will need to somehow expose the event loop scheduler to bring it all together. The end result would still be very small but it would have the power necessary to easily build network services, system admin tools, and so on.

Edit: You could then import "networking" for HTTP or whatever, which could be a pure wren library out in the ecosystem that builds on the basic OS functionality that wren-cli exposes from libuv.

[ChayimFriedman2]

I don't think libuv includes HTTP support, only raw sockets, am I wrong?

[ruby0x1]

expose the following from libuv: non-blocking sockets, it would have the power necessary to easily build network services

It doesn't! But that isn't what was being suggested. Rather that you could implement them above the baseline.