Document how to hack on the Lingy source code

PEZ commented 1 year ago

Hi! I got all fired up by tour presentation Lingy & YAMLScript! Clojure is my favorite programming language, and I love seeing it being brought to this new/old place. 😄 It is 20+ years since I did any Perl stuff, but I have fond memories.

Anyway, I want to build Lingy, but doing make build gave me this:

mkdir -p /tmp/clojure/
curl -s https://raw.githubusercontent.com/clojure/clojure/clojure-1.11.1/src/clj/clojure/core.clj > /tmp/clojure/core.clj
perl tool/compile-clojure-core /tmp/clojure/core.clj https://raw.githubusercontent.com/clojure/clojure/clojure-1.11.1/src/clj/clojure/core.clj > lib/Lingy/core.clj
Base class package "immutable::map" is empty.
    (Perhaps you need to 'use' the module which defines that package first,
    or make that module available in @INC (@INC contains: lib /Users/pez/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/darwin-2level /Users/pez/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0 /Users/pez/perl5/perlbrew/perls/perl-5.32.0/lib/5.32.0/darwin-2level /Users/pez/perl5/perlbrew/perls/perl-5.32.0/lib/5.32.0).
 at lib/Lingy/HashMap.pm line 4.
BEGIN failed--compilation aborted at lib/Lingy/HashMap.pm line 4.
Compilation failed in require at lib/Lingy/Compiler.pm line 6.
BEGIN failed--compilation aborted at lib/Lingy/Compiler.pm line 6.
Compilation failed in require at (eval 7) line 1.
make: *** [lib/Lingy/core.clj] Error 255
make: *** Deleting file `lib/Lingy/core.clj'

Side note 1: I also had trouble to try build again, because make clean needs zild which I didn't know how to install. But https://learn.perl.org/installing/osx.html told me how to install latest Perl and it also installed cpanm for me, and cpanm Zilla::Dist gave me zild.

Side note 2: Running make build from the project root, generates warnings about locale:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LC_ALL = (unset),
        LANG = "perl "
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

Anyway, from there cpanm immutable::map made the build work. Even though, I don't quite know how to run my dev Lingy version...

Sorry for mixed bag of notes. I'm thinking maybe some notes in the readme, or a doc, about how to hack on Lingy might be good?

PEZ commented 1 year ago

I now made myself a script my-lingy.pm (inspired by the script perl/bin/_lingy.pm):

#!/usr/bin/env perl

use strict; use warnings;
use lib 'perl/lib';  # ChatGPT said to add this line

use Lingy::Main;

Lingy::Main->new->run(@ARGV);

And success:

~/Projects/lingy  main[?✔] 
❯ ./my-lingy.pm            
Lingy 0.1.18 [perl]

user=> (+ 1 2)
3
user=>

ingydotnet commented 1 year ago

Hi @PEZ ,

Wow! Someone from Clojure looking at Lingy already is so exciting!

So you definitely came here from an angle not yet expected (by me), and now obviously I need to think about that, and how the Lingy repo can best welcome people looking at it from this perspective. I love your idea about adding a doc section on how to hack on Lingy. I also need to add a little more to the installation section for people not familiar with the the cpanm Perl CPAN module installer command.

I'll start working on this now and I'll also leave another comment in a few minutes (after I've had a coffee) about how and why the Lingy repo is currently organized. Maybe a bit about how it came to be, if you are interested. :)

Stay tuned, Ingy

ingydotnet commented 1 year ago

Here's a little autobiographical journey into my past. In 2001 I worked at a company called ActiveState that was trying to make a business around several open source languages and projects. I became obsessed with making things that involved multiple languages. That year I started creating the YAML serialization language (with Clark and Oren). I also made the Perl module called Inline.pm that allowed you to write code from C inside Perl and made it work as expected. It was made to be extensible and now works with 25 other languages inside Perl, and I think the concept was used by other languages as well.

I never lost interest in this polyglot journey. This is to explain that these days when I start a new projects like Lingy I want them to apply to as many languages as possible.

In the summer of 2022 I made a prototype language called YAMLScript for various reasons at the time. Around that time I also made https://github.com/ingydotnet/pst#description to help release packages in many languages.

Then last winter I started hacking on YAMLScript again. YS was described as a YAML Lisp but I realized I really knew very little about Lisp. I asked the internet how to learn how to make a lisp and it led me to https://github.com/kanaka/mal/blob/master/process/guide.md :-)

Mal is clojure-based lisplet. After completing my Mal implementation journey in Perl, I then did another Mal quickly in YAMLScript, while reinventing YS completely as I went along. It was a blast!!

I had this last week's perl conference coming up in a few months so I poured time and effort into Lingy and YS.

Which brings us to the current Lingy repo. You'll notice that the top level directory has many directories named after programming languages. Each of these languages is an intended eventual target for Lingy (assuming it continues to work out as well as it has so far).

But the directory you want is ./perl/. And inside that you only want to run make test (if you are hacking on Lingy). All the other make targets are for my own release management of Lingy to CPAN. None of this is in the docs because I hadn't yet expected people to be using it at this level. I'll fix the docs today!

I only imagined people installing the lingy command using cpanm lingy which should just work. It seems you already have cpanm installed but for others reading this see: https://metacpan.org/pod/App::cpanminus#INSTALLATION

I'll also put that in the docs.

I'll explain (without more personal history lessons) how to best hack on Lingy in a follow-up comment.

See Also:

ingydotnet commented 1 year ago

@PEZ,

Your my-lingy.pm script can be done by just a few environment tweaks. I'll explain as fully as I can...

To install Lingy as intended for normal usage as a language you need the following programs installed in your PATH:

perl - almost any version
bash - almost any version
cpanm - to install cpan modules
clojure - for the lingy -C option

Running cpanm Lingy should do everything, including installing all the CPAN module dependencies. All the deps above are probably available as software installs on your system. In some cases you may need to use a system install of the Term::ReadLine::Gnu for reasons I haven't entirely figured out yet.

To hack on Lingy (and hopefully contribute!) I should follow these steps:

git clone https://github.com/ingydotnet/lingy.git
export PATH=$PWD/lingy/perl/bin:$PATH
export PERL5LIB=$PWD/lingy/perl/lib
lingy -e '(println *lingy-version*)'

Now any changes you make to the local sources are live for the next lingy command. There is (use 'lingy.devel) library with debugging functions that I use constantly. Running lingy -D loads that library. Try (XXX *ns*).

I'll get this info into the docs and make a Lingy release today.

@PEZ please feel free to guide me in what's most important to work on next from your perspective. I'm a babe in these woods.

PEZ commented 1 year ago

you may need to use a system install of the Term::ReadLine::Gnu for reasons I haven't entirely figured out yet

For the record (or rather for anyone reading this), I could install that using cpanm Term::ReadLine::Gnu. And for completeness of how I installed things. When installing latest Perl (which I now realize I didn't have to), using the instructions at https://learn.perl.org/installing/osx.html, it updated ~/.bashrc. I just copied over this to ~/.zshrc:

source /Users/pez/perl5/perlbrew/etc/bashrc

And I also put these lines there, because some warnings messages suggested I needed to:

LC_CTYPE=en_US.UTF-8
LC_ALL=en_US.UTF-8

ingydotnet commented 1 year ago

Thanks @PEZ . I'll add those things to the "Hacking on Lingy" doc I'm writing now!

PEZ commented 1 year ago

Thanks for this stellar response to my question! 🙏 I totally appreciate the update on the background to all this. Full disclosure is that I am not considering myself polyglot these days. Clojure changed that for me. 😄 I think it is super cool that you have that approach, since it has the potential for me to stick with Clojure in many environments where I haven't before been able to do so.

I'm a babe in these woods.

Oh, that makes two of use. Depending on which woods we're talking about. I've never ported a programming language to anywhere, but I do have some experience with using Clojure and ClojureScript and quite a lot of experience with editor tooling for Clojure. The latter is why I am interested in getting an nREPL server for Lingy done, see https://calva.io/nrepl_and_cider-nrepl/ for some basic intro to why that is important.

Anyway, in terms on guidance, you seem to have great direction! Your comments in that talk and in the docs in this repo about trying to reuse as much of Clojure core.clj as possible got me to think about this presentation from one of your fellow travelers (on the journey to bring Clojure to new places):

https://www.youtube.com/watch?v=Yw4IAY4Nx_o&list=PLZdCLR02grLpIQQkyGLgIyt0eHE56aJqd&index=14

Where Jeaye Wilkinson spends some time on this very question. (Not suggesting Lingy will have the same constraints as Jank, just thinking it is probably relevant to listen to his approach to the trade-offs).

It would be cool if you joined Clojurians Slack, created a #lingy channel, and also posted about Lingy in the #announcement channel there. I bet there are people there that will find this super exciting!

ingydotnet commented 1 year ago

@PEZ

Can you review this commit for me: https://github.com/ingydotnet/lingy/pull/12

I'd love to join your slack. Maybe invite ingy@ingy.net if you can.

Otherwise I'll figure it out later today.

I'll merge and release that PR after I go attend to some things IRL.

Many thanks!

PEZ commented 1 year ago

I'd love to join your slack. Maybe invite ingy@ingy.net if you can.

Invite sent!

Anyone else reading this, wanting to joining the Clojurians Slack, I think it should work visiting http://clojurians.net/ and self-invite from there.

PEZ commented 1 year ago

Can you review this commit for me: #12

Looks great! I haven't tried the instructions yet, but will let you know how I fare when I do.

ingydotnet commented 1 year ago

Can you review this commit for me: #12

Looks great! I haven't tried the instructions yet, but will let you know how I fare when I do.

Thanks. I'll ship it.

Doc (and all the rest) PRs welcome!

ingydotnet commented 1 year ago

Thanks for this stellar response to my question! pray I totally appreciate the update on the background to all this. Full disclosure is that I am not considering myself polyglot these days. Clojure changed that for me. smile I think it is super cool that you have that approach, since it has the potential for me to stick with Clojure in many environments where I haven't before been able to do so.

My motivation for Lingy is about "fixing" other host languages (starting with Perl) in the ways the Clojure "fixed" Java. Also about bringing together more programming language communities.

Oh, that makes two of use. Depending on which woods we're talking about. I've never ported a programming language to anywhere, but I do have some experience with using Clojure and ClojureScript and quite a lot of experience with editor tooling for Clojure. The latter is why I am interested in getting an nREPL server for Lingy done, see https://calva.io/nrepl_and_cider-nrepl/ for some basic intro to why that is important.

Calva looks cool. Already installed it and watched the video link you sent. We'll get Lingy integrated asap. Created https://github.com/ingydotnet/lingy/issues/13 to do that

https://www.youtube.com/watch?v=Yw4IAY4Nx_o&list=PLZdCLR02grLpIQQkyGLgIyt0eHE56aJqd&index=14 Where Jeaye Wilkinson spends some time on this very question. (Not suggesting Lingy will have the same constraints as Jank, just thinking it is probably relevant to listen to his approach to the trade-offs).

I've already watched that Jank video! My reader actually does tokenize all of Lingy using a single regex. And it can lex all of clojure/core.clj. :)

It would be cool if you joined Clojurians Slack, created a #lingy channel, and also posted about Lingy in the #announcement channel there. I bet there are people there that will find this super exciting!

Done. Very excited to join you.

ingydotnet commented 1 year ago

Closing this issue now that #12 is merged.

lingy-lang / lingy

Document how to hack on the Lingy source code #8