Cannot build Onyx

[stugol]

Right, I've just tried to build Onyx for the first time. And it's kinda broken:

# git clone https://github.com/ozra/onyx-lang.git
Cloning into 'onyx-lang'...
remote: Counting objects: 67639, done.
remote: Compressing objects: 100% (849/849), done.
remote: Total 67639 (delta 524), reused 0 (delta 0), pack-reused 66782
Receiving objects: 100% (67639/67639), 24.53 MiB | 2.37 MiB/s, done.
Resolving deltas: 100% (45520/45520), done.

# cd onyx-lang
# make bootstrap
./bootstrap.sh
Checks for required utils
/usr/bin/wget
/usr/bin/wget
/usr/bin/git
/usr/bin/git

If shit happens - create an issue at https://github.com/ozra/onyx-lang/issues (first make sure it's not already reported!)

Let's sudo you first, so that's done:
./bootstrap.sh: line 27: CRYSTAL_VERSION: command not found
Gets Crystal
--2016-04-08 01:39:18--  https://github.com/manastech/crystal/releases/download/$%7BCRYSTAL_VERSION%7D/crystal-$%7BCRYSTAL_VERSION%7D-1-linux-x86_64.tar.gz
Resolving github.com (github.com)... 192.30.252.130
Connecting to github.com (github.com)|192.30.252.130|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://github.com/crystal-lang/crystal/releases/download/$%7BCRYSTAL_VERSION%7D/crystal-$%7BCRYSTAL_VERSION%7D-1-linux-x86_64.tar.gz [following]
--2016-04-08 01:39:19--  https://github.com/crystal-lang/crystal/releases/download/$%7BCRYSTAL_VERSION%7D/crystal-$%7BCRYSTAL_VERSION%7D-1-linux-x86_64.tar.gz
Reusing existing connection to github.com:443.
HTTP request sent, awaiting response... 404 Not Found
2016-04-08 01:39:19 ERROR 404: Not Found.

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
./bootstrap.sh: line 33: Installs Crystal ${$CRYSTAL_VERSION} at /opt/cr-ox/: bad substitution
cp: cannot stat `crystal--1/': No such file or directory
mv: cannot stat `/opt/cr-ox/bin/crystal': No such file or directory

Compiles Onyx compiler in release mode.
This may take a while - meanwhile you have some options:

  - stretch your legs and grab a coffee.
  - or a cigar.
  - or perhaps tea?
  - I see... you prefer beer?
  - Or do yoga
  - Or ninja moves! - same same but cooler name!
  - whatever makes your clock tick (literally)

You need to have a crystal executable in your path!
Installs Onyx at /opt/onyx/

cp: cannot stat `/opt/cr-ox/embedded/': No such file or directory
cp: cannot stat `.build/onyx': No such file or directory
All done! Enjoy, and don't let the beta-bugs bite!

However:

# crystal -v
Crystal 0.15.0 [0f62400] (Fri Apr  1 17:26:27 UTC 2016)

The problems seem to be:

• You are attempting to assign CRYSTAL_VERSION with spaces around the =
• You cannot interpolate 'single quoted strings'
• The Crystal source has "301 Moved Permanently" (non-fatal error)
• The script reports success even after horrible errors occur
• You don't test your damn scripts!

Might I suggest writing the setup script in Ruby instead? It's a lot less painful. In fact, if you would like, I could write it for you?

(Incidentally, be aware that a very recent version of Crystal is horribly broken. See Crystal issue 2395)

[stugol]

Alright, I corrected errors in your build script, and I'm not getting any more horrible failure messages. But it's still not working:

# onyx
/usr/local/bin/onyx: line 6: /opt/onyx/bin/../embedded/bin/onyx: No such file or directory

It seems that the Onyx executable has inexplicably not been created...

[stugol]

Oh, also:

# /opt/cr-ox/embedded/bin/crystal
zsh: exec format error: /opt/cr-ox/embedded/bin/crystal

Are you doing naughty 64-bit compilings on my poor little 32-bit system?

(actually my systems are all 64-bit, but I'm doing this in a 32-bit virtual machine)

[ozra]

I'll be the first to admit I've spent like 5 seconds on that script ;-) Bash is something I've never took the time to learn properly, I usually just get it "working for my case".

I'd be more than happy for help in this department, since I prefer spending my limited time on the compiler!

Issues:

• The script is really banal, as mentioned
• It is hard-coded to fetch 64-bit binary of Crystal for the bootstrapping (should of course download appropriate according to machine)
• Any issue you find with it is most likely that - a bug / careless code.

Using ruby will of course create a dependency for that also :-/ Good to avoid if possible.

[ozra]

Oh, right, I made some changes the other day regarding CRYSTAL_VERSION, I checked in an untested change. Damn it. I'll push that. Several of the issues still remain though.

[ozra]

It was in a WIP-branch. Now that's up at least...

[ozra]

Parentheses: as can be seen in the script, it does not replace your crystal compiler, it brings down and renames a version only for compiling onyx, as to not "disturb your general installation".

[stugol]

Well, as you will see in Crystal issues 2396 and 2434, I'm _done_ with Crystal. Asterite refuses to help me if I show even a tiny amount of exasperation, and Ybaddadden is just a twat.

So, if using Onyx now - or at any point in the future - requires that I post on the Crystal forums for help, then I'm afraid I'll have to abandon that as well. What's the likelihood of this?

Further, I require the ability to statically compile Onyx code without dependancies. Asterite specifically does not support this in Crystal. If Onyx cannot be guaranteed to support this, then it's pretty much useless to me.

I'm quite prepared to write your setup script for you, but it has to be in Ruby. Bash script is like pulling teeth. So it's up to you.

[ozra]

I understand your frustration (hence Onyx), but watch your emotions and words. This is only possible thanks to the hard work of asterite and others involved. It's a fantastic project with years of effort put into it, we should only be glad we can expand even further in an even better direction (subjectively). Before putting blame, think about what your own investments in the projects are.

Obviously Onyx is still in the starts, the idea is of course to have a separate "world" of discussion, even though efforts on stdlibs are utilized. Problem being of course that I'll probably be the only one of help at the time being, but I'm happy to help where I can. A friend of mine is setting up some IRC-bounce-server-thingie so that I don't miss comments if I'm offline or Konversation crashes or the likes. So please join in there :-) Once the core of Onyx is all in place, it's ready for spreading the word around, and hopefully it attracts coders who wants a "really fucking good language" (TM) (personally I can't see how people would not love it, but then again, I'm quite biased ;-) ).

As for fully statical compilation, I think that's something several of us wants, it will take time to get there though. I only have two hands, one brain and western-world-style-cramped time available ;-) As always, PR's are highly welcome!

I hear you on the Bash-scripting. Please do, I or someone else could always rewrite it to Bash again later on if needed. The important thing now is that it actually works, deps or not.

[stugol]

Right, I'm doing the script now. Why bashscript has to be such a horrible language I've never understood.

[ozra]

Right! Wtf was the Bash-developers on? haha

Thanks!

[stugol]

Why does the script hard-code the Crystal version? Is there some reason we don't want to get the latest version?

[ozra]

Nope - that would be even better :-)

[stugol]

In fact, shouldn't I just clone the current repo, instead of downloading a zip?

[ozra]

No, the binary is wanted, it's only for compiling Onyx, so an additional step of building Crystal too would only be time consuming. Later on (when deemed stable enough) an onyx-binary will be downloaded to compile instead.

[stugol]

FYI, at least one Ruby gem will be required, in order to parse the HTML and grab the latest release. Ordinarily that might be cause for concern, but when you consider the massive pile of other crap that we need - LLVM, Boehm etc - plus the fact that the install script can install the gem as part of the build process it performs, I reckon it's unimportant.

[stugol]

Regarding the fact that building Onyx requires that we already have an environment suitable for building Crystal - that's right, isn't it? - would it be helpful if the Ruby script checked that, and configured all that crap too? Essentially, you'd be able to run the script on a fresh Debian system, and it would install all the LLVM and Boehm crap for you.

Of course, we'd want to support non-Debian systems as well; but these need to be supported explicitly in the script. What distros are the most important to support, do you think?

[ozra]

Sounds good.

• I think the web-scraping could be done simply with a regexp instead of requiring a gem, avoiding an additional dep (ultimately we want to reduce them one by one)
• The current script was only tested on (my) debian based (a buddy failed to use it on Arch Linux). Wider support is of course essential, but starting somewhere is better than nothing
  • I'm not to enlightened about distros (I'm as oblivious as Linus in that regard), this would be best to quickly web-research to see which distros has the most coding-geeks in numbers I guess, to avoid spending time on less common. Mac is wanted of course.

[stugol]

I could use a regex, certainly, but it would be less robust, and more prone to failure. Are you sure it's a good idea?

Fortunately I have a Mac, so Mac support can be tested. But installing the LLVM and such on my Mac could break other things, and that would be a serious problem for me. Linux support can be tested in a VM, but Mac support has to be tested live, and I don't want to spend days reinstalling the damn operating system.

What about Windows support? I can help with the Win32 APIs - I have a lot of experience there. If you can make it compile, I can write a simple GUI framework for it.

I think perhaps the package managers are the important thing to support rather than the distros. apt and yum for example. There are undoubtedly others.

[stugol]

Ah. It would appear that I can run a MacOS VM, so this may not be a problem.

[ozra]

RegEx

Your call!

Mac

Awesome!

Windows

There's lots that need to be done to get win working (the inherited things from Crystal as usual...) - window support is highly important for adoption (and to be able to release one's apps on the platform of course), but I guess that will end up in post Phase 3 prio... A goto UI is definitely something one would need. Best, for time invested, would be to make a bridge to some existing cross platform one. GTK or such. Qt can be a beast because of wrapping transpiled-QT-C++ to C...

pkg managers

Yes, though, the packages can be named a bit different in debian, ubuntu, etc., also from version to version some times... :-/

[stugol]

pkg managers

I reckon yum, apt and rpm should be enough. People can always PR another one if they like ;)

The first version of the script will support apt only, though. Your Debian should be fine with it. Although I doubt you'd need to use it anyway. Your environment must already be set up for Onyx; plus it's clear you never used the original script ;)

Windows

Actually, a bare-bones Windows GUI framework could be written from the ground up in less than a week, supporting the most common controls [window, menu, button, listbox, listview, combobox, textbox, contextmenu, checkbox, optionbutton, panel, groupbox, scrollbar, tabcontrol, toolbar]. I've done it before.

Given the complexity (and questionable look-and-feel) of cross-platform GUIs, I think offering a simple native Windows library makes a lot of sense. Native GUIs always look better than cross-platform ones, and - most importantly - would be a hell of a lot easier to code with. People wouldn't need experience in Qt or GTK, they'd just have to read a short document. Provided they were familiar with Win32 GUI programming from C++ or such, they could just go ahead and use it; whereas I get the impression that these other frameworks are a bit more complicated.

[stugol]

I'm having trouble getting my script to build Onyx. I assume I'm doing something stupid, but I've never been very good with command-line compilation. Perhaps you could take a look?

Here's where we are:

# pwd
/home/infinity/onyx-lang
# ls
BACKERS.md    CHANGELOG_CRYSTAL.md  deps-tips.sh  LICENSE             onyx-bootstrap.rb  spec
bin           CHANGELOG_ONYX.md     docs          Makefile            README.md          src
bootstrap.sh  Contributing.md       etc           onyx-bootstrap.log  samples

And here's the command:

# CRYSTAL_CONFIG_PATH=/home/infinity/onyx-lang/src /opt/cr-ox/bin/cr-ox build --release -o .build/onyx src/compiler/onyx.cr

And here's the error:

Error in ./src/compiler/onyx.cr:2: while requiring "./onyx/**"

require "./onyx/**"
^

in ./src/compiler/onyx/syntax/parser.cr:5: while requiring "../../debug_utils/global_pollution"

require "../../debug_utils/global_pollution"
^

in ./src/compiler/debug_utils/global_pollution.cr:2: while requiring "wild_colors": can't find file 'wild_colors' relative to '/home/infinity/onyx-lang/src/compiler/debug_utils'

require "wild_colors"

[ozra]

I think it might just have been "working though it shouldn't" on my box. "wild_colors" is in src/, if you quickly wanna try it: require "../../wild_colors" in global_pollution might do the trick. But since require "anything" should look in stdlib, which is in "src/", it should work. So the compile might be looking at the wrong stdlib.

[stugol]

Error in ./src/compiler/onyx.cr:4: instantiating 'Crystal::OnyxCommand:Class#run()'

Crystal::OnyxCommand.run
                     ^~~

instantiating 'run(Array(String))'

in ./src/compiler/onyx/command.cr:46: instantiating 'Crystal::OnyxCommand#run()'

    new(options).run
                 ^~~

in ./src/compiler/onyx/command.cr:120: instantiating 'Crystal:Module#version_string()'

        puts "Onyx #{Crystal.version_string}"
                             ^~~~~~~~~~~~~~

in ./src/compiler/crystal/config.cr:4: expanding macro

    VERSION   = {{ env("CRYSTAL_CONFIG_VERSION") || `(git describe --tags --long 2>/dev/null)`.stringify.chomp }}
                ^

in ./src/compiler/crystal/config.cr:4: error executing command: (git describe --tags --long 2>/dev/null), got exit status 128

    VERSION   = {{ env("CRYSTAL_CONFIG_VERSION") || `(git describe --tags --long 2>/dev/null)`.stringify.chomp }}

Probably because:

# git describe --tags --long
fatal: No names found, cannot describe anything.

Can I make a suggestion here? How about you clone the git repo and try compiling it? When you've got that working, I can work on my part of the problem, which is the script ;)

[ozra]

Ah, edited above a bit late.

[ozra]

Yes, I'll have a look at that, gotta be tomorrow though.

The only way I tried the prior script was by running it in my current repo then and when to use it to update the "installed onyx version". Of course, as we already know, the script probably just worked on my machine :-/

[stugol]

Well the script is my problem; but the build command is yours ;)

[ozra]

Well, sleep train went without me, so I took a stab at it anyway.

I had that exact same problem the first time I compiled crystal. I'm no git expert (learn as I go for what I need) and I don't know why it doesn't get tags on the clone. In any event it's clearly not a reliable method. So I've added a script to bump version that sets it in source code and "git tag"s from that - instead of fetching from a git tag that may or may not be there. Also a lot of changes to bootstrap.sh, Makefile, and an added install.sh.

So try a fresh clone, and the current bootstrap script just to see what happens, will you?

[stugol]

You need to have at least one tag in the commit history in order for git describe to tell you the latest tag.

So that's why.

[ozra]

Yep. And tags don't seem to be fetched by clone.

[stugol]

Still getting while requiring "wild_colors": can't find file 'wild_colors' relative to '/home/infinity/onyx-lang/src/compiler/debug_utils'

[ozra]

When running make bootstrap? Any difference when running make onyx directly? (It must be some problem with which stdlib path it looks at). Could be I have some ENV-path set that "rescues" the situation here. I'll try a raw box tomorrow.

[stugol]

Actually, I was running the exact command I gave previously:

CRYSTAL_CONFIG_PATH=/home/infinity/onyx-lang/src /opt/cr-ox/bin/cr-ox build --release -o .build/onyx src/compiler/onyx.cr

Is that wrong?

Also, it would be really helpful if the build process (that is, crystal build) could output verbose messages. I know there's a --verbose flag, but it only lists a couple of things, and only after a huge delay. I want messages DURING the delay!

The script will work a lot better if the build process can give regular output of some kind. It doesn't matter what it outputs, just that it does.

[stugol]

Unfortunately, we'd have to build Onyx first, before we got those messages, because you can't arbitrarily change the Crystal build process, so... :(

[stugol]

Why not build Onyx with Onyx, though? Why build it with Crystal?

[ozra]

Try the new scripts, go from there, the old command skips building deps and all kinds of things.

Onyx is changing so rapidly atm, so it would be a bit unreliable atm, doing a "real bootstrap", that's why. It will be used later on.

[stugol]

I've managed to get it to build with your new script - after fixing it for 32-bit again, of course. So I'll resume work on my Ruby script. Shouldn't take long.

[ozra]

Oh it works? But is ruby dep really necessary then? I have some tooling on my machine that strictly requires a specific version of both ruby and some gems to work (fucking bull shit), so that's likely true for others too, so if ruby-dep can be avoided - less problems...

[stugol]

Well, it kinda works. But it only works for the 64-bit build, unless I edit the script. It needs to autodetect the correct platform. So...do you want me to write my Ruby script, or do you want to continue wading through the mire that is bashscript?

[ozra]

Well, what the fuck, as soon as Onyx is used for bootstrapping, all the build script should be in Onyx too - minimal deps and cleanest code :-) And then porting from ruby will be simpler. Go :-)

[stugol]

Alright, script is done. How the hell do I give you it? Do I really want to do a PR...? Sigh.

[ozra]

Haha, what? Yes, PR! :-)

[stugol]

Currently making the script more robust. And also install the Crystal prereqs.

[ozra]

Prereqs too? That's great!

[stugol]

PR ready. Script only works on Debian currently; but supports 32- and 64-bit; and both Debian 7 and 8. It will work on a freshly-installed system with no dependencies - except for git and ruby.

Should work with either Ruby 1.9 or the superior - but less common - Ruby 2.x.

[stugol]

FYI, compiling Onyx in 64-bit mode requires at least 2.5GB of RAM. This seems....excessive.

[ozra]

That will likely decrease with the new rewrite of the crystal compiler (latest master).

[stugol]

Onyx cannot be built anymore. I get:

# make all
g++ -c -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc `/usr/bin/llvm-config-3.5 --cxxflags`
cc -fPIC   -c -o src/ext/sigfault.o src/ext/sigfault.c
ar -rcs src/ext/libcrystal.a src/ext/sigfault.o
CRYSTAL_CONFIG_PATH=`pwd`/src ./bin/cr-ox build --threads 3 -o .build/onyx src/compiler/onyx.cr -D without_openssl -D without_zlib
/usr/local/bin/cr-ox: line 6: 30887 Killed                  "$INSTALL_DIR/embedded/bin/crystal" "$@"
make: *** [.build/onyx] Error 137

[stugol]

Any movement on this?

[ozra]

Onyx can't be built with cr 0.15 anymore, crystal segfaults(!). It works with latest master of crystal and it can also build itself (chicken or the egg...).