Tools for 2017: LuaJIT, DynASM, R, Nix, Pharo

[lukego]

Here is a little braindump on the programming tools I am excited about using in 2017 for my ongoing Snabb hacking.

• LuaJIT for general purpose system programming. Simple high-level language, blazing fast JIT, and fantastic FFI. I suspect that LuaJIT is the root of a new family tree of trace-based language implementations, analogous to MACLISP in the 1960s, and I am eager to participate in the evolution.
• DynASM (Lua-mode) assembler for accessing special x86 instructions like CRC32, AES-NI, RDPMC, SIMD, etc. This is a macro-assembler that is powerful and operates Just-In-Time. I enjoy writing assembler code directly, referring to manuals from Intel and Anger Fog, much more so than writing it indirectly using C compiler intrinsics.
• R for data science and for statistical modeling. Lately it seems like every interesting test or benchmark produces thousands of data points to interpret, for example from a matrix of platforms/versions/configurations/workloads. I find ggplot2 and dplyr especially versatile and pleasant to work with.
• Nix for automation. Nix takes a task - build a program, deploy a server, run a test, run a series of benchmarks, etc - and makes it scalable and repeatable. This is indispensible for running large software testing campaigns: scheduling the builds, archiving the raw results, post-processing to create reports, and making each case immediately reproducible. It's also great for programming-in-the-small like avoiding dependency hell for shell scripts.
• Pharo for inspection. This is speculative: I want to build application-specific development tools with a frontend based on Glamour and a backend based on Nix that can call out to the plethora of tools in nixpkgs (R for stats, gas for disassembly, gdb to locate objects via DWARF, wireshark to decode network traffic, etc.)

I am enjoying working with a collection of complementary domain-specific tools. Gonna be another fun year of hacking :-).

[Etiene]

You can use LuaJIT for data science as well! http://torch.ch By searching around I found that data scientists with Computer Science background rather than, let's say, Economy, usually prefer Lua or Python over R.

[lukego]

@Etiene I peeked at Torch before R, because of the LuaJIT connection, but it didn't take. Seems like the community is smaller, more focused on machine learning, and doing most work behind closed doors. I have since come to really love R and RStudio :).

[Etiene]

fair enough! :) I did get loads of help with Torch on their gitter chat though, seems more active than the email list.

[lukego]

@Etiene Good tip about Gitter. I didn't know about that so I assumed that quiet mailing list = quiet project. This is also something to reflect on for our Snabb project: we are active on Github and Slack but our mailing list is idle these days. Maybe we should delete it to avoid misleading people!

[akostrikov]

Hi, Luke! Any progress with pharo? Is it worth it?

[lukego]

@akostrikov Yes. I'm spending most of my time hacking in Pharo lately. I am a huge fan. I'm mostly using it for graphically inspecting application data structures (from a JIT compiler). See https://github.com/studio/studio/pull/96.

I did have some trouble getting started for three reasons.

First, Pharo always has a stable version and a development version, and it seems best to stick with the stable version. The development version is more exciting, and it's all that anybody talks about on the mailing lists, but it changes fast and that's a distraction from building your own application.

Second, it's hard to build Pharo images within my build environment (Nix.) The basic problem is that Pharo wants to be able to run around the internet downloading software dependencies but Nix runs in a locked down sandbox with no network access. The compromise that I settled on is to build in two parts. First I use the Pharo Jenkins infrastructure to build a "base image" that includes all of the third party libraries that I want to use. Then I build an "application image" using Nix and this only loads my own code in a sandbox-friendly way. So over time I rebuild the base image only every few months (to pull updates and add new libraries) but I also rebuild the application image every day while hacking and testing my code.

Third, it took a while to make Smalltalk code live happily inside a subdirectory of a Git repository. I am saving my source code from the image onto files using Monticello. I started off using the "filetree" backend but this was awkward: the code is split into many tiny files (one per method) and includes metadata (e.g. timestamp when I exported the code) which always conflicts between branches. The solution was to use the new "tonel" format that writes to disk in a more suitable format (one file per class and no noisy metadata.)

So these days I am hacking away very productively and everything is very smooth and stable. I'm really enjoying working with frameworks like GTInspector, Spotter, and Roassal.

[akostrikov]

Thanks, Luke! I am going to try Studio.