Other targets than JavaScript - neko very slow

[cancerberoSgx]

Hello @Tw1ddle ,

This is a question about executing the API in another target than JS and neko and also a heads up since in neko it runs very slow.

I'm writing a simple program that shows the whole user experience using haxe API. I want to be generic so the code works not only in JavaScript target but in other too.

My motivation for this is measuring the performance of real world use and writing a small documentation on how to use the haxe API since there's none.

See below a working example that creates a bitmap and generates SVG and the .hxml files used to run them in js and neko.

This kind of example, even with really big images, work fine and fast in node.js but in neko is very slow. An empty bitmap of 100x100 with default options and 20 iterations takes 8 seconds while in node takes 0.3 seconds.

I'm not familiar with neko nor I don't know if this is serious but I really would like to execute it in another target then JavaScript and so far I couldn't. I tried also in python, java, and cppia , but both throws errors, python with floats being used as array index, and Java with a symbol not being found. Probably I'm missing something...

My question is:

Were you able to execute the library in another target then js and neko ?

I wanted to know before I try in another languages... Thanks

package issue;

import geometrize.bitmap.Bitmap;
import geometrize.bitmap.Rgba;
import geometrize.runner.ImageRunner;
import geometrize.exporter.SvgExporter;
import geometrize.runner.ImageRunnerOptions;

class IssueNeko {
    public static function main():Void {
        var iterations = 20;
        var bitmapSize = 100;

        var bitmap = Bitmap.create(bitmapSize, bitmapSize, Rgba.create(255, 255, 255, 255));
        // painting a red rectangle will make it much much slower but instead is slow wihhout the rectangle
        for (i in Math.round(bitmapSize / 5)...Math.round(bitmapSize / 2)) {
            for (j in Math.round(bitmapSize / 5)...Math.round(bitmapSize / 2)) {
                bitmap.setPixel(i, j, Rgba.create(255, 0, 0, 255));
            }
        }
        var runner = new ImageRunner(bitmap, Rgba.create(255, 255, 255, 255));
        var options:ImageRunnerOptions = {
            shapeTypes: [0, 1, 2, 3, 4],
            alpha: 128,
            candidateShapesPerStep: 50,
            shapeMutationsPerStep: 100
        };
        var svgData = [];
        for (i in 0...iterations) {
            var step0 = Date.now().getTime();
            var result = runner.step(options);
            trace(Date.now().getTime() - step0);
            svgData.push(SvgExporter.exportShapes(result));
        }
        var svg = SvgExporter.getSvgPrelude() + SvgExporter.getSvgNodeOpen(bitmap.width, bitmap.height) + svgData.join('\n') + SvgExporter.getSvgNodeClose();
        trace(svg);
    }
}

[cancerberoSgx]

I just saw https://github.com/Tw1ddle/geometrize-haxe/issues/5 so I will try C++ as suggested. I'm keeping this open though since the neko performance problem, but if you think it's not feel free to close it. Thanks

[Tw1ddle]

Hi, I find neko is quite slow, certainly is when I'm testing other apps. It's probably just not very optimized.

I've used the cpp target in the past with no issue when I was coding a demo using HaxeFlixel: https://github.com/Tw1ddle/geometrize-haxe-demo (that was quite a long time ago so a lot may have broken in the meantime, haha).

I had it compiling on most targets at least. When I find time I'll try fleshing out these stubbed unit tests and see what starts breaking - https://github.com/Tw1ddle/geometrize-haxe-unit-tests/blob/master/.travis.yml#L18

[cancerberoSgx]

Hi, I find neko is quite slow, certainly is when I'm testing other apps. It's probably just not very optimized.

If I pass this project logo.png as input, and candidateShapesPerStep: 2, shapeMutationsPerStep: 4, around 2-5 iteration it won't returns from runner step(). Probably looping in a recursion or iteration - no idea/ much time to debug / trace where... I notice haxe number types, and the standard libraries are "special" and since it the issue happens with complex images and not so with blank images I suspect it's a problem with floating point implementation or some casting to int / float...

My first impression with haxe was excellent from the POV of the language, compiler and target languages, macros, etc, but not so good regarding standard libraries, and the number types. In python for example, Math.round will still return a float - I needed to use Std.int instead. Date.now() or Sys.time returns seconds as float but Sys.time()*1000.0 returns ms correctly... well just getting used to it - if I make it work the C++ target I will be much happier :)

I've used the cpp target in the past with no issue when I was coding a demo using HaxeFlixel: https://github.com/Tw1ddle/geometrize-haxe-demo (that was quite a long time ago so a lot may have broken in the meantime, haha).

Oh didn't knew that one, I'm cloning...

The tests I always end writing are not so "unit" and more like integrations so if I had other haxe project probably won't use it and end up with something like node.js and another target like C++ or Java.

Right now I'm kind of blocked trying to read a PNG - I'm using imagemagick to convert images to .rgba form command line first so I can test (another alternative is drawing the bitmap). But I would like to make it work with pure haxe so I can write a small tutorial about using the API.

Tried a couple of libraries (like format) but the only test they have is for other ARGB ad even with that one I couldn't make it work :( will try again tomorrow and check your demo how/if you are doing.

thaks

[cancerberoSgx]

Running unit tests C++ target works with the command

haxelib run travix cpp -lib hxcpp

Now trying to run without travix - thanks!

(the rest don't in my system - will try later in another OS - but with C++ I'm fine)

[Tw1ddle]

I think Neko uses 31-bits (not 32) for integers by default, that's probably causing various issues. I'll try switching all the Ints with Int32 and debugging that at somepoint :smile:

I had a go at fixing up the demo (this one again: https://github.com/Tw1ddle/geometrize-haxe-demo), it's green on Travis for flash/html5/linux c++ and works for me on Windows.

[cancerberoSgx]

I think Neko uses 31-bits (not 32) for integers by default. I'm having fun and learning a lot, now testing in linux.I think Neko uses 31-bits (not 32) for integers by default, that's probably causing various issues. I'll try switching all the Ints with Int32 and debugging that at somepoint

31 bits... ? Well I think it's good to test there are less casualities and heuristics.. ;)

In the library code I didn't see conditions that could cause a infinite loop, but there are some parts of the rasterizer that maybe we can review. And perhaps is not your code at all. using lime it behaves better although still slow it doesn't doesn't block.

I don't think supporting neko that is important, could be better to test other targets maybe. Also runing with lime it works better, slow but does't hang (lime test neko).

As totally new with haxe I think is understandable that haxe team didn't focus on their own interpreter since code generation for mature runtimes technologies works well and fast. I would love to see targets for rust or go. It also supports emscripten which I didn't tried yet. Same for lua.

An update on my research (so you don't surprise) :) :

Learning in your project about lime and using it II succeed loading images, processing and measuring with Cpp. Is almost doing what I want so jsut in case I don't loose it I'm committing in this repo in a branch. May be it does not belong here, just tell me and I can move it.

They run with the UI/Window hidden and no application so are command line and. Quoting from a little readme I made:

These tests are not meant to be executed in travis. The idea is to be personal developer tests to compare, in the same machine, the evolution of speed in the short term when implementing / fixing important things.

For me having this kind of things when making big changes is very useful, but again we can move this to another place.

In the future we can enable the UI (in Cpp) to show the results and research optimal ways showing the evolution of steps in different technologies (gl, cairo, canvas, svg, html, etc).

I didn't compile your demo since I needed to read about lime and the other framework first. Now I kind of understand them.

Thanks!

[Tw1ddle]

In the end I've focused my attention on the C++ implementation (https://github.com/Tw1ddle/geometrize-lib) and used it in my desktop version of this. Performance is generally a lot better in C++ land and I use multithreading there.

I'm not planning to work on optimizing performance for the Haxe library (and certainly not Neko, we'd prefer the Hashlink target nowadays for sure) so I'm going to close this!

Appreciate you helping out :smile: