Stack overflow

[kevinresol]

Sometimes I get stack overflow (at compile time) that doesn't seems to be caused by infinite loops. Instead the stack looks legit but it is just too deep causing the overflow. It might be related to the compilation server. But I am not sure yet. I am not able to reproduce it in a predictable way for now. I will update this when I get to it.

[kevinresol]

Well, I found that when the overflow happens, restarting the compilation server will make it go away.

[kevinresol]

well, it may or may not be related to the compilation server.

But I have finally succeeded in reproducing it in a rather ugly way:

@:await class Main {
  @:await static function main() {
    var data = {
      b0: @:await bool(),
      b1: @:await bool(),
      b2: @:await bool(),
      b3: @:await bool(),
      b4: @:await bool(),
      b5: @:await bool(),
      b6: @:await bool(),
      b7: @:await bool(),
      b8: @:await bool(),
      b9: @:await bool(),

      b10: @:await bool(),
      b11: @:await bool(),
      b12: @:await bool(),
      b13: @:await bool(),
      b14: @:await bool(),
      b15: @:await bool(),
      b16: @:await bool(),
      b17: @:await bool(),
      b18: @:await bool(),
      b19: @:await bool(),

      b20: @:await bool(),
      b21: @:await bool(),
      b22: @:await bool(),
      b23: @:await bool(),
      b24: @:await bool(),
      b25: @:await bool(),
      b26: @:await bool(),
      b27: @:await bool(),
      b28: @:await bool(),
      b29: @:await bool(),

      b30: @:await bool(),
      b31: @:await bool(),
      b32: @:await bool(),
      b33: @:await bool(),
      b34: @:await bool(),
      b35: @:await bool(),
      b36: @:await bool(),
      b37: @:await bool(),
      b38: @:await bool(),
      b39: @:await bool(),

      b40: @:await bool(),
      b41: @:await bool(),
      b42: @:await bool(),
      b43: @:await bool(),
      b44: @:await bool(),
      b45: @:await bool(),
      b46: @:await bool(),
      b47: @:await bool(),
      b48: @:await bool(),
      b49: @:await bool(),

      b50: @:await bool(),
      b51: @:await bool(),
      b52: @:await bool(),
      b53: @:await bool(),
      b54: @:await bool(),
      b55: @:await bool(),
      b56: @:await bool(),
      b57: @:await bool(),
      b58: @:await bool(),
      b59: @:await bool(),

      b60: @:await bool(),
      b61: @:await bool(),
      b62: @:await bool(),
      b63: @:await bool(),
      b64: @:await bool(),
      b65: @:await bool(),
      b66: @:await bool(),
      b67: @:await bool(),
      b68: @:await bool(),
      b69: @:await bool(),

      b70: @:await bool(),
      b71: @:await bool(),
      b72: @:await bool(),
      b73: @:await bool(),
      b74: @:await bool(),
      b75: @:await bool(),
      b76: @:await bool(),
      b77: @:await bool(),
      b78: @:await bool(),
      b79: @:await bool(),

      b80: @:await bool(),
      b81: @:await bool(),
      b82: @:await bool(),
      b83: @:await bool(),
      b84: @:await bool(),
      b85: @:await bool(),
      b86: @:await bool(),
      b87: @:await bool(),
      b88: @:await bool(),
      b89: @:await bool(),
    }
  }

  @:async static function bool() (@:await int()) == 1;
  @:async static function int() return Std.int(@:await float());
  @:async static function float() return 1.2;
}

[kevinresol]

And here is another way to crash it:

(the recursive reference is not related, I just want some "complex" expressions)

typedef Api = {
  api:Api,
  main:Main,
}

@:await class Main {
  @:await static function main() {
    var api:Api = {
      api: null,
      main: new Main(),
    }
    api.api = api;

    var data = {
      b0: @:await api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.main.bool(),
      b1: @:await api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.main.bool(),
      b2: @:await api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.main.bool(),
      b3: @:await api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.main.bool(),
      b4: @:await api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.main.bool(),
      b5: @:await api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.main.bool(),
      b6: @:await api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.main.bool(),
      b7: @:await api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.main.bool(),
      b8: @:await api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.main.bool(),
      b9: @:await api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.main.bool(),
    }
  }

  public function new(){}

  @:async function bool() (@:await int()) == 1;
  @:async function int() return Std.int(@:await float());
  @:async function float() return 1.2;
}

[kevinresol]

In my real life code, I don't really have that deep field access, but my expressions are quite complex so it ran very deep in the macro.

For example:

var promotions = @:await @:futurize db.promotions.find({_id: {"$in": data.promotions}}, $cb);

I guess this expression is ~8-to-10-level deep in the transformation macro (e.g. db, db.promotions, db.promotions.find, data, data.promotions, {"$in": data.promotions}, etc). And I have like 10~15 of them in a single function.

And I guess that's why the stack overflows.

[kevinresol]

This also crashes (note that there is only one @await

typedef Api = {
  api:Api,
  main:Main,
}

@:await class Main {
  @:await static function main() {
    var api:Api = {
      api: null,
      main: new Main(),
    }
    api.api = api;

    var data = {
      b0: @:await api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.main.bool(),
      b1: api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.main.bool(),
      b2: api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.main.bool(),
      b3: api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.main.bool(),
      b4: api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.main.bool(),
      b5: api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.main.bool(),
      b6: api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.main.bool(),
      b7: api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.main.bool(),
      b8: api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.main.bool(),
      b9: api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.api.main.bool(),
    }
  }

  public function new(){}

  @:async function bool() (@:await int()) == 1;
  @:async function int() return Std.int(@:await float());
  @:async function float() return 1.2;
}

[benmerckx]

Thanks for examples! I'm off for two weeks though, so I'll a look after.

[kevinresol]

Is there any chance that this is a haxe issue? e.g. the macro interpreter isn't optimized so its stack overflows easily?

[benmerckx]

Well there might be a few things I can do so the calls aren't nested too deep, but I'm pretty sure in the end this is just a call stack limit which might be hard to get around.

[benmerckx]

Reading up on it there's multiple ways to avoid the recursion. Either loops or trampolines should do. I'll see about implementing a solution.

[kevinresol]

I uses the compilation server, sometimes the stack overflow problem can be mitigated by restarting the server. Seems something got stuck in the server...

[benmerckx]

This should be resolved in 0.1.5. It could run slower than before but should not hit the call stack limit as fast as before, if ever.

[kevinresol]

Thanks! I will try it for some time and report back.

[kevinresol]

Hm... it failed to build in my codebase after upgrading to b6ffda3 (was fine before):

Log too long, deleted, see next comment

[kevinresol]

Reproducible with my first code example in this thread.

Seems that it overflows if the object declaration is really long (or deep, in terms of expressions)

[benmerckx]

The code is now officially unreadable, but I can compile your first example with 0.1.6

[kevinresol]

Wow, it works fine for my really complex code now! Thanks!