Macro becomes very slow

[kevinresol]

The tests took >30s to compile on my MBP:

The --times report shows that the bottleneck is from finalize and flush, and I have no idea what it is.

Kevins-MacBook-Pro:tink_sql kevin$ haxe -lib tink_sql tests.hxml -lib travix -js bin/node/tests.js -lib hxnodejs --times -D macro-times
name                  | time(s) |   % |     # | info
-----------------------------------------------
analyzer              |   0.170 |   0 | 11335 | 
filters               |   0.150 |   0 |     1 | 
generate              |   0.030 |   0 |     1 | 
  js                  |   0.030 |   0 |     1 | 
macro                 |  41.060 |  97 |   449 | 
  execution           |   0.570 |   1 |   385 | 
    build             |   0.020 |   0 |     1 | ANSI
    run               |   0.000 |   0 |     1 | DbFixture
    include           |   0.000 |   0 |     1 | _hxnodejs.VersionWarning
    allowPackage      |   0.000 |   0 |     1 | haxe.macro.Compiler
    library           |   0.000 |   0 |     1 | jdbc.mysql.Include
    build             |   0.290 |   1 |   273 | tink.SyntaxHub
    use               |   0.000 |   0 |     1 | tink.SyntaxHub
    use               |   0.000 |   0 |     1 | tink.await.Await
    resolveDirectType |   0.000 |   0 |    23 | tink.macro.Types
    join              |   0.010 |   0 |     2 | tink.sql.Dataset
    where             |   0.020 |   0 |     7 | tink.sql.Dataset
    on                |   0.010 |   0 |     4 | tink.sql.JoinPoint
    init              |   0.000 |   0 |     4 | tink.sql.TableSource
    build             |   0.000 |   0 |     1 | tink.sql.macros.DatabaseBuilder
    build             |   0.050 |   0 |     4 | tink.sql.macros.TableBuilder
    assert            |   0.020 |   0 |    29 | tink.unit.Assert
    use               |   0.000 |   0 |     1 | tink.unit.AssertionBufferInjector
    make              |   0.000 |   0 |     1 | tink.unit.TestBatch
    build             |   0.150 |   0 |     3 | tink.unit.TestBuilder
    make              |   0.000 |   0 |     3 | tink.unit.TestSuite
    assert            |   0.000 |   0 |    23 | tink.unit._AssertionBuffer.AssertionBuffer_Impl_
  finalize            |  26.640 |  63 |    21 | 
  flush               |  13.790 |  33 |    22 | 
  typing              |   0.060 |   0 |    21 | 
    build             |   0.000 |   0 |     1 | ANSI
    run               |   0.000 |   0 |     1 | DbFixture
    include           |   0.000 |   0 |     1 | _hxnodejs.VersionWarning
    allowPackage      |   0.000 |   0 |     1 | haxe.macro.Compiler
    library           |   0.000 |   0 |     1 | jdbc.mysql.Include
    build             |   0.000 |   0 |     1 | tink.SyntaxHub
    use               |   0.000 |   0 |     1 | tink.SyntaxHub
    use               |   0.000 |   0 |     1 | tink.await.Await
    resolveDirectType |   0.000 |   0 |     1 | tink.macro.Types
    join              |   0.000 |   0 |     1 | tink.sql.Dataset
    where             |   0.000 |   0 |     1 | tink.sql.Dataset
    on                |   0.000 |   0 |     1 | tink.sql.JoinPoint
    init              |   0.050 |   0 |     1 | tink.sql.TableSource
    build             |   0.000 |   0 |     1 | tink.sql.macros.DatabaseBuilder
    build             |   0.000 |   0 |     1 | tink.sql.macros.TableBuilder
    assert            |   0.000 |   0 |     1 | tink.unit.Assert
    use               |   0.000 |   0 |     1 | tink.unit.AssertionBufferInjector
    make              |   0.000 |   0 |     1 | tink.unit.TestBatch
    build             |   0.000 |   0 |     1 | tink.unit.TestBuilder
    make              |   0.000 |   0 |     1 | tink.unit.TestSuite
    assert            |   0.010 |   0 |     1 | tink.unit._AssertionBuffer.AssertionBuffer_Impl_
other                 |   0.040 |   0 |     1 | 
parsing               |   0.330 |   1 |   364 | 
typing                |   0.490 |   1 |    53 | 
-----------------------------------------------
total                 |  42.270 | 100 |     0 | 

[back2dos]

I have no idea. What's your Haxe version?

[kevinresol]

3.4.2

[kevinresol]

With -v I see the log stayed at Typing macro tink.sql.macros.TableBuilder.getInt for a long time. @simn can you give us some directions?

[Simn]

Try with https://github.com/HaxeFoundation/haxe/pull/6319

[back2dos]

@Simn I can't wait to do that, but as of late recent versions just break everything for me. Some hints as to what "flush" and "finalize" mean, so we can try to fix it for the latest stable?

[Simn]

Unfortunately, these times are not exactly accurate. "finalize" basically means "typing", and "flush" basically means "adding stuff to macro context".

[kevinresol]

I got this in latest haxe (https://github.com/HaxeFoundation/haxe/commit/96a60cb) and eval haxe (https://github.com/Simn/haxe/commit/6d7ca60c70f30fcca3f25a3fc80ac3cfd9af651b)

/Users/kevin/Codes/tink_sql/tests/Db.hx:44: characters 9-13 : tink.sql.Table0<Unknown<0>> has no field init

Apparently macro method is not inherited by child class.

[kevinresol]

Let alone the above error, the compiler still pauses at the line

Typing macro tink.sql.macros.TableBuilder.getInt

for a long time.

Does that literally mean the compiler spent a lot of time typing the getInt function?

[Simn]

Any instructions on how to reproduce this?

I don't get very far trying to run the tests:

C:\GitHub\tink_sql>haxelib run travix neko
> haxe -lib tink_sql tests.hxml -lib travix -neko bin/neko/tests.n
tests/DbFixture.hx:21: characters 7-19 : Type not found : sys.db.Types

I tried adding -lib record-macros to tests.hxml but that didn't help.

[Simn]

It's probably not the typing itself but something that happens afterwards and doesn't have a dedicated print.

[kevinresol]

This would be tricky, because it depends on the git versions of a few libraries. Let me try to come up with a reproducible example.

[Simn]

My entire haxelib repo is git versions, so that's not a problem.

[kevinresol]

If you don't mind using lix It should be as simple as:

git checkout pure
npm install
npm run travix node .

[Simn]

I probably don't mind but you'll have to tell me what lix is.

[kevinresol]

https://github.com/lix-pm/lix :smile:

[kevinresol]

But I think you don't need to do anything except running the three lines of code I quoted.

[Simn]

It fails because I'm not running a mysql server...

[kevinresol]

delete fixture.n

comment out DbFixture (non-macro part) like so:

// import sys.db.*;
// import sys.db.Types;

class DbFixture { 
  // static function connect(db, cb) {
  //   var cnx = Mysql.connect( { host: '127.0.0.1', user: 'root', pass: '', database: db } );
  //   cb(cnx);
  //   cnx.close();
  // }
  // static function clear() {
  //   connect('mysql', function (cnx) {    
  //     cnx.request('DROP DATABASE IF EXISTS test');
  //     cnx.request('CREATE DATABASE test');
  //   });
  // }
  static function main() {
    // clear();
  }
}

[Simn]

Now it does something. How do I make it use my Haxe version? It seems to bundle 3.4.2 and use that by default.

[kevinresol]

on my mac I do this:

ln -s /usr/local/lib/haxe ~/haxe/versions/custom

this links the haxe folder, so that the haxe executable is inside ~/haxe/versions/custom

then edit .haxerc and change to "version": "custom"

[back2dos]

The easiest way would be npm install switchx -g and then switchx install 00b70f9 or something (it has to be the short hash used on the nightly builds). Or as Kevin suggests, you can dump it where it is looked up and edit your .haxerc file to reference it.

[Simn]

Thanks, that works I guess. I can reproduce the error you're getting, gonna bisect that.

[Simn]

The pattern matcher struggles with this switch: https://github.com/haxetink/tink_sql/blob/pure/src/tink/sql/macros/TableBuilder.hx#L66

I guess the extractors cause some exponential code explosion. I'll see what I can do, but it might not be so easy to change this. Maybe you could try rewriting that switch to see if it is indeed the main problem.

[kevinresol]

Thanks. I pushed a commit that seems to fix the speed issue. Closing for now.