Fast incremental recompile

[jedbrown]

As of current master (5b869e630b3535935a5b28e4d5498482586b6770),

$ touch lib/cg.jl
$ time make
julia --startup-file=no --project build.jl
[ Info: PackageCompiler: creating base system image (incremental=false)...
[ Info: PackageCompiler: creating system image object file, this might take a while...
cc -o main main.o -L/usr/lib -L. -ljulia -lm -Wl,-rpath,"/usr/lib:$ORIGIN" -lcg
249.878 real   247.404 user   2.529 sys   100.02 cpu

Can this be made fast (< 5 seconds, say)? It's about 2 seconds with present PETSc (after editing a *.c source file).

[simonbyrne]

Not at the moment, but Revise.jl can do live-refresh in a running process, I suspect something similar that updates an object file could be made to work.

[jedbrown]

Refresh in a running process is interesting, but I think a different use case. Suppose I have some user code (written in C or Fortran) that is failing and I want to try a fix in the library. Is a 4 minute edit-compile-debug latency unavoidable?

Can the Julia compilation be parallelized? Building (C) PETSc from scratch takes 11 seconds on my server node (versus 2 minutes on laptop), but this Julia example takes about the same time either way.

I realize the requirements are different with non-Julia clients, but how do those of you who work on large projects (like CLIMA) typically work? Are you using Revise.jl or do you somehow factor packages for fast incremental updates?

[simonbyrne]

Refresh in a running process is interesting, but I think a different use case. Suppose I have some user code (written in C or Fortran) that is failing and I want to try a fix in the library. Is a 4 minute edit-compile-debug latency unavoidable?

Yes, what I was thinking is that it might be possible to leverage the same tracking/invalidation that Revise uses, so that julia is able to update the resulting object file in place (and then retrigger building the shared library from that). This is a beyond my expertise though, @keno might be able to give further ideas.

Can the Julia compilation be parallelized? Building (C) PETSc from scratch takes 11 seconds on my server node (versus 2 minutes on laptop), but this Julia example takes about the same time either way.

Not at the moment, as far as I know. There is considerable effort at the moment in improving performance of the compiler, I'm not sure if parallelism or multi-threading is part of that though.

I realize the requirements are different with non-Julia clients, but how do those of you who work on large projects (like CLIMA) typically work? Are you using Revise.jl or do you somehow factor packages for fast incremental updates?

For most cases of interactive development, Revise.jl works very well, though it is obviously limited to programs that can be run interactively (so no MPI), and occasionally it can break (e.g. when using a lot of metaprogramming).

We also typically build a custom system image of all our dependencies, which significantly reduces latency in package loading: https://clima.github.io/ClimateMachine.jl/latest/DevDocs/SystemImage/ this works very well (though you need to rebuild the system image if you modify dependencies).

[Keno]

There's a couple of different modes that could potentially be interesting here. One thing to do might be to just have a development mode where the shared library automatically checks if any of the source got modified (via Revise) and if so patches itself with any changes. That way you don't need an explicit recompile at all (though of course you'd be paying increased JIT time the father the changes are from when the library was originally built). Incrementally updating the system image is also possible, but not really something people have looked into much. For deployment, you do definitely want to do a full/clean rebuild, since we make heavy use of what is essentially LTO (just as you would for a modern C++ app). I think for development, the thing to do is just figure out what the easiest workflow is. There's some implementation work required also, but I don't think it's that bad, since Revise basically already knows how to do this.

[RoyiAvital]

Is there a configuration to minimize the JIT overhead for those kind of situations?