Add Gradle dependencies from javatools to the XDK xtc module (that can be disabled in the build env)

[lagergren]

Add Gradle dependencies from javatools to the XDK xtc modules.

This dependency currently does not exist, and this is deliberate, because it's not an ergonomic experience to change one line that gets javatools.jar to rebuild, if it leads to the recompilation of the entire XDK platform. Strictly speaking, this should be the case, since changes in javatools.jar, at least for the Compiler, affects how the XDK builds itself, and should result in having to rebuild all XTC modules.

However - this causes no ends of questions and confusion when people run ./gradle clean, or just need to get the XDK rebuilt after changing something in javatools.jar. I have repeatedly explained that we do want to use an incremental build, and that we for all practical purposes never really should have to run 'gradlew clean', and that if we do, it's not the fault of gradle clean that we rebuild with cached (now stale) resources. Since I get this question or "bug reports" several times a week, no matter how much I explain it, I propose that I set up this missing input dependency, so that the XDK WILL actually rebuild all XDK modules at the slightest change in javatools.jar. Then every part of the build lifecycle should be incremental and work as Gradle intended with the build cache.

In order to preserve the current use case, where we very quickly want to run an XTC module that we know doesn't need to be rebuilt itself, after touching javatools.jar, I'll move the old behaviour into a settable state, e.g. an environment variable, which has to be explicitly modified for the build to ignore the dependency between javatools.jar and the xtc modules.

This, in turn, should provide us with a build that we 1) Know is sound and works incrementally 2) Gives us a stable foundation to improve build speed and even more incremental behaviour.

I have asked what kind of changes are the most typical ones in javatools.jar, to see if we can split javatools up into separate modules with separate dependencies. I have a feeling that compilation and runtime are quite different domains, and that the only thing the XTC modules need to build is an up to date compiler. Hence, we can depend only on that in a future version, which would get rid of the bottleneck completely if we are only modifying Runner code.

However, the javatools are pretty monolithic, because data structures are reused for all the launchers, but also because the Runner has a large amount of dependencies to the compiler, and probably requires the compiled to be built fresh in all modes where xec can compile any source code. Again - I have trouble seen the benefits of the xec source code files use case, because as a user, I would think that a runtime should not modify any previously build code of the application, if I have finished an up to date compile. This, however, is the case, and as long as we support that, it's going to be tricky to separate out the Javatools jar functionality into different subgroups that can act as different dependencies.

Suggestions?

[ggleyzer]

I'm not sure why you say there is no dependency "from javatools to the XDK xtc modules". The way the 'build' behaves now is that if you touch a file that belongs to javatools module, all Ecstasy libraries are in fact attempted to be recompiled. You can trivially verify that but adding a single space to any java file (I just tried it with Component.java). The .gradlew build --info command shows that every library is recompiled like this:

Starting process 'command '/Library/Java/JavaVirtualMachines/zulu-21.jdk/Contents/Home/bin/java''. Working directory: /Users/ggleyzer/Development/xvm/lib_ecstasy Command: /Library/Java/JavaVirtualMachines/zulu-21.jdk/Contents/Home/bin/java -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant -ea -cp /Users/ggleyzer/Development/xvm/javatools/build/libs/javatools-0.4.43.jar org.xvm.tool.Compiler -o /Users/ggleyzer/Development/xvm/lib_ecstasy/build/xtc/main/lib -r /Users/ggleyzer/Development/xvm/lib_ecstasy/build/xtc/main/resources --set-version 0.4.43 -L /Users/ggleyzer/Development/xvm/lib_ecstasy/build/xtc/main/lib /Users/ggleyzer/Development/xvm/javatools_turtle/build/xtc/main/resources/mack.x /Users/ggleyzer/Development/xvm/lib_ecstasy/src/main/x/ecstasy.x

It is true that since there is no -rebuild flag, the Ecstasy compiler doesn't do anything, but that is a feature of the compiler, not the gradle build.

With that in mind, I'm not sure what are you proposing... What would that new environment variable control?

[cpurdy]

Perhaps the Ecstasy compiler should consider the timestamp on all of the compiler libraries themselves as part of the input to its "should I rebuild?" decision ...

[ggleyzer]

But as you correctly pointed out, a change [however rare] in the compiler may require a full rebuild even if no timestamp or version would point to it

[cpurdy]

That's what I meant: The compiler should look at the timestamp on the compiler itself! (Or alternatively, every potentially-needing-recompile change should increment the binary version inside the compiler so that it is forced to rebuild?)

[ggleyzer]

I'm not following. Are you saying than any change that may require a full rebuild would have to bump this internal version?

[ggleyzer]

I think we're discussing items with extremely low ROI. Add a flag for what we used to use 'clean' task and we're done

[lagergren]

I'm not sure why you say there is no dependency "from javatools to the XDK xtc modules". The way the 'build' behaves now is that if you touch a file that belongs to javatools module, all Ecstasy libraries are in fact attempted to be recompiled. You can trivially verify that but adding a single space to any java file (I just tried it with Component.java). The .gradlew build --info command shows that every library is recompiled like this:

Starting process 'command '/Library/Java/JavaVirtualMachines/zulu-21.jdk/Contents/Home/bin/java''. Working directory: /Users/ggleyzer/Development/xvm/lib_ecstasy Command: /Library/Java/JavaVirtualMachines/zulu-21.jdk/Contents/Home/bin/java -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant -ea -cp /Users/ggleyzer/Development/xvm/javatools/build/libs/javatools-0.4.43.jar org.xvm.tool.Compiler -o /Users/ggleyzer/Development/xvm/lib_ecstasy/build/xtc/main/lib -r /Users/ggleyzer/Development/xvm/lib_ecstasy/build/xtc/main/resources --set-version 0.4.43 -L /Users/ggleyzer/Development/xvm/lib_ecstasy/build/xtc/main/lib /Users/ggleyzer/Development/xvm/javatools_turtle/build/xtc/main/resources/mack.x /Users/ggleyzer/Development/xvm/lib_ecstasy/src/main/x/ecstasy.x

It is true that since there is no -rebuild flag, the Ecstasy compiler doesn't do anything, but that is a feature of the compiler, not the gradle build.

With that in mind, I'm not sure what are you proposing... What would that new environment variable control?

There is no explicitly declared dependency between the javatools and the XTC modules. This is to avoid rebuilding everything when we touch something in javatools. A sound build would have that behaviour. So I'm asking if there is a natural granularity in javatools so that you can be sure that you have an area where changes won't affect how the XTC platform is recompiled when you change them. Those would not need to be a declared dependency. Others would. That would make it faster.

[ggleyzer]

If there is no "explicitly declared dependency between the javatools and the XTC modules", why does ./gradlew build recompiles all the libraries?

[lagergren]

I think we're discussing items with extremely low ROI. Add a flag for what we used to use 'clean' task and we're done

There is no such flag. There never was such a flag. Gradle clean deletes build directories. This is what you used to do. Then it rebuilt everything in series, based on manual time stamp checks on source files. If Gradle believes that identical build directories in the cache can be re-used, it will reuse them. If it believes that if all inputs and outputs known to gradle for a task do not seem to change between runs. If they do in a way that matters, and Gradle isn't told, then Gradle will break.

Gradle does not allow you to implement a flag like the one you describe, unless it's by turning off the build cache, in which case you will rebuild every deleted build directory from clean when you rerun the build.

There is no clean with the build cache on.

There has never been a clean with the build cache on.

You can't run caching Gradle, and expect clean to not reuse caches if build caching is enabled. The best you can do is to use the --rerun-tasks and (more rarely) the --recompute-dependencies flag, and the former probably covers pretty most of these use cases, but since the build isn't sound it's impossible to tell 100% that it's a remedy for our issues. For issues I face, typically i get cached results that shouldn't be there, and by checking the scan and understanding this, I know if --rerun-tasks will get me back in a working state.

But there is no clean flag.

If you don't want to have false cache hits, we need to declare dependencies between javatools and the xdk build, or we need to turn off the build cache.

There is a huge return on investment to make the build better and more incremental and sounder since:

1) I spend most of my time explaining and reexplaining this and I still cannot adequately get across why deleting build dirs with gradle clean does not rebuild everything from scratch, why the build cache gets in the way, why the build cache is a good thing, and that the --rerun-tasks and --recompute-dependencies flags exist, and to some extent can work around this. But you have to understand what is happening and why. These flags are not a full substitute for a sound build, since there is a dependency and we haven't told the build about it.

2) But it's not just my time - a working incremental build gets incrementally better. We can turn on the configuration cache once we have laid things in place for that to increase the speed even for that. I think it's hard to get anywhere unless we make the build initially sound though. Experiments with the configuration cache seem to turn XDK module rebuilds into instantaneous and correct cache deductions, and configuration is where we spend most of our time right now. However - if the finest way we can express granularity is "all of java tools", then this is a complex and likely not too successful of an undertaking.

3) Hence, I ask the question if there we can look at what parts of javatools are needed to build the XDK in some novel way, because we certainly do not use everything in it to build the XDK. Far from it. Could we find such "natural" sub-dependencies, the ROI would be a sound and fast incremental build.

[lagergren]

I'm not sure why you say there is no dependency "from javatools to the XDK xtc modules". The way the 'build' behaves now is that if you touch a file that belongs to javatools module, all Ecstasy libraries are in fact attempted to be recompiled. You can trivially verify that but adding a single space to any java file (I just tried it with Component.java). The .gradlew build --info command shows that every library is recompiled like this:

Starting process 'command '/Library/Java/JavaVirtualMachines/zulu-21.jdk/Contents/Home/bin/java''. Working directory: /Users/ggleyzer/Development/xvm/lib_ecstasy Command: /Library/Java/JavaVirtualMachines/zulu-21.jdk/Contents/Home/bin/java -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant -ea -cp /Users/ggleyzer/Development/xvm/javatools/build/libs/javatools-0.4.43.jar org.xvm.tool.Compiler -o /Users/ggleyzer/Development/xvm/lib_ecstasy/build/xtc/main/lib -r /Users/ggleyzer/Development/xvm/lib_ecstasy/build/xtc/main/resources --set-version 0.4.43 -L /Users/ggleyzer/Development/xvm/lib_ecstasy/build/xtc/main/lib /Users/ggleyzer/Development/xvm/javatools_turtle/build/xtc/main/resources/mack.x /Users/ggleyzer/Development/xvm/lib_ecstasy/src/main/x/ecstasy.x

It is true that since there is no -rebuild flag, the Ecstasy compiler doesn't do anything, but that is a feature of the compiler, not the gradle build. With that in mind, I'm not sure what are you proposing... What would that new environment variable control?

There is no explicitly declared dependency between the javatools and the XTC modules. This is to avoid rebuilding everything when we touch something in javatools. A sound build would have that behaviour. So I'm asking if there is a natural granularity in javatools so that you can be sure that you have an area where changes won't affect how the XTC platform is recompiled when you change them. Those would not need to be a declared dependency. Others would. That would make it faster.

If this is true, then I don't understand where our Gradle breakages come from?

[ggleyzer]

As you can trivially test, a trivial change will make build to recompile all the XDK libraries without -rebuild flag. As such, the compiler sees that the libraries are "up to date" and skip the compilation. So the "breakage" comes from the fact that some changes in the compiler may require a force rebuild even though the libraries seem up to date.

[ggleyzer]

Another scenario I have encountered quite often is switching from one branch to another. On many occasions, the simple build would not work correctly and in the absence of any other solution I would do clean, which would always fix the following build

[lagergren]

Another scenario I have encountered quite often is switching from one branch to another. On many occasions, the simple build would not work correctly and in the absence of any other solution I would do clean, which would always fix the following build

This is an interesting observation. If you bump into these, could you save some build scans and figure out what it things it can reuse and what it's actually not properly reusing? You have a better idea of the requirements involved and why we don't pick them up.

[lagergren]

As you can trivially test, a trivial change will make build to recompile all the XDK libraries without -rebuild flag. As such, the compiler sees that the libraries are "up to date" and skip the compilation. So the "breakage" comes from the fact that some changes in the compiler may require a force rebuild even though the libraries seem up to date.

"Without" the rebuild flag? Wait - the the rebuild flag is disabled by default. You need to set forceRebuild=true in a configuration to get the plugin to pass the rebuild flag.

Are you saying that not using the rebuild flag, which is currently the case for the build as far as I know, will not check if any xtc modules need to be updated? Even if I send in an explicit compile request because I know that the source code has changed? Because Gradle never runs a compile task unless its inputs have changed, for example something in the source set. It will consider the compile output cached. (It also looks at flags, configuration and various other things, of course, and a lot of it is implicit and not something we have to implement)

Does this mean that I potentially don't recompile changed source code if I don't use the rebuild flag, even with command line launched compile jobs? Or that the compiler can chose to ignore recompiling if I explicitly invoke it? That is terrible if true, because then the build is not reliable at all.

[ggleyzer]

Why don't you try to add a space to Component.java and run build --info. You will see everything I'm talking about

[cpurdy]

An alternative way to look at it is that the build should always specify the --rebuild flag, since it has decided by that point that a build is required. Let's not over-complicate this :)

[lagergren]

An alternative way to look at it is that the build should always specify the --rebuild flag, since it has decided by that point that a build is required. Let's not over-complicate this :)

Then you might as well turn off the build cache, because that is pretty much what the rebuild flag does? A compilation with the rebuild flag marks any task with it in its configuration as "not up to date". This means that it will be rerun, i.e. is equivalent to using the --rerun-tasks flags built into gradle. Furthermore, it touches all source code in the source set, which I think is overkill at the current design, but initially there was some trouble getting xcc to respect the rebuild flag, or it wasn't always sent, or its changing value wasn't correctly treated as an input to the compile task, or something. But even if we removed that, we would get behaviour that would be equivalent to always rerunning all build tasks and never caching anything about them.

[lagergren]

Why don't you try to add a space to Component.java and run build --info. You will see everything I'm talking about

I do see that, and that is "good", in the way it means that at least the build is sound but takes time. Given that behaviour, the build should always "work", but we may need to spend time getting a more fine grained depenency graph or better incremental builds to get that to go faster. However - now I am confused and fail to remember the cases we discussed where you were happy that your existing use case from the build system, where XTC modules weren't unnecessarily rebuilt came from.

I am wondering if the ways the plugin invokes the launchers is a problem here, for example if I pass a directory of modules instead of individual module files on the module path, does it somehow make different decisions of what is stale, or something like that? Or if a source set is changed - the compile task for that source set WILL be invoked once again, but can the XTC compiler decide that it doesn't need to do work here in certain circumstances?

[lagergren]

Actually the problem is that javatools invalidates fine and is indeed a correct input property to xtc compile

However, if javatools change Gradle does indeed invoke the compiler, and it chooses to do nothing in many cases when the source tree hansn't changed. Thus, even though it's a coarse dependency to Java tools as discussed, the build is actually sound. The same problems would have happened from the command line, as the breakages we see here come from things reacting to the change in Java tools but skipping some of the recompilations in xtc. Mostly you can get away with that, but in other cases you need to force rebuild or delete the xtc modules or something. It has been established this is on the xtc level now, and explains why rerun task as gradle flag isn't a cure all. The task is rerun, it just doesn't do anything.

We should probably make this state opt in for Gene, and isolate the rest of the team from weird build system issues, that when briefly reported could be anywhere and are a huge time sink to look for, especially if it's not a gradle issue.

Gene - does it reduce to the halting problem or could you actually write logic that understands that a single module with a changed Javatools needs to be rebuilt or cause trouble later?

Gradle has a mechanism for this kind of dependency, and you can check which was invalidated btw, but it's overkill to let the compile task call compile with rebuild if the Java tools have been replaced even though technically possible (https://docs.gradle.org/current/userguide/custom_tasks.html). See incremental tasks

I'm going to enable rebuilding for Java tools changes by default and provide you with a simple opt out, like an environment variable you can keep set in shell or ide.

I have great hope that remaining build system bug reports are actually broken dependencies that can be fixed in the build now that this dark horse is identified.

[ggleyzer]

I'm absolutely fine with this approach

[cpurdy]

Then you might as well turn off the build cache, because that is pretty much what the rebuild flag does?

Sorry, let me be clear: When you invoke java to run the Ecstasy compiler (or invoke xcc or whatever), you should always invoke it with --rebuild because you have already figured out that the build is necessary, and there is no reason for the Ecstasy compiler to (i) check the timestamps again or (ii) over-ride your decision.

The problem, right now, is that you invoke the Ecstasy compiler, and then the Ecstasy compiler (built for end users, not for Gene fiddling around) looks at the source files and says "hey, it's all up to date, I'm not going to recompile". In other words, xcc doesn't ever think about the compiler itself changing, because (in the normal world) that never happens. But for me and Gene (etc.) it happens 73 times a day. Or maybe 973 times.

So unless the XDK_SUPPRESS_REBUILD (just making the name up here) env var is set, gradle will always emit the --rebuild when invoking the Ecstasy compiler. Does that make sense?

[ggleyzer]

Sounds good

[cpurdy]

It will have the added benefit of slightly speeding up the build :)

[lagergren]

Then you might as well turn off the build cache, because that is pretty much what the rebuild flag does?

Sorry, let me be clear: When you invoke java to run the Ecstasy compiler (or invoke xcc or whatever), you should always invoke it with --rebuild because you have already figured out that the build is necessary, and there is no reason for the Ecstasy compiler to (i) check the timestamps again or (ii) over-ride your decision.

The problem, right now, is that you invoke the Ecstasy compiler, and then the Ecstasy compiler (built for end users, not for Gene fiddling around) looks at the source files and says "hey, it's all up to date, I'm not going to recompile". In other words, xcc doesn't ever think about the compiler itself changing, because (in the normal world) that never happens. But for me and Gene (etc.) it happens 73 times a day. Or maybe 973 times.

So unless the XDK_SUPPRESS_REBUILD (just making the name up here) env var is set, gradle will always emit the --rebuild when invoking the Ecstasy compiler. Does that make sense?

Wait. So I have trouble with "classic" vs "xtc" behaviour again, I think. This doesn't fit into the "classic" use case for "rebuild", much like "gradle clean" doesn't fit into the classic use case for "make clean".

If I have a java or c++ source tree with a huge amount of files, and touch one of them, doing a "rebuild" would, at least to my intuition, indiscriminately recompile all c++ source code, even parts of the tree that do not need to rebuild, and where the compiler knows that they are older than their .o files. A "classic" rebuild is equivalent to running a touch operation on all source code files in a project. Are you saying that xtc --rebuild does not do that, but still tries to make decisions of which part of this huge tree needs to be recompiled?

You are actually saying the opposite - "using --rebuild will slightly speed up the build". Could you explain why that would be the case?

I should also disable my "extra caution" pass where I do touch all the source code in the tree if the rebuild flag is set, because that likely just causes a lot of extra rebuild time if rebuild doesn't work like that.

I would love to have some documentation of how xcc and xec resolve their workloads e.g. exactly which files need to be recompiled, and why xec gets its in its head to start compiling source code, while I expect it to just take my module path as a read only truth and assume that those are all the modules that exist in its closed world, and so on.