Slow build-steps: ~2-5s per module on cache hit

[bkase]

Thanks again, @dreiss for helping with my last issue.

The build is fast now, but there seems to a constant ~18sec overhead whenever buck build app is executed.

I attached traces, but it seems like there isn't much information on cache hit. Let me know if there is some way to produce more verbose profiling information.

clean build trace no-op trace

The gen-rule to hack in kotlin takes almost no time, but the cache hit for the module seems way too long.

I don't quite understand how the [cache] field in .buckconfig is supposed to affect performance, but changing mode to dir under [cache] doesn't seem to noticeably affect build times.

Our build time has already halved, but it would be really nice to shave off another 20s.

Thanks!

[bkase]

EDIT: The cache-hit-check-time is back to ~2-5s per module. Before I built with gradle, it was 100-200ms. Post gradle build, I can't seem to get that to com back.

I'll keep my old comment here, but I assume it is unrelated to the speedup I briefly saw.

I don't know why, but my cache-hit-check-time went down to ~100-200ms!

The only thing I changed was this:

(in a file that declares a prebuilt jar for the kotlin runtime)

 genrule(
        name = 'kotlin-runtime-jar',
        out = 'kotlin-runtime.jar',
-       cmd = 'ln -s ~/kotlin2/dist/kotlinc/lib/kotlin-runtime.jar $OUT',
+       cmd = 'ln -s ' + os.environ["KOTLINPATH"] + '/dist/kotlinc/lib/kotlin-runtime.jar $OUT',
 )

and

(in a script that pretends to be javac)

-    exit(call(["/Users/highlight/kotlin2/dist/kotlinc/bin/kotlinc"] + params))
+    exit(call([os.environ["KOTLINPATH"] + "/dist/kotlinc/bin/kotlinc"] + params))

It's slow immediately before this commit, fast immediately after.

I'll leave this issue open (I'm happy to tweak my build settings to help you guys figure out why this was happening), but feel free to close it. Builds are fast.

[k21]

From the traces it looks like almost all of the time in the no-op build is spent computing the rule keys, which usually means hashing the source files. If you are using buckd, then hashes of files in the repo should be cached, so the reason of the slow build is either that you are not using buckd or that the rules refer to files outside of the repository.

[bkase]

I am using buckd (at least "Using buckd." is printed every time buck runs).

I have a genrule that copies all *.kt to *.kt.java to spoof the android_library rule. Could this be causing the hashes to be recomputed? If this is the cause, is there an easy way to make the android_library rule send my "javac" *.kt files as well as *.java? I don't mind forking Buck if necessary.

[sdwilsh]

What does your genrule look like? It's possible it's doing something to cause this worst-case scenario.

With that said, I don't see anywhere that we actually filter out anything but *.java files. Do you have a test case that demonstrates that?

[bkase]

I want to build *.kt files but utilize all the machinery that buck provides via android_library (like build R.java properly first etc.). The android_library rule seems to only passes the java files to the compiler for the build steps that require compiling code (which makes sense if all of your code is in java). Running the rule without changing the filenames results in an empty jar.

It would be nice if I could get the android_library rule to give kotlin files to the compiler.

The genrule is a hack modified from [https://github.com/kargs/buckexp]()

android_library(
    name = name,
    visibility = visibility,
    srcs = [':kotlin-src-' + name],
    compiler = 'kotlincw',
    deps = kwargs.get('deps', []),
    exported_deps = kwargs.get('exported_deps', []),
  )

  genrule(
    name = 'kotlin-src-' + name,
    srcs = srcs,
    out = 'kotlin-' + name + '.src.zip',
    cmd = 'echo "$SRCS" | awk -F" " \'{ for(i = 0; ++i <= NF;) print $i }\' | xargs -I{} cp {} {}.java && echo "$SRCS" | awk -F" " \'{ for(i = 0; ++i <= NF;) print $i }\' | xargs -I{} echo {}.java | zip -@ "$OUT"',
  )

Essentially, we copy *.kt files to *.kt.java and stick them in kotlin-srcs-<name>.zip then rewrite them back *.kt in the compiler script.

[sdwilsh]

I think at this point, you want to implement Kotlin in Buck. The good news is that @mread and @grumpyjames have been doing a bunch of work to support other JVM languages, so this should be a lot easier to do now.

[Coneko]

That genrule shouldn't be a problem: if the sources don't change it should not be executed.

The fact it outputs a zip with timestamps in it might be annoying for some flows, but it shouldn't be a big problem either.