Gradle build

[Arcnor]

Is this considered/going to be considered?

I think it's the fastest way of getting LWJGL3 to be used by other projects without the current LWJGL2 hassle of generating POMs separated from the main build. It can be used to compile without ant at all, or it can be mixed.

I started adding support and got the kotlin plugin to compile all the sources, but got confused about the Generator.kt

If you're not planning on adding this, please close as wont fix so I don't spend any more time on it :D

[Spasi]

Yes, building with gradle would be interesting. Let me know if you still have trouble with the Generator. If you got it to compile, then it's a simple Java application, see the "generate" Ant target.

[Arcnor]

Ok, I'll continue then :).

Yeah, I know it's just a program, but it was strange that the Kotlin code was compiled using a Java tool. I need to look a bit more into it.

Also, do you mind if I move the code around? For ease of integration with gradle, it will be nice to have a module per "unit", i.e. "core" for the core library, "templates" for the templates library, and "natives" for the native library (or maybe "natives-linux", "natives-macosx" and "natives-windows", I need to check what needs to be generated). That means the code for each module will be inside /src/main/[java|kotlin]

[Spasi]

I understand the src/main/java thing is the Maven convention (over configuration), but it's not really compatible with how this project is set up. Especially, if you consider the generated code that's written in a different folder hierarchy (and I'd like to keep it this way).

I believe you can break the convention and configure everything manually in Gradle.

[Arcnor]

Yeah, I can break the convention, although some things like the Kotlin plugin might not like it. Also, it will be way more verbose.

I'll try to see what I can do.

Also, if you change where the generated code goes, IDEA will pick that up automatically and will detect the code as generated. I like that, but this it's your project ;)

[Spasi]

Thanks, I didn't know that feature. I didn't have to move the generated code though, just marked the root as generated. Noticed that IDEA disables inspections for the generated classes, do you know if it affects anything else?

[Arcnor]

You can either mark it as generated all the time, or put it on target/generated. Then IDEA will mark it automatically.

It doesn't affect much, although if you try to modify a generated class, IDEA will warn you.

[Arcnor]

Ok, an update on this.

I've managed to compile everything, although hardcoding some paths for now to make it simpler. Also, I managed to do it moving the code around like I said before.

I'm now trying to convert my code to use your non standard paths. However, it seems that the Kotlin plugin chokes on that, it compiles the Kotlin code fine, except for Types.kt (that depends on Java code, PointBuffer, that's going to be compiled at the same time). When I move everything into src/main/java and src/main/kotlin (it compiles Kotlin and Java code separatedly) everything works again.

I'm going to continue trying to make it work, but if I can't, will you accept that module (src/templates) being converted to some more standard? I'll also ask on the Kotlin forums in case I'm doing something wrong.

[Spasi]

The PointerBuffer dependency is resolved with classpath += "src/templates" in config/Templates.ktm. You should make sure that kotlinc is configured with Templates.ktm, because in the future it will be used for filtering the supported bindings. You'll be able to do custom builds, for example without OpenCL or with only the core OpenGL functionality, filter out old/obsolete extensions, etc.

[Arcnor]

Ok, I'll try to use the Templates.ktm file, although so far no luck with that one. Also, what that script is doing can be done with Gradle as well, and using Gradle flavors it will be even more elegant. But I'll try to make it work.

[big-guy]

hey @Arcnor, have you done anything with this yet that's available publically?

[virtuoushub]

Hi @Arcnor , like @big-guy, I am wondering if you have any code for the Gradle build I can take a look at.

[Arcnor]

Hi,

Sorry, but I don't have anything on the intertubes yet. I got it to compile, more or less, but never got it to use that Templates.ktm file (asked in their forums about the .ktm file, but got no response other than "use the mechanism native to Gradle: https://devnet.jetbrains.com/message/5527371)

I really want this to get with some project of mine (or even LibGDX) sooner or later, so if you can help, that'd be great :)

I'll try uploading what I had as soon as I get the time.

[dhild]

@Arcnor I think I figured out the kotlin configuration; I ended up just adding 'config' as a source directory for the templates compile. https://github.com/dhild/lwjgl3/tree/gradle_project if anyone is interested. Currently the java & kotlin files compile (haven't verified that they're functionally identical), and I've started looking at using gradle's native compilation mechanisms. Still a work in progress, but might give @big-guy or @virtuoushub a reference.

[dhild]

I've spent a little more time figuring out the native compilation, and got it working on windows. Using entirely the gradle build for compilation, I was able to get the WGLDemo and UniformArrayDemo classes to run. Still remaining:

• Setting up the Linux build (I'll probably do this later)
• Setting up the Mac build (I can write the script, but I don't have a Mac to test with)
• Consolidating the existing config (there's a lot of "if windows ..." statements, this could be handled better)
• Extending with a few more tasks, etc. to fully replicate the ant scripts:
  • Javadoc needs some tweaking
  • Release task has not been started
  • demo task (or tasks) could be created
  • I'm still figuring out exactly how the generated & wiki pieces fit into the build / workflow

I'll keep updating the fork I made as I progress.

[octylFractal]

@dhild I've got a Mac.

[dhild]

@kenzierocks Cool - would you want to tackle adding it into what I have, or would you like me to get it started first?

[octylFractal]

I think I only have time to test right now, so you should probably get started.

[Arcnor]

@dhild if @kenzierocks doesn't have time to test it, I can also do it after you've submitted what you need to submit.

Awesome that you got this to work. I stopped at the native libraries compilation for lack of time.

[dhild]

@kenzierocks @Arcnor Sounds great. I've been doing a lot of gradle configuration at work lately so it flows pretty quickly for me. From the gradle docs, not too sure how the Mac config even works; it looks like it uses the Clang configurations, so it could be pretty easy after getting the linux build down.

[dhild]

Okay - haven't tried it out on my linux environment yet, but I have an initial configuration for mac & linux uploaded. There was one hiccup in the mac setup, some sort of config relating to the sysroot setting and another related to the version in use. I left those out for the moment, so you might have to manually set them up. I'm betting that gradle does some of that configuration for us, but I'd have to try it out to find out....

Look for the TODO statement in the build.gradle file to see what I'm talking about.

[Arcnor]

I did ./gradlew build and that didn't go well:

/tmp/lwjgl3@gradle_project✗✗✗✗✗✗% ./gradlew build
:compileTemplatesKotlin
:compileTemplatesJava
warning: [options] bootstrap class path not set in conjunction with -source 1.6
1 warning
:processTemplatesResources UP-TO-DATE
:templatesClasses
:compileKotlin
No Kotlin files found, skipping Kotlin compiler task
:generate
Exception in thread "main" java.lang.IllegalArgumentException: Invalid generation target path: /private/tmp/lwjgl3/generated
    at org.lwjgl.generator.GeneratorPackage$Generator$4582b1a4$main$validateDirectory$1.invoke(Generator.kt:37)
    at org.lwjgl.generator.GeneratorPackage$Generator$4582b1a4$main$validateDirectory$1.invoke(Generator.kt)
    at org.lwjgl.generator.GeneratorPackage$Generator$4582b1a4.main(Generator.kt:41)
    at org.lwjgl.generator.GeneratorPackage.main(Unknown Source)
:generate FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':generate'.
> Process 'command '/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/bin/java'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 36.119 secs

[octylFractal]

Your latest commit (878664b3e08d701ca63f946421a2f8210378afad) should use it.options, not options.

Never mind, I don't know what's failing it.

[Arcnor]

@kenzierocks you can add comments directly to commit lines if you like :smile:

[octylFractal]

I do know, it's just quicker for me to use what I've already got.

[dhild]

@kenzierocks Whoops. I've always just used the implicit reference to the delegate object though, mostly because I think it reads easier.

@Arcnor Okay, I'm guessing that the generate failed because that directory didn't exist. I added a quick fix to create the directories first. I'm a little nervous about that because of the symlinks in the original script (don't yet know about those), but it might fix it.

[Arcnor]

@dhild ok, that got me further. Now:

 warnings
:processResources UP-TO-DATE
:classes
:jar
:compileX64DebugLwjglSharedLibraryLwjglC
In file included from /private/tmp/lwjgl3/src/native/system/common_tools.c:5:
In file included from /private/tmp/lwjgl3/src/native/system/common_tools.h:26:
/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/jre/../include/jni.h:45:10: fatal error: 'jni_md.h' file not found
#include "jni_md.h"
         ^
1 error generated.
:compileX64DebugLwjglSharedLibraryLwjglC FAILED

jni_md.h is on /Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/include/darwin/jni_md.h, not sure if you can automate that

[Arcnor]

Ok, commented too quickly. I edited build.gradle and added the include for /darwin, but now:

:compileX64DebugLwjglSharedLibraryLwjglC
/private/tmp/lwjgl3/src/native/system/org_lwjgl_system_MemoryUtil.c:24:19: fatal error: use of undeclared identifier 'intptr_t'
        return (jobject)(intptr_t)globalRef;
                         ^
1 error generated.
:compileX64DebugLwjglSharedLibraryLwjglC FAILED

[dhild]

@Arcnor That's weird. stdint.h should have intptr_t defined (or at least include its definition). Not sure where to go with that one.

https://github.com/LWJGL/lwjgl3/blob/master/src/native/system/macosx/MacOSXConfig.h#L7

[Arcnor]

If I add #include <stdint.h> at the top of the file, I get past that error, so it seems stdint.h is not being included...

[Arcnor]

Ok, found the problem. Your defines inside build.gradle should not include the initial D

[dhild]

@Arcnor Gotcha - copy/paste error, I think

[Arcnor]

Next error:

:compileX64DebugLwjglSharedLibraryLwjglC
In file included from /private/tmp/lwjgl3/generated/native/system/linux/org_lwjgl_system_linux_DynamicLinkLoader.c:7:
/private/tmp/lwjgl3/src/native/system/linux/LinuxLWJGL.h:8:11: fatal error: 'X11/X.h' file not found
        #include <X11/X.h>
                 ^
1 error generated.
:compileX64DebugLwjglSharedLibraryLwjglC FAILED

Why is this trying to compile Linux is unknown to me :smile:

[dhild]

Try commenting out https://github.com/dhild/lwjgl3/blob/gradle_project/build.gradle#L278-L279 and see if that fixes it. Might need some more powerful checks there?

Doing "gradlew components" should show the source directories it's picking up.

[Arcnor]

It did. Final one:

:compileX64DebugLwjglSharedLibraryLwjglC
/private/tmp/lwjgl3/generated/native/system/macosx/org_lwjgl_system_macosx_CGL.c:106:15: fatal error: 'CGLCopyContext' is deprecated: first deprecated in OS X 10.8 [-Wdeprecated-declarations]
        return (jint)CGLCopyContext(src, dst, mask);
                     ^
/System/Library/Frameworks/OpenGL.framework/Headers/OpenGL.h:51:17: note: 'CGLCopyContext' has been explicitly marked deprecated here
extern CGLError CGLCopyContext(CGLContextObj src, CGLContextObj dst, GLbitfield mask) OPENGL_DEPRECATED(10_0, 10_8);
                ^
1 error generated.
:compileX64DebugLwjglSharedLibraryLwjglC FAILED

I added Wno-deprecated and it worked for now, and then when linking:

:linkX64DebugLwjglSharedLibrary
clang: error: unknown argument: '-framework AppKit'
clang: error: unknown argument: '-framework CoreVideo'
clang: error: unknown argument: '-framework IOKit'
clang: error: unknown argument: '-framework OpenGL'
:linkX64DebugLwjglSharedLibrary FAILED

[octylFractal]

Yes, Macs are Darwin->BSD->FAMILY_UNIX so that check is probably true

[dhild]

@Arcnor As far as the frameworks go, that's totally out of my league in Mac-know-how.

[dhild]

@kenzierocks Good old ant.taskdefs.condition.Os is letting me down!

[Arcnor]

@dhild problem was in the passing of the arguments. It should be "-framework", "AppKit" and not "-framework AppKit" because that will pass the name of the argument as the whole thing :smile:

[Arcnor]

Next error:

:updateMacosxX64 UP-TO-DATE
:linkX64DebugLwjglSharedLibrary
:x64DebugLwjglSharedLibrary
:compileX64DebugLwjglStaticLibraryLwjglC
:createX64DebugLwjglStaticLibrary
:x64DebugLwjglStaticLibrary
:compileX64ReleaseLwjglSharedLibraryLwjglC
:linkX64ReleaseLwjglSharedLibrary
:x64ReleaseLwjglSharedLibrary
:compileX64ReleaseLwjglStaticLibraryLwjglC
:createX64ReleaseLwjglStaticLibrary
:x64ReleaseLwjglStaticLibrary
:compileX86DebugLwjglSharedLibraryLwjglC
:linkX86DebugLwjglSharedLibrary

FAILURE: Build failed with an exception.

* What went wrong:
Static library file /private/tmp/lwjgl3/build/libs/macosx/x86/libglfw3.a does not exist for prebuilt library 'glfw3'.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

[dhild]

@Arcnor Added the warning, fixed the framework statements. I also pushed up a change in the OS detection, I think that's why it's trying to do the x86 compile. The original build script only has x64 for Mac, so I copied that. It's probably picking that up because of the linux configurations....

[Arcnor]

Yeah, I removed the UNIX stuff before continuing because as Kenzie said, OSX is UNIX. I thought you added that on purpose :).

@Arcnor https://github.com/Arcnor Added the warning, fixed the framework statements. I also pushed up a change in the OS detection, I think that's why it's trying to do the x86 compile. The original build script only has x64 for Mac, so I copied that. It's probably picking that up because of the linux configurations....

— Reply to this email directly or view it on GitHub https://github.com/LWJGL/lwjgl3/issues/16#issuecomment-93213974.

[dhild]

Nope, totally not intentional. Should have thought about that one a bit more :)

[Arcnor]

Ok, after fixing & pushing a few things, now I got to tests failing! I'll try to find out what the problem is, but if anybody has some clue, please comment.

org.gradle.api.internal.tasks.testing.TestSuiteExecutionException: Could not complete execution for process 'Gradle Test Executor 2'.
    at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
    at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
    at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
    at com.sun.proxy.$Proxy2.stop(Unknown Source)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.stop(TestWorker.java:116)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
    at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360)
    at org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:64)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.testng.TestNGException: 
An error occurred while instantiating class org.lwjgl.opencl.CLTest: Failed to load the native library: lwjgl
    at org.testng.internal.ClassHelper.createInstance1(ClassHelper.java:398)
    at org.testng.internal.ClassHelper.createInstance(ClassHelper.java:299)
    at org.testng.internal.ClassImpl.getDefaultInstance(ClassImpl.java:115)
    at org.testng.internal.ClassImpl.getInstances(ClassImpl.java:200)
    at org.testng.internal.TestNGClassFinder.<init>(TestNGClassFinder.java:120)
    at org.testng.TestRunner.initMethods(TestRunner.java:409)
    at org.testng.TestRunner.init(TestRunner.java:235)
    at org.testng.TestRunner.init(TestRunner.java:205)
    at org.testng.TestRunner.<init>(TestRunner.java:153)
    at org.testng.SuiteRunner$DefaultTestRunnerFactory.newTestRunner(SuiteRunner.java:536)
    at org.testng.SuiteRunner.init(SuiteRunner.java:159)
    at org.testng.SuiteRunner.<init>(SuiteRunner.java:113)
    at org.testng.TestNG.createSuiteRunner(TestNG.java:1299)
    at org.testng.TestNG.createSuiteRunners(TestNG.java:1286)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
    at org.testng.TestNG.run(TestNG.java:1057)
    at org.gradle.api.internal.tasks.testing.testng.TestNGTestClassProcessor.runTests(TestNGTestClassProcessor.java:136)
    at org.gradle.api.internal.tasks.testing.testng.TestNGTestClassProcessor.stop(TestNGTestClassProcessor.java:80)
    at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:59)
    ... 21 more
Caused by: java.lang.UnsatisfiedLinkError: Failed to load the native library: lwjgl
    at org.lwjgl.LWJGLUtil.loadLibrarySystem(LWJGLUtil.java:339)
    at org.lwjgl.Sys$1.run(Sys.java:36)
    at java.security.AccessController.doPrivileged(Native Method)
    at org.lwjgl.Sys.<clinit>(Sys.java:33)
    at org.lwjgl.LWJGLUtil.initialize(LWJGLUtil.java:309)
    at org.lwjgl.system.MemoryUtil.<clinit>(MemoryUtil.java:36)
    at org.lwjgl.Pointer.<clinit>(Pointer.java:22)
    at org.lwjgl.PointerBuffer.<init>(PointerBuffer.java:24)
    at org.lwjgl.PointerBuffer.allocateDirect(PointerBuffer.java:281)
    at org.lwjgl.BufferUtils.createPointerBuffer(BufferUtils.java:190)
    at org.lwjgl.system.libffi.Closure.<clinit>(Closure.java:45)
    at org.lwjgl.opencl.CLTest.<clinit>(CLTest.java:47)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at org.testng.internal.ObjectFactoryImpl.newInstance(ObjectFactoryImpl.java:29)
    at org.testng.internal.ClassHelper.createInstance1(ClassHelper.java:387)
    ... 39 more

[Arcnor]

Ok, fixed that one but now I'm getting:

6:49:37.581 [ERROR] [system.err] 2015-04-15 16:49:37.581 java[5884:18423] +[NSUndoManager(NSInternal) _endTopLevelGroupings] is only safe to invoke on the main thread.
1
...

...which clearly indicates that the XstartOnFirstThread flag is not being correctly passed. I think that flag is getting to the wrong process, but not sure how to fix that one.

[dhild]

I looked around to try to find the correct test configuration, but didn't have much luck.

In other news: we might not even need to have the JVM argument to get the Mac display working. I'm totally unsure about how much work that would be though, but here' my thought process:

Looks like there's a check to ensure that certain code runs on the main thread: https://github.com/LWJGL/lwjgl3/blob/master/src/core/org/lwjgl/system/macosx/EventLoop.java#L18-L21

This method appears to be called only in one place; in the generated code: https://github.com/LWJGL/lwjgl3-generated/blob/master/java/org/lwjgl/glfw/GLFW.java#L427-L430

A little searching turned up http://stackoverflow.com/questions/12937762/invalid-thread-access-even-with-xstartonfirstthread-in-vm-args, which leads to https://bugs.eclipse.org/bugs/show_bug.cgi?id=303869 (see the very bottom for their solution). Looks like there's some apple-specific classes that can be used to run an arbitrary code block on the main thread, and that it's been used for some other projects successfully.

Not knowing about what exactly is being started & needs to run on the main thread, I'm a little unsure as to how this would integrate, or if there would be a significant refactor for it.

[dhild]

So I pulled down the latest set of changes from upstream, and saw that module support is being started. Awesome! It does make the gradle configuration a great deal more complex though.

I understand that this project has a custom layout, which has worked so far. That being said, I'm much more in favor of reorganizing the sources according to modules, which would be a good opportunity to move to a conventional layout, e.g. src/main/java, src/main/kotlin, and organizing each module as it's own sub-folder. Test sources specific to a given module could live alongside each module's main source, and a larger, integration test module would also make sense.

As @Arcnor said, moving to gradle would help make LWJGL3 easier to use for developers, both internal and external to the project. As far as I can tell, I've been able to duplicate the build functionality from the ant scripts. The only hiccup so far is the mac issue where the tests are not honoring the -XstartOnFirstThread flag, which is a separate (but relevant) issue.

In conclusion, the gradle configuration will end up being more complicated than it is worth, unless a source reorganization is acceptable. I could knock this out pretty quick, and I think I'll try just to see what it looks like in the end. Any input from @Spasi would be appreciated here.

[Spasi]

I've been following this issue and porting the Ant scripts to Gradle is still something I'd like to do. I've been struggling with Ant, especially with non-trivial stuff like the modular build. The scripts have become hard to understand and maintain, we need something better.

With that said, time for LWJGL development has been limited lately and my main focus is getting the 3.0a released. Once that's done, I'll have time to get comfortable with Gradle and start the porting process. The work you guys have done so far will obviously be a big help, but I don't have the Gradle experience atm to evaluate it (or help with issues).

[Spasi]

I will of course also consider changing the source layout, as required by Gradle.

[Arcnor]

@Spasi if you don't mind changing the layout, @dhild and me can take care of the whole building stuff & doing the porting if you want, even if you don't have the time to learn Gradle, and even for 3.0a.

[dhild]

@Spasi I completely understand limited time. Sounds like @Arcnor and I will keep working on the port in parallel for now. Thanks for the quick response!