search

ccw-ide / ccw

Counterclockwise is an Eclipse plugin helping developers write Clojure code
https://github.com/laurentpetit/ccw/wiki/GoogleCodeHome
Eclipse Public License 1.0
220 stars 50 forks source link

Problems linking against native libraries #836

Open noncom opened 9 years ago

noncom commented 9 years ago

With my current project I've stepped into a problem with native libraries. The issue is that lwjgl cannot find its natives although the jar with the natives is properly linked to the project and appears in Leiningen Dependencies.

I have made a little research and found out that this problem is known for lwjgl:

1) http://wiki.lwjgl.org/wiki/Downloading_and_Setting_Up_LWJGL look at the "I keep getting an java.lang.UnsatisfiedLinkError: no lwjgl in java.library.path" paragraph

2) this is how it is solved normally in eclipse: http://stackoverflow.com/questions/19344914/getting-java-lang-unsatisfiedlinkerror-no-lwjgl-in-java-library-path

3) and the official guide: http://wiki.lwjgl.org/index.php?title=Setting_Up_LWJGL_with_Eclipse (mostly same as point 2), but with neat pictures ).

So, in my project there is the lwjgl natives jar, however, the java lwjgl jar native library path points to "project/target/native/windows/x86_64" and in the project build path setup it says "not modifiable". The interesting part is that the natives are actually do get extracted and are there ready to be picked up, but they are located at "project/target/native/windows".

Resume: if I manually copy the dlls from "project/target/native/windows" to "project/target/native/windows/x86_64", all starts working okay. I know leiningen and java in general have their quirks in working with native libraries, but what is different in this case, is that the path "project/target/native/windows/x86_64" is incorrect and cannot be changed. Is there something in CCW that is getting in touch with this?

A small GitHub project illustrating the issue: https://github.com/noncom/test-natives Upon loading the core ns into the repl, you'll get:

;; Clojure 1.6.0
<Namespace test-natives.core>
CompilerException java.lang.UnsatisfiedLinkError: no lwjgl in java.library.path, compiling:(test_natives\core.clj:4:3)

and if you move the natives as I've described above, it will compile fine. even calling (go) will work ok and return nil. It does not matter that you have no actual opengl context active

laurentpetit commented 9 years ago

thanks, I will take a look at it

laurentpetit commented 9 years ago

I have the same problem with leiningen on the command line, no need to reach for ccw to get the UnstatisfiedLinkError exception.

laurentpetit commented 9 years ago

And as you suggested, moving my target/natives/macosx/ files to target/natives/macosx/x86_84 files did the trick and now leiningen on the command line works again

laurentpetit commented 9 years ago

There's a word about lwjgl in the leiningen samples project.clj, by the way:

https://github.com/technomancy/leiningen/blob/master/sample.project.clj#L55

laurentpetit commented 9 years ago

Please, talk about this issue with the Leiningen folks first. There's a lot on my plate these days, it would be good if you could confirm that you also get the same problem with leiningen, and then maybe find a solution, or some workaround. Then we'll be able to see if this work-around can or cannot be embedded into CCW. Deal?

noncom commented 9 years ago

Alright! I will do experiments with Leiningen first and see if it is possible to get a fix from some manipulations with it. Eh, I am sorry that I have loaded you with this one, since it apparently is a core Leiningen issue... on the other hand, I think a user expects his IDE to handle such things automatically... the case seems vague, I'll explore more and report when I have some results on this.

laurentpetit commented 9 years ago

You did the right thing, don't worry. Thanks for exploring the case.

2015-08-17 12:37 GMT+02:00 noncom notifications@github.com:

Alright! I will do experiments with Leiningen first and see if it is possible to get a fix from some manipulations with it. Eh, I am sorry that I have loaded you with this one, since it apparently is a core Leiningen issue... on the other hand, I think a user expects his IDE to handle such things automatically... the case seems vague, I'll explore more and report when I have some results on this.

— Reply to this email directly or view it on GitHub https://github.com/laurentpetit/ccw/issues/836#issuecomment-131773086.

Laurent Petit

noncom commented 9 years ago

Small update: specifying :native-path "target/native/native/windows" in project.clj makes it work. Not sure what that means though. With this parameter specified and even Leiningen project reset in CCW, the Eclipse dependencies window still shows project/target/native/windows/x86_64, non modifiable, but this time compilation works.

I am still contemplating on this...

laurentpetit commented 9 years ago

So to summarize :

2015-08-20 15:23 GMT+02:00 noncom notifications@github.com:

Small update: specifying :native-path "target/native/native/windows" in project.clj makes it work.Not sure what that means though. With this parameter specified and even Leiningen project reset in CCW, the Eclipse dependencies window still shows project/target/native/windows/x86_64, non modifiable, but this time compilation works.

I am still contemplating on this...

— Reply to this email directly or view it on GitHub https://github.com/laurentpetit/ccw/issues/836#issuecomment-133005533.

Laurent Petit

noncom commented 9 years ago

Yes, it works in Windows command line Leiningen, as well as from CCW if "launch projects with Leiningen" is enabled - same. However, if I uncheck "launch projects with Leiningen", it looks like that CCW goes with what Eclipse is saying in the deps and does not find the libs, failing as before.

Idk if there is even a solution for that. Probably this is a quirk of dealing with the lwjgl library in particular.. As hyPiRion said on #leiningen, not many people work with native deps and this area is highly underdocumented and it is unknown if some uniformity (i.e. a generalized solution) can be applied to the area at all. If it will become clear that it is impossible to formalize a universal approach for this situation, then maybe a plugin for CCW+LWJGL could be considered. Because the problem is there and other people can stumble upon it and there is not much info on this..

laurentpetit commented 9 years ago

@noncom which version of leiningen are you using on the command line?

laurentpetit commented 9 years ago

@noncom also, can you please update the github project test-natives with the fix you found?

laurentpetit commented 9 years ago

I still cannot make it work on the command line. Here's what I do:

Then I check the target directory, and there's nothing inside it.

So I may be missing the obvious, which is ... ?

laurentpetit commented 9 years ago

I will try to work this out, once I get the right configuration for making it work in OS X command line Leiningen :-)

2015-08-20 15:29 GMT+02:00 noncom notifications@github.com:

Yes, it works in Windows command line Leiningen, as well as from CCW if "launch projects with Leiningen" is enabled - same. However, if I uncheck "launch projects with Leiningen", it looks like that CCW goes with what Eclipse is saying in the deps and does not find the libs, failing as before.

— Reply to this email directly or view it on GitHub https://github.com/laurentpetit/ccw/issues/836#issuecomment-133007753.

Laurent Petit

noncom commented 9 years ago

Whoa... everything is more complicated than I originally thought. After I saw your last comments I went and tried to repeat the scenario again, doing this in the command line:

So I went to see what's up in the target dir and saw weird things. The dlls were lying in this actual location: target\native\native\windows\native\windows\.. oh my this looked very wrong... it didn't work however I tried to make it to. Then I did:

So, there is a mystery going on. Somehow specifying :natives-path before the first creation of target wreaks havoc on the directory structure.

Alright, so I decided to go check what's really going on! Here was my experiment:

Obviously this is a bug in Leiningen. It treats the :native-paths parameter differently in cases of the first and subsequent runs. Maybe I should first file an issue on Leiningen?

My current Leiningen version is 2.5.1... looking at https://github.com/technomancy/leiningen/blob/master/NEWS.md I can conclude that 2.5.2 makes no difference.

Also, I have updated the repository.

noncom commented 9 years ago

Ah, sorry, the fact that the solution did not work made me embarrassed and I went checking it out right away. Now I re-read what you're saying and I see you mention that the target directory is just empty in your case? How weird... I do not even know what to attribute this to.. surely it is because of Mac.. becaues the directory is not empty on Windows, Leiningen manages to unpack things there.. Mrhh, this is one more mystery to solve... but i am sure it is a Leiningens bug.

UPDATE: I tried what your described on a Mac on my job, the target directory was not empty (without even adding :natives-path to project.clj). It gets populated just like on Windows... hmmmm... are you sure it's empty on your machine, after you do lein repl?

laurentpetit commented 9 years ago

Yes, this seems like a bug in leiningen, and I'd be more than happy if you can manage the leiningen part of this. In the meantime, I think the workaround you found should be enough, right?

2015-08-21 17:08 GMT+02:00 noncom notifications@github.com:

Whoa... everything is more complicated than I originally thought. After I saw your last comments I went and tried to repeat the scenario again, doing this in the command line:

  • manually remove the target and bin directories
  • lein clean
  • add :native-path "target/native/native/windows" to the project.clj
  • lein repl
  • (require 'test-natives.core) and.. it failed.

So I went to see what's up in the target dir and saw weird things. The dlls were lying in this actual location: target\native\native\windows\native\windows.. oh my this looked very wrong... it didn't work however I tried to make it to. Then I did:

  • manually remove the target and bin directories
  • lein clean
  • lein repl
  • exit lein repl back to command line
  • add :native-path "target/native/native/windows" to the project.clj
  • lein repl
  • (require 'test-natives.core) and.. it worked.

So, there is a mystery going on. Somehow specifying :natives-path before the first creation of target wreaks havoc on the directory structure.

Alright, so I decided to go check what's really going on! Here was my experiment:

  • manually remove the target and bin directories
  • lein clean
  • add :native-path "target/native/a/b/c/d/native/windows" to the project.clj
  • lein repl
  • (require 'test-natives.core) fail!
  • exit lein repl back to command line
  • examine the directory structure and find dlls at target\native\a\b\c\d\native\windows\native\windows\ - the actual path differs from the one specified in project.clj by having an additional \native\windows\ at the end. alright.
  • change :native-path "target/native/a/b/c/d/native/windows" to :native-path "target/native/a/b/c/d/native/windows/native/windows" the project.clj
  • lein repl
  • (require 'test-natives.core) and.. it worked.

Obviously this is a bug in Leiningen. It treats the :native-paths parameter differently in cases of the first and subsequent runs. Maybe I should first file an issue on Leiningen?

My current Leiningen version is 2.5.1...

— Reply to this email directly or view it on GitHub https://github.com/laurentpetit/ccw/issues/836#issuecomment-133457004.

Laurent Petit

noncom commented 9 years ago

Yes, the available workaround allows me to continue the work, it is not a blocking problem. And also, it is pointless to do any adjustments to CCW until Leiningen is fixed. Maybe CCW will not even need any fixes in the end. Okay, then I will work with the Leiningen part of this issue and come back to this one here when we have some info or a fix from Leiningen!

noncom commented 8 years ago

Leiningen people still did not pick up the issue. Probably it is my bad and it was lousely formulated (although I don't think so, considering the amount of information that needs to be presented for complete description). I have reformulated it. I hope it will receive attention.

laurentpetit commented 8 years ago

Ok, thanks for the update. I also hope it will receive the attention it deserves.

2015-10-13 14:29 GMT+02:00 Alex Fowler notifications@github.com:

Leiningen people still did not pick up the issue. Probably it is my bad and it was lousely formulated (although I don't think so, considering the amount of information that needs to be presented for complete description). I have reformulated it. I hope it will receive attention.

— Reply to this email directly or view it on GitHub https://github.com/laurentpetit/ccw/issues/836#issuecomment-147701095.

Laurent Petit

noncom commented 8 years ago

Yeah, well, you know, some long time ago, even before having this exact issue, I've been talking to hyPiRion about Leiningen and native libraries. He said that as there are really not many people who work with this kind of programs at all, including themselves (the Leiningen team), nobody even know or check how it all works because they have no immediate need to do this while these cases really require some dedication.

noncom commented 8 years ago

Good news on this one! The issue has been resolved: https://github.com/technomancy/leiningen/commit/adb7b999fdc9050502032dcefd99c42c8d707ca7 I did not have a chance yet to test it myself though.. I'll get to it as soon as possible.

laurentpetit commented 8 years ago

Great ! If possible I'll include it in version 0.35.0 !