Rubygem version of cake fails to add /lib jars to classpath

[sritchie]

I've been running into an issue lately where certain other libraries that need to search my classpath can't find any of my main dependencies... pallet, for example, could only find services that I'd added as dev dependencies. Tonight, digging around the ruby with dsantiago, I figured out why!

on line 686 of bin/cake, lib/dev/* makes it in, but it looks like lib/* didn't.

I think a simple change of 686 to

[bakepath, cakepath, "src", "src/clj", "lib/*", "lib/dev/*", "#{$home}/.cake/lib/dev/*"],

will fix this right up.

Thanks so much for cake! Hope this helps.

Best, Sam

[ninjudd]

Did you try wrapping the task that needs the dependencies in (bake ...)? See src/cake/tasks/test.clj for an example. Remember, cake has a separate classloader from you project, and you have to use bake to switch to the project classloader in tasks.

[davidsantiago]

This isn't something you can wrap in bake, it's definitely a problem with the classpath, and I don't think sritchie's suggested fix fixes it either. Put this project.clj in a dir and run cake deps:

(defproject the-lame "0.0.1-SNAPSHOT"
  :dependencies [[org.clojure/clojure "1.2.0"]
                 [org.clojure/clojure-contrib "1.2.0"]
                 [org.cloudhoist/pallet "0.4.10"]]
  :dev-dependencies [[swank-clojure/swank-clojure "1.2.1"]]
  :repositories {"sonatype"
                 "https://oss.sonatype.org/content/repositories/releases/"})

After cake deps, run cake repl, use 'pallet.compute, and run (pallet.compute/supported-providers). You'll get an empty list (even on git checkout version). In reality, this should be a list like ("cloudservers" "ec2" "bluelock-vcdirector" "nova" "trmk-vcloudexpress" "eucalyptus" "trmk-ecloud" "vcloudexpress" "slicehost" "gogrid" "vcloud" "stub" "rimuhosting" "node-list").

Now move [org.cloudhoist/pallet "0.4.10"] out of :dependencies and into :dev-dependencies, clean lib/ get deps and try again. Now supported-providers will pull up the right list above. Why should this work as a dev-dependency but not a regular dependency?

supported-providers is a function in pallet that uses some regexes on the classpath to find namespaces that start with pallet.compute, and it should be finding quite a few, but it's finding none. Why is this happening? I thought it was because of the ruby code that sritchie found only adding lib/* to the classpath on git checkout, but I'm trying it and the problem still happens on git checkout.

Now, we could definitely ding hugod for how pallet is finding its providers, but I don't think his code should have to be aware of Cake's classloader/etc weirdness. It works right in Leiningen.

• David

[ninjudd]

It must be related to how he's getting the current classloader. it will require a fix in pallet

[davidsantiago]

Why should pallet have to know about whether it's running in cake or not? If it works by itself, and it works in leiningen, then he's in some sense doing something correct.

• David

[ninjudd]

there is no correct way to fix this problem by modifying cake, so if you want it fixed, you will have to patch pallet. I'm happy to help with it on Monday.

[davidsantiago]

OK, we'll talk on monday. Hugo pointed out that the problem is how to get all of the compute providers available to Pallet without hardcoding a list ahead of time. Some of the providers are from external jars. I've only recently and reluctantly been dragged onto the JVM by the joy of Clojure, so I'm not an expert in classpathology. Hopefully there's still a way to do that?

• David

[sritchie]

Hey all, just wanted to note that my suggested fix did actually work for me, on the guy checkout version. Run cake kill -9, then cake swank (or repl) in that project. Then, try (System/getProperty "java.class.path"). before the fix, none of the lib jars show up... After, they all do, and the supported-providers command works for me. success! I'm typing this on a phone on the side of the road, so excuse the formatting.

As dsantiago noted, this problem won't show up for the Cake project, since cakedir/lib/* is explicitly added. I'll add more here later, but it really does look like an addition of "lib/" should have been on 686 all along. If not, what was the reason for explicitly including "lib/dev/"?

[sritchie]

(David, can you check the fix after running "cake kill -9"?)

[ninjudd]

The fix works, but it is incorrect because it adds the project dependencies to cake's classpath. The whole point of classlojure and bake is that cake and your project have separate classloaders and classpaths. So putting lib/* on cake's classpath is not the right fix.

[ninjudd]

The bug is in clojure.contrib.classpath.

https://github.com/richhickey/clojure-contrib/blob/2ede388a9267d175bfaa7781ee9d57532eb4f20f/src/main/clojure/clojure/contrib/classpath.clj#L23

[sritchie]

Got it. But why does lib/dev/* make it in?

[ninjudd]

It should be calling (.getClassLoader clojure.lang.RT) and loading the classloader from there, instead of reading (System/getProperty "java.class.path")

[ninjudd]

Because development dependencies can contain cake tasks, so by definition they have to be in cake's classpath.

I don't have time to talk about this any more until tomorrow, but you will have better luck trying to get Stuart Sierra to fix clojure.contrib.classpath or Hugo to fix pallet than getting me to change cake.

[sritchie]

About that, pallet gets its classpath stuff from clojure.contrib.classpath, or something like that... I'll post the code when I'm back at my machine, but it's that code that uses System/getProperty. More in a bit.

[sritchie]

Great, thanks for taking a look!

[davidsantiago]

You're absolutely right, Justin. Didn't realize that was for the cake VM. Hugo has already put in a fix in pallet, should be releasing 0.4.11 very soon. Thanks a lot for your help.

[ninjudd]

Here is the relevant fix in pallet: https://github.com/pallet/pallet/commit/a2aaf17b695c86c5b49df6fe7e07dc6e79566dbe

[sritchie]

Justin, thanks a lot for your help on this!