reducers not working from ccw repl ?

[GoogleCodeExporter]

I am experimenting with Clara rules, it fails to load.

After some digging I found that core.reducers is not loading:

=> (clojure-version)
"1.6.0"
=> (require '[clojure.core.reducers :as r])
CompilerException java.lang.Exception: namespace 'clojure.core.reducers' not 
found, compiling:(/tmp/form-init8985566267446161322.clj:1:2)

From lein repl at the top of the project it runs.

What am I missing ?

Eclipse Kepler Service Release 2
Ccw 0.26.0.STABLE001
java version "1.7.0_55"
OpenJDK Runtime Environment (IcedTea 2.4.7) (7u55-2.4.7-1ubuntu1)
OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)
Ubuntu 14.04

Thank you,
Luc

Original issue reported on code.google.com by lprefont...@softaddicts.ca on 4 Aug 2014 at 1:50

[GoogleCodeExporter]

Back from holidays here. I'll look into this ASAP.

Original comment by laurent....@gmail.com on 15 Aug 2014 at 10:29

• Changed state: Accepted
• Added labels: next

[GoogleCodeExporter]

I can see the name space if I browse the lein dependencies in the project 
explorer pane.

Here's a minimal project.clj that gives me this headache and the list of 
dependencies as seen by lein (lein deps :tree).

I can require the name space in another older project but I cannot find
why and what is different in this context vs versus the minimal project.

Initially this was seen in a bigger project and a created a minimal one
to isolate it.

Original comment by lprefont...@softaddicts.ca on 16 Aug 2014 at 11:56

Attachments:

• leindeps.lis
• project.clj

[GoogleCodeExporter]

Forgot to add that our private repo in the project.clj is available publicly.
It caches dependencies but all our projects are using this.

I commented it out of the project.clj file, got lein to recompute the 
dependencies
and still failing.

Original comment by lprefont...@softaddicts.ca on 16 Aug 2014 at 12:01

[GoogleCodeExporter]

Hi, 

I thought I would be able to reproduce the issue, and surprisingly, I couldn't.

I took project.clj above, which contains these dependencies:

[org.clojure/clojure "1.6.0"]
[org.clojure/tools.trace "0.7.8"]
[org.toomuchcode/clara-rules "0.6.0"]

I start a leiningen-driven repl, then:

;; Clojure 1.6.0
#<Namespace i649-reducers.core>
#'i649-reducers.core/foo
=> (require '[clojure.core.reducers :as r])
nil

My environment:
!SESSION 2014-08-08 20:21:50.458 -----------------------------------------------
Eclipse Luna ( eclipse.buildId=4.4.0.I20140606-1215 )
java.version=1.8.0
java.vendor=Oracle Corporation
OS X Mavericks

Original comment by laurent....@gmail.com on 16 Aug 2014 at 9:00

[GoogleCodeExporter]

Are you using CCW standalone or CCW software update site on an existing Kepler 
SR2 ?

Original comment by laurent....@gmail.com on 16 Aug 2014 at 9:01

[GoogleCodeExporter]

You can get the precise command line that is used by Counterclockwise: open the 
Debug perspective.

Depending on whether you start in Run or Debug mode, you will see one or two 
processes:
- there will at least be a process for leiningen JVM
- and you will see a remote process connected to the project JVM (in Debug mode)

Right clicking on the first child of each process and choosing Properties will 
show the environment variables and the command line.
Could you report what the values are for each when you have the problem?

Original comment by laurent....@gmail.com on 16 Aug 2014 at 9:11

[GoogleCodeExporter]

Also, you see the file reported in your stack trace: 
/tmp/form-init8985566267446161322.clj
(name may be different each time)

If it is still present (check before closing the jvms), can you also give its 
content?

Original comment by laurent....@gmail.com on 16 Aug 2014 at 9:12

[GoogleCodeExporter]

The java command line + env vars you'll get from the Properties should help 
reproduce the problem without Eclipse, from the command line.

Original comment by laurent....@gmail.com on 16 Aug 2014 at 9:12

[GoogleCodeExporter]

Original comment by laurent....@gmail.com on 16 Aug 2014 at 9:13

• Changed state: InfoNeeded

[GoogleCodeExporter]

CCW on an existing Kepler SR2. I am using several plugins.

Attached the Lein JVM process properties (ccw-lein-props)
None where found on the OpenJDK-Server process, the fields where empty on the 
property window.
Also the tmp file reported in the error message is attached (form-init...).

I made a shell script to start the lein repl as started by CCW (startlein.sh),
started it at the root of the test project, used grenchman to connect to it and 
was
able to require the reducer namespace successfully.

Note that I installed DCEVM tonight , just remove -XXaltjvm=dcevm from the shell
script if you do not want it enabled. It's the same behavior with or without it.

I would doubt that my own bash environment is problematic except if some 
exported
symbols conflict with some referred to by CCW.

Hope that this helps...

On a lighter note, Canada won over France at the Women World Rugby Cup and made 
it 
to the final match :)

Yeah !

Thank you

Original comment by lprefont...@softaddicts.ca on 16 Aug 2014 at 11:11

Attachments:

• form-init410367655698285346.clj
• startlein.sh
• ccw-lein-props

[GoogleCodeExporter]

Well, if starting on the command line the same thing as what  CCW did does not 
work, things start to get real real weird.

2 additional ideas:
- also inject / export the ENV variables in startlein.sh, but first in the 
script remove all existing environment variables via : 

unset $(/usr/bin/env | egrep '^(\w+)=(.*)$' | \
  egrep -vw 'PWD|USER|LANG' | /usr/bin/cut -d= -f1);

(from 
http://crashingdaily.wordpress.com/2008/03/07/clearing-environment-variables-in-
bash-scripts/ )

- Check whether there is an old clojure.jar somewhere in the JVM bootclasspath 
or exts folder 

Original comment by laurent....@gmail.com on 17 Aug 2014 at 12:49

[GoogleCodeExporter]

For the record, I have found similarly reported errors in this mailing list 
thread: 

https://groups.google.com/forum/m/#!topic/clojure/k0_juEDn_Hc

Original comment by laurent....@gmail.com on 17 Aug 2014 at 10:56

[GoogleCodeExporter]

If you switch ccw back to not using leiningen for launching the repl (flag in 
the repl or global clojure preferences), does it change something ?

Original comment by laurent....@gmail.com on 17 Aug 2014 at 10:58

[GoogleCodeExporter]

According to the line where the exception is thrown ( 
https://github.com/clojure/clojure/blob/clojure-1.6.0/src/clj/clojure/core.clj#L
5492 ), it seems like the namespace is already registered as loaded in 
*loaded-libs*, so require does not try to load it! but just to find it via 
find-ns.

And it may be this call to find-ns which causes the exception 

Original comment by laurent....@gmail.com on 17 Aug 2014 at 11:30

[GoogleCodeExporter]

Disabling Lein to launch the REPL works perfectly...

I'll give it a try adding the env vars after stripping them in the next hour.

I can check again for an old clojure version brought somewhere again but that 
was
my first suspicion and I did not find any. Most of the libs now exclude it
explicitly. Lein deps :tree has been of no help. 

I'll dig from the jvm to see what's alive on the class path.

Original comment by lprefont...@softaddicts.ca on 17 Aug 2014 at 11:37

[GoogleCodeExporter]

Forgot to add, launching lein repl at the project root allows me to require
the reducers...

Original comment by lprefont...@softaddicts.ca on 17 Aug 2014 at 11:38

[GoogleCodeExporter]

Analyzing a little deeper, it seems that clojure.lang.Namespace/namespaces map 
does not contain the namespace, while *loaded-libs* does. 

So certainly reducers are loaded during the initialization process of the 
project JVM, and something happens (an exception thrown?) during the load which 
creates this desynchronization. 

I've always been unconvinced by this namespace / lib dichotomy in clojure, and 
it seems that it is biting us right now, obscuring the real problem. 

Original comment by laurent....@gmail.com on 17 Aug 2014 at 11:39

[GoogleCodeExporter]

Had the time to retest manual start with the env. variables exported and 
grenchman.
Working, perfectly...

I will compare the classpaths of the manual startup and the CCW Lein startup
and the CCW Java startup modes.

I'll automate that a bit. Maybe this will shed some light.

I read the post that you found but we have been using Java 7 for a while to 
avoid 
the forkjoin issue entirely. There's no more Java 6 jre or sdk in our 
environments
to avoid confusion.

Reducers were the prime reason from moving away from Java 6 :) Ironic isn'it ?

Thank you,

Original comment by lprefont...@softaddicts.ca on 17 Aug 2014 at 11:46

[GoogleCodeExporter]

Oh... looks like we are approaching something fundamental here.

It maybe that Clara has an internal error trying to load reducers at compile
time.
I avoid most of these issues because we always AOT our stuff before shipping it.

I'll setup AOT for this test project to see if we can nail it better.
It would at least split the investigation between compilation time 
and run time.

Original comment by lprefont...@softaddicts.ca on 17 Aug 2014 at 11:54

[GoogleCodeExporter]

Got it:

Exception in thread "main" java.io.IOException: File name too long, 
compiling:(clara/rules/compiler.clj:498:52)
    at clojure.lang.Compiler.analyzeSeq(Compiler.java:6651)
    at clojure.lang.Compiler.analyze(Compiler.java:6445)
    at clojure.lang.Compiler.analyze(Compiler.java:6406)
    at clojure.lang.Compiler$InvokeExpr.parse(Compiler.java:3665)
    at clojure.lang.Compiler.analyzeSeq(Compiler.java:6646)
    at clojure.lang.Compiler.analyze(Compiler.java:6445)
    at clojure.lang.Compiler.analyze(Compiler.java:6406)
    at clojure.lang.Compiler$LetExpr$Parser.parse(Compiler.java:6012)
    at clojure.lang.Compiler.analyzeSeq(Compiler.java:6644)
    at clojure.lang.Compiler.analyze(Compiler.java:6445)
    at clojure.lang.Compiler.analyzeSeq(Compiler.java:6632)

I do not know why the compiler fails but I'll report this to the author of 
Clara.
I think we should seriously get lib authors to at least do an AOT compilation 
check
before delivering a new version.

On the CCW side of things I cannot see how we can improve this.
The error is probably trapped and hidden at some point at runtime.

Original comment by lprefont...@softaddicts.ca on 17 Aug 2014 at 12:08

[GoogleCodeExporter]

Yeah. 

Still, I don't understand why parts of clojure get desynchronized. Surely there 
is one thing which is updated too early...

Original comment by laurent....@gmail.com on 17 Aug 2014 at 12:17

[GoogleCodeExporter]

I reported the issue on github to the author of Clara. Referenced the ticket 
here.
Will pull myself a copy of Clara and will try to AOT it. This will tell us
if it's inherent to the lib (which would be surprising, others have been using 
it)
or if it's something in our stack here.

Original comment by lprefont...@softaddicts.ca on 17 Aug 2014 at 12:20

[GoogleCodeExporter]

To complement this issue, this is what I found and posted on the Clara issue 
list:

I cloned the project, added aot directives and was able to reproduce the 
problem.
I was able to avoid it by moving the project at the top of my home folder.
Looking at the class files generated, I found some with unusual file name 
length:

target/classes/clara/rules/compiler$fn__369$to_beta_tree__370$fn__371$iter__372_
_378$fn__379$iter__374__380$fn__381$fn__382$iter__383__387.class
target/classes/clara/rules/compiler$fn__369$to_beta_tree__370$fn__371$iter__372_
_378$fn__379$iter__374__380$fn__381$fn__382$iter__383__387$fn__388$fn__389.class

The last one is 134 characters long. My laptop uses ecryptfs which limits file 
name length to around 145 characters since the encrypted file name is longer 
and needs to respect the underlying file system 256 characters limit.

The Clojure compiler does not react on this error because it is a very low 
level error, somewhat unexpected.

Having coded on critical systems were you have to assume that air is not 
breathable 
anymore, it's understandable that this went unnoticed.

You can close this CCW issue. I asked the author of Clara if he was open for 
some
changes to work around this.

Original comment by lprefont...@softaddicts.ca on 17 Aug 2014 at 2:22

[GoogleCodeExporter]

Hi,

When a repl is launched from CCW, event w/o AOT directives, I end up with class
files under the classes folder.

I do not when I launch the lein repl directly.

Is the compilation driven by CCW through the Eclipse build facility before 
launching
the REPL ?
If yes then how come the compiler exception is not displayed ?

Clearly the compiler reports the exception as we can see when using AOT
but it's not propagated upward.

It's not a compilation error, it's an IOException reported by the compiler.
Would this explain why we have no output either at the console or the REPL 
window ?

Original comment by lprefont...@softaddicts.ca on 18 Aug 2014 at 10:53

[GoogleCodeExporter]

To clarify my previous comment, from the lein REPL at the project root
w/o AOT enabled this time:

user=> *compile-path*
"/home/lprefontaine/workspaces/hygiebus/TestClara/target/classes"
user=> (compile 'clara.rules.compiler)

CompilerException java.io.IOException: File name too long, 
compiling:(clara/rules/compiler.clj:498:52) 
user=> (class *e)
clojure.lang.Compiler$CompilerException
user=> (require '[clojure.core.reducers :as r])
nil
user=> r/fold
#<reducers$fold clojure.core.reducers$fold@2527ad0d>
user=> (require '[clara.rules.compiler :as ru])
nil
user=> ru/BetaNode
(either clara.rules.engine.ProductionNode clara.rules.engine.QueryNode 
clara.rules.engine.JoinNode clara.rules.engine.NegationNode 
clara.rules.engine.TestNode clara.rules.engine.AccumulateNode)
user=> 

The compilation error seems to be wrapped in a compiler exception like any 
other 
compilation error.

I cannot compile the clara.rules.compiler name space from the CCW repl,
context is screwed up (cannot require reducers, ...) as soon as I get the REPL 
prompt.

Original comment by lprefont...@softaddicts.ca on 18 Aug 2014 at 11:08

[GoogleCodeExporter]

Original comment by laurent....@gmail.com on 21 Aug 2014 at 8:38

• Added labels: 0.28
• Removed labels: next

[GoogleCodeExporter]

Original comment by laurent....@gmail.com on 25 Sep 2014 at 9:27

• Added labels: 0.28.1
• Removed labels: 0.28

[GoogleCodeExporter]

Original comment by laurent....@gmail.com on 20 Nov 2014 at 4:21

• Removed labels: 0.28.1