Thaw fails when nippy comes from AOT jar

[mbossenbroek]

We're seeing a problem where thaw doesn't work for records if nippy was included via an AOT'ed jar.

Here's a repro: https://github.com/mbossenbroek/pigpen-graph-test

If you comment out the pigpen dependency in project.clj, it works fine. Any idea why this happens if nippy is AOT'ed into another jar?

I also tried this with nippy version 2.8.0 (recompiling pigpen locally) and got the same result.

cc @aconbere

[ptaoussanis]

Hey Matt, what's the urgency on this?

[aconbere]

I'm not sure how @mbossenbroek feels, it's blocking some work that I'm doing, so I'd of course love to see it addressed. That being said, this seems to me to be a pretty unique set of intersecting problems. I don't know of anyone else that I've seen on the internet talking about it.

[mbossenbroek]

Entirely up to @aconbere :)

This was found while using Prismatic's graph library with PigPen. I think we have a few workarounds though (disabling the use of records, not using AOT)

[ptaoussanis]

Where is the GitHub repo for [com.netflix.pigpen/pigpen "0.2.13"]?

[mbossenbroek]

https://github.com/Netflix/PigPen

It's up to 0.2.15 at the moment, but the last two releases were only minor changes. Everything related to this should be the same.

[ptaoussanis]

Could you get me some context? Has anything changed recently with PigPen that prompted this error? This is a new error?

[mbossenbroek]

I don't think that any recent changes would have triggered this. My guess is that no one has ever used pigpen with records before (or didn't tell me that it didn't work). I would speculate that this has always been a problem & we never noticed.

[ptaoussanis]

I would speculate that this has always been a problem & we never noticed.

Okay, makes sense - thanks.

Any idea why this happens if nippy is AOT'ed into another jar?

And could you clarify what you mean here by Nippy being "AOT'ed into another jar"? I'm looking over the PigPen source but am not familiar at all. Are you depending on Nippy internally? Where?

[mbossenbroek]

pigpen-core depends on nippy here: https://github.com/Netflix/PigPen/blob/master/pigpen-core/build.gradle#L7

When switching on AOT, clojuresque seems to take all dependencies and AOT them into the pigpen jar:

jar tf ~/.m2/repository/com/netflix/pigpen/pigpen/0.2.13/pigpen-0.2.13.jar | grep nippy
taoensso/nippy$loading__4958__auto__.class
taoensso/nippy$fn__4174.class
taoensso/nippy$when_debug_mode.class
taoensso/nippy$fn__4179.class
taoensso/nippy$fn__4182$G__4178__4185.class
taoensso/nippy$fn__4182$G__4177__4189.class
taoensso/nippy$fn__4182.class
taoensso/nippy$write_id.class
...

It has the effect of making pigpen an uberjar of sorts.

[ptaoussanis]

clojuresque is this?

Can you provide steps for how to recompile a local PigPen copy using a (local) debug build of Nippy?

[mbossenbroek]

Yep - it's the clojure support for gradle builds.

Rebuilding should be this:

$ ./gradlew clean build install

Just change the build.gradle mentioned above to

    compile 'com.taoensso:nippy:2.8.1-SNAPSHOT'

or whatever the name is. Let me know if that works for you.

[ptaoussanis]

Sorry, am struggling to get the build working so that I can run lein test against the local Gradle build. Will try come back to this on Monday.

The more info you can get me, the more likely I'll be able to help; just juggling some high-priority work atm. Not at all familiar with PigPen or Gradle.

Nippy's record de/serialization is pretty simple:

• During freeze, the record's class name is serialized.
• During thaw, a call is made to (Class/forName <serialized-class-name>).

The actual content serialization is just done as a PersistentMap.

For thawing to work, the thawing JVM process needs to recognize the serialized class name. I suspect that it's Class/forName that's failing, but would like a debug build to confirm.

[ptaoussanis]

When switching on AOT

How are you switching on/off AOT with Gradle?

[mbossenbroek]

No worries - it can be confusing :)

Here's what I just ran:

$ git clone git@github.com:ptaoussanis/nippy.git
$ vim project.clj
##    ^ Change the version to 2.8.0-SNAPSHOT
$ lein install
$ cd ..
$ git clone git@github.com:Netflix/PigPen.git pigpen2
$ cd pigpen2
$ vim pigpen-core/build.gradle
##    ^ Change the nippy version to 2.8.0-SNAPSHOT
$ ./gradlew clean build install
$ cd ..
$ git clone git@github.com:mbossenbroek/pigpen-graph-test.git
$ cd pigpen-graph-test
$ vim project.clj
##    ^ Change the pigpen version to 0.2.16-SNAPSHOT
$ lein test

The AOT flag for pigpen builds is here: https://github.com/Netflix/PigPen/blob/master/build.gradle#L42

Just remove that line to disable AOT.

Let me know if you need more info.

[mbossenbroek]

Were you able to find anything with this?

Thanks, Matt

[ptaoussanis]

Hi Matt, Anders. Sorry about that, have been struggling to find any time to look into this again. Realistically, might not have the time for the next week or two at least (?). Currently travelling + am on some high-priority work.

I'll keep this issue open and let you know if any unexpected opportunities come up. Would suggest one of your workarounds in the meantime if that's acceptable (disabling the use of records, not using AOT).

Sorry I couldn't be of more immediate assistance.

[mbossenbroek]

Thanks for the update. If I get some time, I'll try digging into the compiled jar to see what's different.

[ptaoussanis]

Sure. I'd start with the info here and maybe try a debug build that'll let you know where along the thaw process the error is popping up.

That was my intention before I ran into issues with the Gradle build.

[aconbere]

@ptaoussanis what's the process of producing a debug build? I see the when-debug-mode macro, but not any information on flipping the project into debug mode. (Which I can just change the defn of the macro, but I wasn't sure if there is a "right" way)

[mbossenbroek]

@ptaoussanis Anders found that my instructions above were missing a step. You also need to add mavenLocal() to pigpen's build.gradle file:

 allprojects {
     repositories {
+        mavenLocal()
         mavenCentral() // maven { url: 'http://jcenter.bintray.com' }
         maven { url 'http://clojars.org/repo' }
     }

[ptaoussanis]

Sorry for the delay replying guys, just concluded my last trip yesterday so schedule's back to normal from today.

what's the process of producing a debug build? I see the when-debug-mode macro, but not any information on flipping the project into debug mode. (Which I can just change the defn of the macro, but I wasn't sure if there is a "right" way)

@aconbere There's a custom debug build up on Clojars at [com.taoensso/nippy "2.8.0-debug1"]. If you could run that for me and get me the output, that should give some good basic starting info and we can go from there?

Cheers! :-)