replikativ / hasch

Cross-platform (JVM and JS atm.) edn data structure hashing for Clojure.
Eclipse Public License 1.0
111 stars 13 forks source link

Explicit dependencies #2

Closed jcf closed 7 years ago

jcf commented 7 years ago

As per the discussion in #1, I've made a few small changes to project.clj to enable testing of various Clojure(script) versions.

It may be possible to get Clojure 1.6 working, but I haven't tried. This is the exception that occurs when trying to run those tests:

Exception in thread "main" java.io.FileNotFoundException: Could not locate cljs/compiler__init.class or cljs/compiler.clj on classpath: , compiling:(cemerick/austin.clj:1:1)
    at clojure.lang.Compiler.load(Compiler.java:7142)
    at clojure.lang.RT.loadResourceScript(RT.java:370)
    at clojure.lang.RT.loadResourceScript(RT.java:361)
    at clojure.lang.RT.load(RT.java:440)
    at clojure.lang.RT.load(RT.java:411)
    at clojure.core$load$fn__5066.invoke(core.clj:5641)
    at clojure.core$load.doInvoke(core.clj:5640)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.core$load_one.invoke(core.clj:5446)
[...]

In terms of reducing dependencies, Incognito pulls in quite a bit. I'm not familiar with what Incognito does so don't know if these deps can be excluded.

 [io.replikativ/incognito "0.2.0"]
   [com.cognitect/transit-clj "0.8.285"]
     [com.cognitect/transit-java "0.8.311"]
       [com.fasterxml.jackson.core/jackson-core "2.3.2"]
       [commons-codec "1.10"]
       [org.msgpack/msgpack "0.6.10"]
         [com.googlecode.json-simple/json-simple "1.1.1" :exclusions [[junit]]]
         [org.javassist/javassist "3.18.1-GA"]
   [com.cognitect/transit-cljs "0.8.232"]
     [com.cognitect/transit-js "0.8.755"]
   [org.clojure/data.fressian "0.2.0"]
     [org.fressian/fressian "0.6.3"]
jcf commented 7 years ago

Interestingly, CircleCI won't build this feature branch. Instead they've apparently tested master, and left it at that. :laughing:

Update: I've tricked CircleCI by pushing to the master branch in my fork. Let's see what happens!

jcf commented 7 years ago

I don't think you'll get build notifications without turning on CircleCI in your repo (I've done it in my fork).

Here's the green build: https://circleci.com/gh/jcf/hasch/7

Note: The green build requires #3 in order to run lein all test, and verify all the versions of Clojure that currently work with Hasch.

whilo commented 7 years ago

Incognito provides tagged literal support. In particular hasch will create the same hash for both unknown literals (as read by incognito) and hashed records. This allows to have an open system of data replication without all peers knowing about all types, but still having the proper integrity check. I recommend to use :exclude to drop incognito if you know you don't need it (i.e. you will never read with incognito). Maybe the protocol extension can be determined by incognito on runtime instead of doing it in platform.clj, but then incognito would depend optionally on hasch. I think that this might cause some confusion, as the protocol extension must be established before the ones of IRecord and IPersistentHashMap. So I would prefer to keep the dependency on incognito unless this is transparently solvable.

whilo commented 7 years ago

Ah it cannot be excluded atm. for this reason of course.

whilo commented 7 years ago

We can exclude the dependencies on the serialization libraries in hasch though.

jcf commented 7 years ago

@whilo I'd think the hashing algorithm could live on its own in this library, and the extension of protocols etc. could happen elsewhere. It would certainly be nice to use the EDN hashing features without all of the Incognito literal stuff.

whilo commented 7 years ago

I mean hasch's own protocol, which is basically the core of the hashing algorithm. I have experienced that the order of the extensions matters and even if some are subtypes of others, but added later, the supertype will match first. That is why I first cover the Incognito-Literal before dealing with map types: https://github.com/replikativ/hasch/blob/master/src/hasch/platform.clj#L120

I could also have a special case in the protocol itself (which is against the very intention of protocols), like I have here for maps: https://github.com/replikativ/hasch/blob/master/src/hasch/platform.clj#L132 Honestly I am not sure how exactly the order is determined, I need to reinvestigate this. I will ping back when I know more.

whilo commented 7 years ago

https://clojuredocs.org/clojure.core/extend says:

This is primarily to facilitate interop
with the host (e.g. Java) but opens the door to incidental multiple
inheritance of implementation since a class can inherit from more
than one interface, both of which extend the protocol. It is TBD how
to specify which impl to use.

http://clojure.org/reference/protocols says:

You can implement a protocol on an interface this is primarily to facilitate
interop with the host (e.g. Java)  but opens the door to incidental multiple 
inheritance of implementation since a class can inherit from more than 
one interface, both of which implement the protocol if one interface is derived 
from the other, the more derived is used, else which one is used is unspecified.

So the order is unspecified, since defrecord emits both IRecord and IPersistentMap. IncognitoLiteral is also a record, so I am not sure how the resolution works here yet.

whilo commented 7 years ago

Incognito base is tiny, just < 30 LOC and no side-effects (glue code basically to not wire some serialization format). I will just exclude all the serialization libs, because I want hasch to be compatible with incognito serialization and hence need to call it in the map protocol handler.