PbfIterator fails

[rrrnld]

Hi and first of all thanks for the library. :)

I'm trying to use it from Clojure (not sure if this is relevant) and I have some issues using the PbfIterator. I can construct the object, but iterating it fails:

(-> (io/input-stream "path-to-osm.pbf")
    (PbfIterator. false) ;; works
    (iterator-seq)) ;; fails

My dependencies are as follows:

{:deps
 {de.topobyte/osm4j-xml {:mvn/version "1.2.0"}
  de.topobyte/osm4j-pbf {:mvn/version "1.2.0"}
  de.topobyte/osm4j-tbo {:mvn/version "1.2.0"}
  de.topobyte/osm4j-geometry {:mvn/version "1.2.0"}
  factual/geo {:git/url "https://github.com/Factual/geo" :sha "75722b097d559b5d073aa673fb35bfe69663b50b"}}

 :mvn/repos
 {"central" {:url "https://repo1.maven.org/maven2/"}
  "clojars" {:url "https://clojars.org/repo"}
  "topobyte" {:url "http://mvn.topobyte.de"}
  "slimjars" {:url "http://mvn.slimjars.com"}}}

This is the error message:

Unhandled java.lang.NoClassDefFoundError
   Could not initialize class
   de.topobyte.osm4j.pbf.protobuf.Fileformat$BlobHeader

              PbfUtil.java:   92  de.topobyte.osm4j.pbf.util.PbfUtil/parseHeader
              PbfUtil.java:   80  de.topobyte.osm4j.pbf.util.PbfUtil/parseHeader
          PbfIterator.java:  129  de.topobyte.osm4j.pbf.seq.PbfIterator/advanceBlock
          PbfIterator.java:  120  de.topobyte.osm4j.pbf.seq.PbfIterator/tryAdvanceBlock
          PbfIterator.java:   77  de.topobyte.osm4j.pbf.seq.PbfIterator/hasNext
                   RT.java:  517  clojure.lang.RT/chunkIteratorSeq
                  core.clj: 5729  clojure.core/iterator-seq
                  core.clj: 5721  clojure.core/iterator-seq
                      REPL:   76  wtf.nein.kosmopolit.io/eval10140
                      REPL:   74  wtf.nein.kosmopolit.io/eval10140
             Compiler.java: 7177  clojure.lang.Compiler/eval
             Compiler.java: 7132  clojure.lang.Compiler/eval
                  core.clj: 3214  clojure.core/eval
                  core.clj: 3210  clojure.core/eval
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  665  clojure.core/apply
                  core.clj: 1973  clojure.core/with-bindings*
                  core.clj: 1973  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  665  clojure.core/apply
                  core.clj:  660  clojure.core/apply
                regrow.clj:   20  refactor-nrepl.ns.slam.hound.regrow/wrap-clojure-repl/fn
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  152  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  202  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  201  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  834  java.lang.Thread/run

To make things more complicated I recently switched to using NixOS, so it might very well be that my setup is so unusual that you say you can't support it.

Do you have any idea however how I could debug this? I have no idea how the protocolbuffers bindings are generated or how they work. From what I've read the compiler generates java bindings, so why are there missing classes? Do I need some global packages for this?

I'd be very happy if this could somehow be made to work. Thanks in advance!

[sebkur]

Hi, the missing class de.topobyte.osm4j.pbf.protobuf.Fileformat$BlobHeader should be contained in the dependency that you are already including in your project via de.topobyte/osm4j-pbf {:mvn/version "1.2.0"}

No idea if it has anything to do with NixOS. Did it work with a different OS? I'd like to take a look myself, but I don't know how to set up a Closure project. If you provide a minimal Closure project that tries to read a PBF file, I'm happy to take a look.

[rrrnld]

Thanks for the quick reply. :)

I tried producing a minimal example:

deps.edn in project root:

{:deps
 {de.topobyte/osm4j-pbf {:mvn/version "1.2.0"}}

 :mvn/repos
 {"central" {:url "https://repo1.maven.org/maven2/"}
  "clojars" {:url "https://clojars.org/repo"}
  "topobyte" {:url "http://mvn.topobyte.de"}
  "slimjars" {:url "http://mvn.slimjars.com"}}}

example.clj in the src folder:

(ns example
  (:import [de.topobyte.osm4j.pbf.seq PbfIterator])
  (:require [clojure.java.io :as io]))

(-> (io/input-stream "resources/bremen-latest.osm.pbf")
    (PbfIterator. false)
    (iterator-seq)
    (count))

(You can also stat a repl via clj and paste the contents of the file to try it out interactively.)

Interestingly that didn't break. Being too lazy to bisect the problem for now and following your comment I changed the osm4j-pbf dependency in the problematic project to de.topobyte/osm4j-pbf-full-runtime {:mvn/version "1.2.0"} and I can say that this fixed the problem. :)

This soliution works well enough for me, I don't know if you see a reason to dig deeper. If not I think this issue can be closed.

[sebkur]

OK, good to hear that it's working with the full runtime. There are two versions of the pbf library, one using the lite runtime of protocol buffer and the other using the full / main runtime of protocol buffers. Maybe the lite runtime isn't working well with Clojure.

Thanks anyways for the minimal example!