search

saurfang / sparksql-protobuf

Read SparkSQL parquet file as RDD[Protobuf]
http://spark-packages.org/package/saurfang/sparksql-protobuf
Apache License 2.0
93 stars 36 forks source link

[Question] How to build the project properly? #10

Closed datng2 closed 5 years ago

datng2 commented 5 years ago

Hi saurfang,

Awesome project. I am trying to build it locally. This is my very first time exposing to sbt build tool. I am trying to run sbt compile, but it failed due to not be able to to pick up protobuf and @transient.

~/internship/sparksql-protobuf on  master! ⌚ 9:06:21
$ protoc --version
libprotoc 2.6.1

$ tree -L 2 -d
.
├── lib .     // output of my compiled protobuf 2.6.1
│   ├── bin
│   ├── include
│   └── lib
├── project
│   ├── project
│   └── target
├── src
│   ├── main
│   └── test
└── target
    ├── protobuf_external
    ├── resolution-cache
    ├── scala-2.11
    └── streams

When I am trying to compile the project, sbt throws the following errors:

$ sbt compile -Xlint
[info] Loading global plugins from /Users/datng2/.sbt/0.13/plugins
[info] Loading project definition from /Users/datng2/internship/sparksql-protobuf/project
Missing bintray credentials /Users/datng2/.bintray/.credentials. Some bintray features depend on this.
[info] Set current project to sparksql-protobuf (in build file:/Users/datng2/internship/sparksql-protobuf/)
[warn] Credentials file /Users/datng2/.bintray/.credentials does not exist
[warn] Credentials file /Users/datng2/.ivy2/.sbtcredentials does not exist
[info] Compiling 5 Scala sources and 6 Java sources to /Users/datng2/internship/sparksql-protobuf/target/scala-2.11/classes...
[warn] /Users/datng2/internship/sparksql-protobuf/src/main/scala/com/github/saurfang/parquet/proto/spark/ProtoParquetRDD.scala:18: no valid targets for annotation on value conf - it is discarded unused. You may specify targets with meta-annotations, e.g. @(transient @param)
[warn]                                                         @transient conf: Configuration
[warn]                                                          ^
[warn] one warning found
[error] /Users/datng2/internship/sparksql-protobuf/src/test/generated/com/github/saurfang/parquet/proto/AddressBook.java:79: error: cannot find symbol
[error]     com.google.protobuf.ProtocolStringList
[error]                        ^
[error]   symbol:   class ProtocolStringList
[error]   location: package com.google.protobuf
[error] /Users/datng2/internship/sparksql-protobuf/src/test/generated/com/github/saurfang/parquet/proto/AddressBook.java:1367: error: cannot find symbol
[error]     public com.google.protobuf.ProtocolStringList
[error]                               ^
[error]   symbol:   class ProtocolStringList
[error]   location: package com.google.protobuf
[error] /Users/datng2/internship/sparksql-protobuf/src/test/generated/com/github/saurfang/parquet/proto/AddressBook.java:2247: error: cannot find symbol
[error]       public com.google.protobuf.ProtocolStringList
[error]                                 ^
[error]   symbol:   class ProtocolStringList
[error]   location: package com.google.protobuf
[error] /Users/datng2/internship/sparksql-protobuf/src/test/generated/com/github/saurfang/parquet/proto/AddressBook.java:219: error: cannot find symbol
[error]           address_ = address_.getUnmodifiableView();
[error]                              ^
[error]   symbol:   method getUnmodifiableView()
[error]   location: variable address_ of type LazyStringList
[error] /Users/datng2/internship/sparksql-protobuf/src/test/generated/com/github/saurfang/parquet/proto/AddressBook.java:1692: error: cannot find symbol
[error]           address_ = address_.getUnmodifiableView();
[error]                              ^
[error]   symbol:   method getUnmodifiableView()
[error]   location: variable address_ of type LazyStringList
[error] /Users/datng2/internship/sparksql-protobuf/src/test/generated/com/github/saurfang/parquet/proto/AddressBook.java:2249: error: cannot find symbol
[error]         return address_.getUnmodifiableView();
[error]                        ^
[error]   symbol:   method getUnmodifiableView()
[error]   location: variable address_ of type LazyStringList
[error] Note: /Users/datng2/internship/sparksql-protobuf/src/main/java/com/github/saurfang/parquet/proto/ProtoLISTReadSupport.java uses or overrides a deprecated API.
[error] Note: Recompile with -Xlint:deprecation for details.
[error] Note: Some input files use unchecked or unsafe operations.
[error] Note: Recompile with -Xlint:unchecked for details.
[error] 6 errors
[error] (compile:compile) javac returned nonzero exit code
[error] Total time: 6 s, completed Feb 8, 2019 9:14:21 AM

Any pointer or help would be greatly appreciated. Thanks :)

datng2 commented 5 years ago

It turns out that the problem is related to protobuf backward compatibility. Particularly, it seems that protobuf 2.6.1 does not have ProtocolStringList while 2.5.0 does.

In order to fix it, I have to explicitly set proto verion in build.sbt

...
version in PB.protobufConfig := "2.6.1"
sourceDirectory in PB.protobufConfig := new File("src/test/protobuf")