search

ZuInnoTe / hadoopcryptoledger

Hadoop Crypto Ledger - Analyzing CryptoLedgers, such as Bitcoin Blockchain, on Big Data platforms, such as Hadoop/Spark/Flink/Hive
Apache License 2.0
141 stars 51 forks source link

BUILD.md - how to build source from scratch? #60

Closed rounsavall closed 5 years ago

rounsavall commented 5 years ago

Do you have instructions on how to build hadoopcryptoledger from source, without pre-built components?

jornfranke commented 5 years ago

Yes it is in the wiki https://github.com/ZuInnoTe/hadoopcryptoledger/wiki

On 15. Sep 2018, at 00:31, rounsavall notifications@github.com wrote:

Do you have instructions on how to build hadoopcryptoledger from source, without pre-built components?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

jornfranke commented 5 years ago

Is the information there sufficient? Is the format as Wiki acceptable?

rounsavall commented 5 years ago

Thanks Jörn,

There are instructions on individual projects on how to build with gradle, but all of these depend on prebuilt components from this repo in maven. For example, I have seen:

Could not resolve all files for configuration ':hiveudf:runtime'. Could not find com.github.zuinnote:hadoopcryptoledger-fileformat:1.2.0. Searched in the following locations: …

I am also seeing

Could not resolve all files for configuration ':hiveudf:provided'. Could not find org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde. Searched in the following locations: ...

I wanted to understand how to just build everything locally with only the apache Hadoop MR, Flink and Spark pre-builts from maven repos. As well as which versions of the pre-built libraries need to be in the maven repos

jornfranke commented 5 years ago

There is no problem to achieve what you want. You can build the input format and publish it to a local repo, then use the other projects and they will take the dependency from the local maven repository (or an remote repository). There is even no need to change the build files in the case of a full local repository. The instructions exactly describe this and others have followed them.

However maybe your problem is not clear to me.

On 20. Sep 2018, at 01:53, rounsavall notifications@github.com wrote:

Thanks Jörn,

There are instructions on individual projects on how to build with gradle, but all of these depend on prebuilt components from this repo in maven. For example, I have seen:

Could not resolve all files for configuration ':hiveudf:runtime'. Could not find com.github.zuinnote:hadoopcryptoledger-fileformat:1.2.0. Searched in the following locations: …

I wanted to understand how to just build everything locally with only the apache Hadoop MR, Flink and Spark pre-builts from maven repos.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

rounsavall commented 5 years ago

I figured it out. I essentially have to force download of the local maven repo with all these libraries that the gradle scripts miss so that they get picked up.

The gradle scripts only really work once that repo is locally available.

The scripts as written assume one already has prebuilt binaries available from public maven repo.

rounsavall commented 5 years ago

It would be good to hook in a step to populate the local maven repo.

jornfranke commented 5 years ago

Hmm, I am not sure why this was necessary for you, because all others so far could publish a changed hadoopcryptoledger library the local Maven repository with the instructions provided. In fact, this is how changes are implemented and tested. Maybe you can elaborate what you mean by "populate local Maven repository".

On Thu, Oct 4, 2018 at 10:04 PM rounsavall notifications@github.com wrote:

It would be good to hook in a step to populate the local maven repo.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ZuInnoTe/hadoopcryptoledger/issues/60#issuecomment-427151213, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlR5BSEyHoerk-Z1NZdRcjUlYuP5jGxks5uhmm2gaJpZM4WqCYS .