Compiling on mac and other issues

[drewstone]

The fix

I don't have a good grasp on making Makefiles so versatile, but wanted to document the issues I faced compiling/building this repo on a MacBook 13" Mid-2014 with MacOS High Sierra version 10.13.6 and how I resolved them to build and compile the contracts according to an updated MakeFile.

Dependences:

Yarn - Make sure you have yarn installed as this will be the node package manager used.

I ran into issues initially compiling libsnark, which has seen similar issues on their board. The solution that I found was to include the following in my ~/.bash_profile and use the makefile located in https://github.com/drewstone/ethsnarks with Node 10.9.0 (I use n for nodejs management.

LD_LIBRARY_PATH=/usr/local/opt/openssl/lib:"${LD_LIBRARY_PATH}"                    
CPATH=/usr/local/opt/openssl/include:"${CPATH}"                                    
PKG_CONFIG_PATH=/usr/local/opt/openssl/lib/pkgconfig:"${PKG_CONFIG_PATH}"          
export LD_LIBRARY_PATH CPATH PKG_CONFIG_PATH 

Hopefully this helps someone running on Mac in the future.

For clarity, the entirety of the makefile is below:

ROOT_DIR := $(shell dirname $(realpath $(MAKEFILE_LIST)))

PYTHON ?= python3
NAME ?= ethsnarks
NPM ?= yarn
GANACHE ?= $(ROOT_DIR)/node_modules/.bin/ganache-cli
TRUFFLE ?= $(ROOT_DIR)/node_modules/.bin/truffle

COVERAGE = $(PYTHON) -mcoverage run --source=$(NAME) -p

#######################################################################

all: build/src/libmiximus.so truffle-compile

clean: coverage-clean
    rm -rf build
    find . -name '__pycache__' -exec rm -rf '{}' ';'

#######################################################################

build:
    mkdir -p build

bin/miximus_genKeys: build/Makefile
    make -C build

build/src/libmiximus.so: build/Makefile
    make -C build

build/Makefile: build CMakeLists.txt
    git submodule update --init --recursive && cd build && CPPFLAGS=-I/usr/local/opt/openssl/include LDFLAGS=-L/usr/local/opt/openssl/lib PKG_CONFIG_PATH=/usr/local/opt/openssl/lib/pkgconfig cmake  .. -DWITH_PROCPS=OFF -DWITH_SUPERCOP=OFF

#######################################################################

.PHONY: test
test: cxx-tests truffle-test python-test

python-test:
    $(COVERAGE) -m unittest discover test/

cxx-tests:
    ./bin/test_longsightf
    ./bin/test_one_of_n
    ./bin/test_shamir_poly
    ./bin/test_sha256_full_gadget
    ./bin/test_field_packing > /dev/null
    time ./bin/hashpreimage_cli genkeys zksnark_element/hpi.pk.raw zksnark_element/hpi.vk.json
    ls -lah zksnark_element/hpi.pk.raw
    time ./bin/hashpreimage_cli prove zksnark_element/hpi.pk.raw 0x9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a089f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 zksnark_element/hpi.proof.json
    time ./bin/hashpreimage_cli verify zksnark_element/hpi.vk.json zksnark_element/hpi.proof.json
    time ./bin/test_load_proofkey zksnark_element/hpi.pk.raw

#######################################################################

coverage: coverage-combine coverage-report

coverage-clean:
    rm -rf .coverage .coverage.* htmlcov

coverage-combine:
    $(PYTHON) -m coverage combine

coverage-report:
    $(PYTHON) -m coverage report

coverage-html:
    $(PYTHON) -m coverage html

#######################################################################

python-dependencies: requirements requirements-dev

requirements:
    $(PYTHON) -m pip install -r requirements.txt

requirements-dev:
    $(PYTHON) -m pip install -r requirements-dev.txt

fedora-dependencies:
    dnf install procps-ng-devel gmp-devel boost-devel cmake g++

ubuntu-dependencies:
    apt-get install cmake make g++ libgmp-dev libboost-all-dev libprocps-dev

zksnark_element/pk.json: ./bin/miximus_genKeys
    $< 3 zksnark_element/pk.json zksnark_element/vk.json

#######################################################################

nvm-install:
    ./utils/nvm-install
    nvm install --lts

node_modules:
    $(NPM) install

$(TRUFFLE): node_modules

$(GANACHE): node_modules

.PHONY: truffle-test
truffle-test: $(TRUFFLE)
    $(NPM) run test

truffle-compile: $(TRUFFLE)
    $(TRUFFLE) compile

testrpc: $(TRUFFLE)
    $(NPM) run testrpc

#######################################################################

python-pyflakes:
    $(PYTHON) -mpyflakes $(NAME)

python-pylint:
    $(PYTHON) -mpylint $(NAME)

[HarryR]

Looks good. I will test on my Mac.

I use nvm on Linux to manage node versions, the nvm-install script will install the most recent LTS version of Node and NPM.

It would be good to be able to use either npm or yarn, depending on what's available.

re:

LD_LIBRARY_PATH=/usr/local/opt/openssl/lib:"${LD_LIBRARY_PATH}"                    
CPATH=/usr/local/opt/openssl/include:"${CPATH}"                                    
PKG_CONFIG_PATH=/usr/local/opt/openssl/lib/pkgconfig:"${PKG_CONFIG_PATH}"          
export LD_LIBRARY_PATH CPATH PKG_CONFIG_PATH 

I think these rely on brew aren't available on Mac as standard.

There are Makefile targets, fedora-dependencies: and ubuntu-dependencies: to install the necessary dependencies for Fedora and Ubuntu. What would be necessary for Mac?

[drewstone]

Gotcha, yea I think nvm wasn't causing issues for me in that case. As for dependencies for mac, I guess we only need openssl crypto libraries, which I have gotten from brew.

[HarryR]

Ooh, Travis-CI supports OSX: https://docs.travis-ci.com/user/reference/osx

Building on multiple platform: https://docs.travis-ci.com/user/multi-os/

[HarryR]

I have an initial build with Travis-CI on OSX: https://travis-ci.org/HarryR/ethsnarks/jobs/419645264

One of the tests fail, and I broke the Linux build on Travis at the same time, but it's a good start.

Might need to take this into account for OSX? - https://bitbucket.org/gallen/libvariant/pull-requests/2/resolves-cmake-warning-pertaining-to-rpath/diff

[HarryR]

I've fixed the multi-OS travis build for Linux, and OSX is building but failing on one of the tests.

See: https://travis-ci.org/HarryR/ethsnarks/builds/419717071

./bin/test_sha256_full_gadget
full_output_bytes mismatch
FAIL

I can't reproduce that test failure with Clang on Linux.