Installation blockscipy

[effezeta88]

Hello guys, I'm trying to install the new version of blocksci (with the new instruction) but when I execute "pip3 install -e blockscipy" the installation give me the follow error:

Command "/usr/bin/python3 -c "import setuptools, tokenize;file='/opt/BlockSci/blockscipy/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" develop --no-deps" failed with error code 1 in /opt/BlockSci/blockscipy/

Could anyone help me?

Thanks :)

[hkalodner]

I @effezeta88,

Looks like there are some installation issues when there was a previous version of BlockSci already install. Try deleting /usr/local/include/blocksci, /usr/local/bin/blocksci* , /usr/local/bin/mempool_recorder*, and /usr/local/lib/libblocksci* to get rid of the old install and then reinstall the new version of BlockSci and blockscipy.

That's just my guess. The error you posted doesn't show very much. There's probably some more information above that in the terminal output with either a gcc or a cmake error.

[effezeta88]

Thank you for the answer @hkalodner :)

I haven't previous versions installed, all those folders were empty...by the way this is the full error stack:

_CMakeFiles/_blocksci.dir/build.make:86: recipe for target 'CMakeFiles/_blocksci.dir/src/cluster/cluster/cluster_range.cpp.o' failed make[2]: [CMakeFiles/_blocksci.dir/src/cluster/cluster/cluster_range.cpp.o] Error 1 make[2]: Waiting for unfinished jobs.... CMakeFiles/_blocksci.dir/build.make:62: recipe for target 'CMakeFiles/_blocksci.dir/src/cluster/cluster/cluster_py.cpp.o' failed make[2]: [CMakeFiles/_blocksci.dir/src/cluster/cluster/cluster_py.cpp.o] Error 1 CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/_blocksci.dir/all' failed make[1]: [CMakeFiles/_blocksci.dir/all] Error 2 Makefile:83: recipe for target 'all' failed make: *** [all] Error 2 Traceback (most recent call last): File "", line 1, in File "/opt/BlockSci/blockscipy/setup.py", line 76, in 'dateparser>=0.6.0' File "/usr/lib/python3.5/distutils/core.py", line 148, in setup dist.run_commands() File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands self.run_command(cmd) File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/usr/lib/python3/dist-packages/setuptools/command/develop.py", line 34, in run self.install_for_development() File "/usr/lib/python3/dist-packages/setuptools/command/develop.py", line 119, in install_for_development self.run_command('build_ext') File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/opt/BlockSci/blockscipy/setup.py", line 32, in run self.build_extension(ext) File "/opt/BlockSci/blockscipy/setup.py", line 57, in build_extension subprocess.check_call(['cmake', '--build', '.'] + build_args, cwd=self.build_temp) File "/usr/lib/python3.5/subprocess.py", line 581, in checkcall raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--config', 'Release', '--', '-j2']' returned non-zero exit status 2

I use cmake version 3.10.1 on ubuntu

[abirke]

Which g++ version are you using? I could solve a similar problem by making g++-7 the default as in earlier compilation instructions of BlockSci.

update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 60 --slave /usr/bin/g++ g++ /usr/bin/g++-7

In the current version of the compilation instructions g++-7 is only used for cmaking BlockSci, but there are also compilations triggered by the pip installation.

[hkalodner]

Ahhh. Awesome @abirke. I totally forgot to include that. @effezeta88 try running the install with CC=gcc-7 CXX=g++-7 sudo -H pip3 install -e blockscipy

[Voelundr]

Why does sudo -H pip3 install -e blockscipy take so long?

[hkalodner]

Essentially BlockSci and particularly blockscipy depends on multiple template heavy header only libraries leads to pretty crappy compile times. The main increase in compile times from v0.4.5 to v0.5.0 is due to the introduction of Range and Iterator classes to BlockSci which depends on a large amount of compile time work for the compiler.

It's an ongoing problem which I don't think there's a great solution to. The python binding module takes large amounts of time and memory to compile. I split it into a large number of files which helps reduce the memory/file required to compile, but makes the compile time even longer. I don't know a ton about compiler performance so it's possible that there are ways to improve the situation. I'm planning on continuing to explore if it's possible to improve this situation, but luckily, it doesn't need to be build very often, and incremental rebuilds with small changes should be much faster.

[Voelundr]

I needed the time_seen function for a Tx that seems to be new and thought to myself, "Ok I will install the new version, that will take about 10 minutes." And now I am sitting here and thinking: "Ok Feierabend" ;)

[hkalodner]

I don't remember exactly how long it takes from scratch, but it shouldn't be too bad, under 30 minutes I think.

Block.time_seen and Tx.time_seen, both return data produced by the mempool_recorder. This means that they will both return None for all transactions and blocks that have not been recorded. Once you start running the mempool recorder, txes and blocks after that point will have timestamps recorded.

The mempool_recorder is pretty straightforward. It just polls the RPC interface super frequently and watching for update.

[effezeta88]

Thank you @abirke and @hkalodner, actually now I don't have any kind of error, but the installation of blockscipy seems stucked on :

Installing collected packages: blocksci Running setup.py develop for blocksci

I'm waiting for more than 1 hour and nothing happen, no message, no status..nothing. I read that compile takes time but I don't think so long, right?

[Voelundr]

Unfortunaly it took longer than 1 hour for me. I started sudo -H pip3 install -e blockscipy at 2018-05-08 16:16:08 and finished at 2018-05-08 18:56:11 6 GB RAM 4 Cores a 3,4 GHz

You can look at htop If there is a compiler running then the setup is still running and take some time to finish. Maybe @hkalodner can change the setup to show the status. (I saw in htop that the complier ran with quite parameter, maybe change this)

[effezeta88]

Thanks @yiwesi :)

So now I know what waiting for me... Time and patience Thanks again

[hkalodner]

If you add a -v right after pip3 in the command it will compile in verbose mode and you will be able to see more progress. You won't lose much from cancelling and starting again since the build incrementally compiles each file and keeps its progress.

[hkalodner]

I set up the build with the assumption that people would have 16GB or more. If you have less than that you'll probably benefit from changing the -j2 to -j1 in setup.py which controls the number of parallel files building. Currently it's set to build two files at a time instead of 1 which is probably causing your computer to start swapping and slowing everything down.

[zjfheart]

Hi I didn't install BlockSci before, I follow this instruction and get the following error:

jingfeng0@jingfeng-test-01:~/BlockSci/release$ CC=gcc-7 CXX=g++-7 cmake -DCMAKE_BUILD_TYPE=Release .. CMake Error at /usr/share/cmake-3.5/Modules/CMakeDetermineCCompiler.cmake:57 (message): Could not find compiler set in environment variable CC:

gcc-7. Call Stack (most recent call first): CMakeLists.txt:2 (project)

CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage -- Configuring incomplete, errors occurred! See also "/home/jingfeng0/BlockSci/release/CMakeFiles/CMakeOutput.log".

This is my system information: Linux jingfeng-test-01 4.4.0-124-generic #148-Ubuntu SMP Wed May 2 13:00:18 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[zjfheart]

1 Refer to my previous question: I follow this link to use gcc 7.1 and cmake setup. It can solve the previous question.

[zjfheart]

@hkalodner @abirke Hi, there is cmake error in Cmakelist.txt. Is that I use the wrong version of cmake? My cmake version is 3.5.1

jingfeng0@jingfeng-test-01:~/BlockSci$ CC=gcc-7 CXX=g++-7 sudo -H pip3 install -e blockscipy Obtaining file:///home/jingfeng0/BlockSci/blockscipy Requirement already satisfied (use --upgrade to upgrade): multiprocess>=0.70.5 in /usr/local/lib/python3.5/dist-packages (from blocksci===0.5.0) Requirement already satisfied (use --upgrade to upgrade): psutil>=5.4.2 in /usr/local/lib/python3.5/dist-packages (from blocksci===0.5.0) Requirement already satisfied (use --upgrade to upgrade): pycrypto>=2.6.1 in /usr/local/lib/python3.5/dist-packages (from blocksci===0.5.0) Requirement already satisfied (use --upgrade to upgrade): pandas>=0.22.0 in /usr/local/lib/python3.5/dist-packages (from blocksci===0.5.0) Requirement already satisfied (use --upgrade to upgrade): dateparser>=0.6.0 in /usr/local/lib/python3.5/dist-packages (from blocksci===0.5.0) Requirement already satisfied (use --upgrade to upgrade): dill>=0.2.6 in /usr/local/lib/python3.5/dist-packages (from multiprocess>=0.70.5->blocksci===0.5.0) Requirement already satisfied (use --upgrade to upgrade): numpy>=1.9.0 in /usr/local/lib/python3.5/dist-packages (from pandas>=0.22.0->blocksci===0.5.0) Requirement already satisfied (use --upgrade to upgrade): python-dateutil>=2.5.0 in /usr/local/lib/python3.5/dist-packages (from pandas>=0.22.0->blocksci===0.5.0) Requirement already satisfied (use --upgrade to upgrade): pytz>=2011k in /usr/local/lib/python3.5/dist-packages (from pandas>=0.22.0->blocksci===0.5.0) Requirement already satisfied (use --upgrade to upgrade): regex in /usr/local/lib/python3.5/dist-packages (from dateparser>=0.6.0->blocksci===0.5.0) Requirement already satisfied (use --upgrade to upgrade): tzlocal in /usr/local/lib/python3.5/dist-packages (from dateparser>=0.6.0->blocksci===0.5.0) Requirement already satisfied (use --upgrade to upgrade): six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.5.0->pandas>=0.22.0->blocksci===0.5.0) Installing collected packages: blocksci Running setup.py develop for blocksci Complete output from command /usr/bin/python3 -c "import setuptools, tokenize;file='/home/jingfeng0/BlockSci/blockscipy/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" develop --no-deps: running develop Checking .pth file support in /usr/local/lib/python3.5/dist-packages/ /usr/bin/python3 -E -c pass TEST PASSED: /usr/local/lib/python3.5/dist-packages/ appears to support .pth files running egg_info writing dependency_links to blocksci.egg-info/dependency_links.txt writing requirements to blocksci.egg-info/requires.txt writing blocksci.egg-info/PKG-INFO writing top-level names to blocksci.egg-info/top_level.txt warning: manifest_maker: standard file '-c' not found

reading manifest file 'blocksci.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*' under directory 'blocksci/pybind11/'
writing manifest file 'blocksci.egg-info/SOURCES.txt'
running build_ext
-- pybind11 v2.2.2
-- Configuring done
CMake Error in CMakeLists.txt:
  The CXX_STANDARD property on target "_blocksci" contained an invalid value:
  "17".

-- Generating done
-- Build files have been written to: /home/jingfeng0/BlockSci/blockscipy/build/temp.linux-x86_64-3.5
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/jingfeng0/BlockSci/blockscipy/setup.py", line 76, in <module>
    'dateparser>=0.6.0'
  File "/usr/lib/python3.5/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/usr/lib/python3/dist-packages/setuptools/command/develop.py", line 34, in run
    self.install_for_development()
  File "/usr/lib/python3/dist-packages/setuptools/command/develop.py", line 119, in install_for_development
    self.run_command('build_ext')
  File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/home/jingfeng0/BlockSci/blockscipy/setup.py", line 32, in run
    self.build_extension(ext)
  File "/home/jingfeng0/BlockSci/blockscipy/setup.py", line 56, in build_extension
    subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
  File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/home/jingfeng0/BlockSci/blockscipy', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/jingfeng0/BlockSci/blockscipy/blocksci', '-DPYTHON_EXECUTABLE=/usr/bin/python3', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1

----------------------------------------

Command "/usr/bin/python3 -c "import setuptools, tokenize;file='/home/jingfeng0/BlockSci/blockscipy/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" develop --no-deps" failed with error code 1 in /home/jingfeng0/BlockSci/blockscipy/ You are using pip version 8.1.1, however version 10.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. jingfeng0@jingfeng-test-01:~/BlockSci$

[abirke]

To me it also seems the cmake version you are using is too old. According to its documentation, C++17 is only supported since version 3.8.

I successfully compiled with cmake 3.11.0.

[zjfheart]

@abirke thanks for the reply.

I update the cmake to 3.11.1 and use update-alternatives --install .... --master, finally make it stuck for around one hour: Installing collected packages: blocksci Running setup.py develop for blocksci

But another problem appear (c++: internal compiler error: Killed (program cc1plus)).

Full errors are reported as followed: `` [ 95%] Building CXX object CMakeFiles/_blocksci.dir/src/python_interface.cpp.o [ 96%] Building CXX object CMakeFiles/_blocksci.dir/src/range_conversion.cpp.o c++: internal compiler error: Killed (program cc1plus) Please submit a full bug report, with preprocessed source if appropriate. See file:///usr/share/doc/gcc-7/README.Bugs for instructions. CMakeFiles/_blocksci.dir/build.make:816: recipe for target 'CMakeFiles/_blocksci.dir/src/range_conversion.cpp.o' failed make[2]: [CMakeFiles/_blocksci.dir/src/range_conversion.cpp.o] Error 4 make[2]: Waiting for unfinished jobs.... CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/_blocksci.dir/all' failed make[1]: [CMakeFiles/_blocksci.dir/all] Error 2 Makefile:83: recipe for target 'all' failed make: [all] Error 2 Traceback (most recent call last): File "", line 1, in File "/home/jingfeng0/BlockSci/blockscipy/setup.py", line 76, in 'dateparser>=0.6.0' File "/usr/lib/python3.5/distutils/core.py", line 148, in setup dist.run_commands() File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands self.run_command(cmd) File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/usr/lib/python3/dist-packages/setuptools/command/develop.py", line 34, in run self.install_for_development() File "/usr/lib/python3/dist-packages/setuptools/command/develop.py", line 119, in install_for_development self.run_command('build_ext') File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/home/jingfeng0/BlockSci/blockscipy/setup.py", line 32, in run self.build_extension(ext) File "/home/jingfeng0/BlockSci/blockscipy/setup.py", line 57, in build_extension subprocess.check_call(['cmake', '--build', '.'] + build_args, cwd=self.build_temp) File "/usr/lib/python3.5/subprocess.py", line 581, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--config', 'Release', '--', '-j2']' returned non-zero exit status 2

[abirke]

This error message often seems to appear related to memory exceedance. Did you assure to provide enough memory or swap respectively?

[zjfheart]

@abirke I exclusively uset a VM This is my memory and swap. What should I do, if Mem and Swap is not enough? Any way to solve this without upgrading the hardware?

MemTotal: 8173868 kB MemFree: 52680 kB MemAvailable: 3948312 kB Buffers: 9400 kB Cached: 3848444 kB SwapCached: 42476 kB Active: 4784380 kB Inactive: 3092296 kB Active(anon): 2858724 kB Inactive(anon): 1164420 kB Active(file): 1925656 kB Inactive(file): 1927876 kB Unevictable: 3652 kB Mlocked: 3652 kB SwapTotal: 2096444 kB SwapFree: 1486032 kB Dirty: 184740 kB Writeback: 1068 kB AnonPages: 4019256 kB Mapped: 2011392 kB Shmem: 1852 kB Slab: 150220 kB SReclaimable: 120252 kB SUnreclaim: 29968 kB KernelStack: 4464 kB PageTables: 21496 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 6183376 kB Committed_AS: 3814580 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 102400 kB DirectMap2M: 7364608 kB DirectMap1G: 2097152 kB

[abirke]

Maybe try raising the swap to 8GB and watch its usage during the compilation to see how much is needed.

[nelsontodd]

Just wanted to second some of the comments people have made. From a clean Ubuntu 16.04 install I had to, in addition to the instructions: 1) sudo apt-get install g++-7 2) install cmake 3.9 instead of 3.5 (the version installed by apt-get install) 3) run 'sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 60 --slave /usr/bin/g++ g++ /usr/bin/g++-7'

In order to get the python library installed.

[zjfheart]

Hi,

I successfully installed blocksci. (see the following quote. )

Successfully built multiprocess psutil pycrypto dill tzlocal regex Installing collected packages: dill, multiprocess, psutil, pycrypto, numpy, pytz, python-dateutil, pandas, tzlocal, regex, dateparser, blocksci Running setup.py develop for blocksci Successfully installed blocksci dateparser-0.7.0 dill-0.2.7.1 multiprocess-0.70.5 numpy-1.14.3 pandas-0.23.0 psutil-5.4.5 pycrypto-2.6.1 python-dateutil-2.7.3 pytz-2018.4 regex-2018.2.21 tzlocal-1.5.1

I come to share some of my ideas.

This is my instance with Ubuntu system. MemTotal: 132016016 kB SwapTotal: 2096444 kB You need large enough RAM, possibly 64GB RAM.

Then run this command in the terminal: 1 sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y 2 sudo apt-get update 3 sudo apt install libtool autoconf libboost-filesystem-dev libboost-iostreams-dev \ libboost-serialization-dev libboost-thread-dev libboost-test-dev libssl-dev libjsoncpp-dev \ libcurl4-openssl-dev libjsoncpp-dev libjsonrpccpp-dev libsnappy-dev zlib1g-dev libbz2-dev \ liblz4-dev libzstd-dev libjemalloc-dev libsparsehash-dev python3-dev python3-pip 4 sudo apt-get install g++-7 5 install cmake 3.9 or higher instead of 3.5 (the cmake 3.5 version installed by apt-get install) (or follow this link to remove old version of cmake and get the new version.) 6 sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 60 --slave /usr/bin/g++ g++ /usr/bin/g++-7 7 git clone https://github.com/citp/BlockSci.git 8 cd BlockSci 9 mkdir release 10 cd release 11 cmake -DCMAKE_BUILD_TYPE=Release .. 12 make 13 sudo make install 14 cd .. 15 sudo -H pip3 install -e blockscipy (this takes around 2 hours) Then done.

[heojae91]

Hi all, I'm suffering from the error while pip3 step. Whenever I'm trying to install BlockSci on Ubuntu 16.04, but it fails with error message :

...
    /home/jayheo/blockchain/BlockSci/blockscipy/src/range_conversion.hpp:225:8: error: expected ‘(’ before ‘constexpr’
         if constexpr (is_optional<value_type>::value) {
            ^
    /home/jayheo/blockchain/BlockSci/blockscipy/src/cluster/cluster/cluster_range.cpp:17:1: error: expected ‘}’ at end of input
     }
     ^
    /home/jayheo/blockchain/BlockSci/blockscipy/src/cluster/cluster/cluster_range.cpp:17:1: warning: no return statement in function returning non-void [-Wreturn-type]
    In file included from /home/jayheo/blockchain/BlockSci/blockscipy/src/self_apply_py.hpp:12:0,
                     from /home/jayheo/blockchain/BlockSci/blockscipy/src/cluster/cluster/cluster_py.cpp:11:
    /home/jayheo/blockchain/BlockSci/blockscipy/src/range_conversion.hpp: In function ‘constexpr bool isNestedOptionalRange()’:
    /home/jayheo/blockchain/BlockSci/blockscipy/src/range_conversion.hpp:225:8: error: expected ‘(’ before ‘constexpr’
         if constexpr (is_optional<value_type>::value) {
            ^
    /home/jayheo/blockchain/BlockSci/blockscipy/src/cluster/cluster/cluster_py.cpp:86:1: error: expected ‘}’ at end of input
...
     }
     ^

I think this is because my gcc-7 compiler's problem because it seems that compiler cannot recognize the if constexpr context.

My current setting is : Ubuntu 16.04 cmake 3.11.2 gcc-7 (Ubuntu 7.3.0-16ubuntu3 16.04.1) 7.3.0 g++-7 (Ubuntu 7.3.0-16ubuntu3 16.04.1) 7.3.0

What could be an answer of the problem?

[boshmaf]

Update your cmake and make sure you're actually using C++17 (use update-alternatives as specified above).