Unable to build

[vishnukl-alation]

Hi,

I am trying to build rocksplicator for the first time and am having trouble doing so. I am not a docker pro and was just running the commands mentioned in the README. I pulled the docker image cdonaghy/rocksplicator-build and cloned the repo to ~/code/rocksplicator

Can you tell me if what should be present in $HOME/docker-root? "Get into the docker build environment. We are assuming the rocksplicator repo is under $HOME/code/, and $HOME/docker-root is an existing directory."

docker run -v <SOURCE-DIR>:/rocksplicator -v $HOME/docker-root:/root -ti rocksplicator-build bash

What should be the value of <SOURCE-DIR>?

I just ran with

docker run -v ~/code/rocksplicator -v $HOME/docker-root:/root -ti rocksplicator-build bash

but it says

Unable to find image 'rocksplicator-build:latest' locally docker: Error response from daemon: repository rocksplicator-build not found: does not exist or no pull access.

$ docker images
REPOSITORY                     TAG                 IMAGE ID            CREATED             SIZE
cdonaghy/rocksplicator-build   latest              fa0cbfa908de        7 weeks ago         2.48 GB

Thanks

[shuzhang1989]

<SOURCE-DIR> is the path you git clone the rocksplicator code this is the example command i use to run docker: docker run --name rocksplicator -v /home/shu/code/rocksplicator:/rocksplicator -v /home/shu/code/docker_root:/root -w /rocksplicator -ti cdonaghy/rocksplicator-build:latest bash

[angxu]

hi @vishnukl-alation

Looks like the command in README is a bit off. The following one should do it right. docker run -v $HOME/code/rocksplicator:/rocksplicator -v $HOME/code/docker-root:/root -ti cdonaghy/rocksplicator-build:latest bash

[vishnukl-alation]

Thanks all, it works now. But during make -j. I'm getting this error c++: internal compiler error: Killed (program cc1plus) (cmake seems to work fine)

[ 22%] Built target rocksdb_glogger
c++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[2]: *** [common/stats/CMakeFiles/stats.dir/status_server.cpp.o] Error 4
make[2]: *** Waiting for unfinished jobs....
c++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[2]: *** [common/tests/thrift/CMakeFiles/dummy_service_thrift.dir/gen-cpp2/dummy_service_types.cpp.o] Error 4
make[2]: *** Waiting for unfinished jobs....
c++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[2]: *** [rocksdb_replicator/thrift/CMakeFiles/replicator_thrift.dir/gen-cpp2/replicator_types.cpp.o] Error 4
make[2]: *** Waiting for unfinished jobs....
c++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[2]: *** [tgrep/CMakeFiles/tgrep.dir/tcp_connection.cpp.o] Error 4
make[2]: *** Waiting for unfinished jobs....
c++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[2]: *** [tgrep/CMakeFiles/tgrep.dir/tcp_flow.cpp.o] Error 4
c++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[2]: *** [rocksdb_replicator/thrift/CMakeFiles/replicator_thrift.dir/gen-cpp2/Replicator_client.cpp.o] Error 4
c++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[2]: *** [common/tests/thrift/CMakeFiles/dummy_service_thrift.dir/gen-cpp2/DummyService.cpp.o] Error 4
make[1]: *** [tgrep/CMakeFiles/tgrep.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 23%] Linking CXX static library libjsoncpp.a
[ 23%] Built target jsoncpp_lib_static
make[1]: *** [common/stats/CMakeFiles/stats.dir/all] Error 2
make[1]: *** [rocksdb_replicator/thrift/CMakeFiles/replicator_thrift.dir/all] Error 2
make[1]: *** [common/tests/thrift/CMakeFiles/dummy_service_thrift.dir/all] Error 2
[ 24%] Linking CXX static library libgmock_main.a
[ 24%] Built target gmock_main
[ 25%] Linking CXX static library libgmock.a
[ 26%] Linking CXX static library libgtest.a
[ 26%] Built target gmock
[ 26%] Built target gtest
make: *** [all] Error 2

[shuzhang1989]

can you retry by giving it less threads, like make -j2 or only make

[vishnukl-alation]

using just make seems to work. Also when I run python rocksdb_admin.py cluster_name ping it just logs No handlers could be found for logger "kazoo.client" and doesn't seem to be doing anything. Can't kill the process also by ctrl+C. Could you also let me know how I can use rocksreplicator outside of the docker container? I have 2 rocksdb instances ouside of docker and would like rocksreplicator ideally work by just using the built libraries and binaries outside of docker.

[shuzhang1989]

ping() doesn't return anything, so if you see nothing, it means the ping is good.

for the docker question, one way is to ldd the binary you built inside docker to copy them to outside. another is to install all required libraries we listed in DockerFile in your own environment. the purpose of the docker is to give people a convenient way to try rocksplicator out.

[vishnukl-alation]

I'm still not sure how exactly to setup a working system. Let's say I have two rocksdb stores one at host1 (/home/kvstore) and another store at host2 (/home/kvstore). I have a custom go code wrapper around the kvstore c library that servers as a key value server. How would the admin know where the stores are? Do I need to change the way my kvserver interacts with rocksdb store? What are the things need to be running on host1 and host2 for the replication to happen? Can rocksreplicator just take two rocksdb store paths and keep both in sync without having to change the way data is inserted?

[shuzhang1989]

@vishnukl-alation you can refer to our counter service example. https://github.com/pinterest/rocksplicator/blob/master/examples/counter_service/counter.cpp

every db runs a server process, and it will need to read the config from the configfile to know which role the local db is