How to run examples

[ahoora08]

hey,

I did the steps specified for installing kspp library and now I want to run an example filter-example. But I faced error #include <kspp/impl/serdes/text_serdes.h>.

Should I do anything more like linking something in compiling or else?

[skarlsson]

What platform? I know that the windows build might be outdated, since I normally use linux.

[solsson]

@skarlsson Do you have any Dockerfiles lying around that do something like https://github.com/bitbouncer/kspp#ubuntu-16-x64? I clearly felt the need for one while reading :)

I created https://github.com/edenhill/kafkacat/pull/110 which also builds librdkafka, but only thanks to an existing build script.

[ahoora08]

@skarlsson on ubuntu 16.04

[skarlsson]

@solsson , sure I'll dig it up. The build steps came from a Dockerfile

[skarlsson]

@soheil08 I just did a clean build on ubuntu 17. It works just fine. Should be the same on 16.

[ahoora08]

The build is done correctly. But running the example it errors. I think something more should be done.

[skarlsson]

@solsson Just pushed an updated Dockerfile. (it was already there...) cd docker ./build.sh this will give you a 21 MB alpine image. (including examples) use that as a base for you own stuff.

@soheil08 docker images ... replace the hash below with your hash docker run -it 22e459513a93 bash
export KSPP_KAFKA_BROKER_URL=broker_ip:port example6-filter

[skarlsson]

@soheil08 complaints about #include <kspp/impl/serdes/text_serdes.h> seems like a compilation error to me.

when you run it it will try to connect to kafka broker at plaintext://localhost:9092. You can override that by defining env KSPP_KAFKA_BROKER_URL.

[ahoora08]

at the end of the building kscpp it's better the command sudo make install to be added. Doing so, kscpp installed widely on path /usr/local/include.

Now we have another error undefined reference tofLB::FLAGS_logtostderr`

I think something is needed to be link at compile time

[skarlsson]

@soheil08 that should be in glog - CMakelists.txt:164. I guess you already has done sudo apt-get install libgoogle-glog-dev libgflags-dev ? Check if you have more than on copy of libraries in your include or link path

[ahoora08]

yes, package libgoogle-glog-dev libgflags-dev is installed. Should I remove it?

[skarlsson]

No, they should be installed - but I was wondering if you had several version installed or something like that, Since you can compile but not link. I'll make a ubuntu 16 docker build today and then you can compare.

[ahoora08]

Hey, I didn't use docker. Should I?

You know, I just run the commands you provide for installing kspp and its dependencies and using codeblock I want to run filter example. nothing more.

[ahoora08]

Can you say what exactly should I link? Just -lkspp? please.

[ahoora08]

Here is all errors:

/home/soheil/kafka_streaming/main.cpp|22|undefined reference tofLB::FLAGS_logtostderr'| /home/soheil/kafka_streaming/main.cpp|23|undefined reference to google::InitGoogleLogging(char const*)'| obj/Debug/main.o||In functionstatic_initialization_and_destruction_0(int, int)':| /usr/include/boost/system/error_code.hpp|221|undefined reference to boost::system::generic_category()'| /usr/include/boost/system/error_code.hpp|222|undefined reference toboost::system::generic_category()'| /usr/include/boost/system/error_code.hpp|223|undefined reference to `boost::system::system_category()'| obj/Debug/main.o||In functionkspp::kafka_source<void, std::cxx11::basic_string<char, std::char_traits, std::allocator >, kspp::text_serdes>::parse(std::unique_ptr<RdKafka::Message, std::default_delete > const&)':| /usr/local/include/kspp/sources/kafka_source.h|227|undefined reference to google::LogMessage::LogMessage(char const*, int, int)'| /usr/local/include/kspp/sources/kafka_source.h|227|undefined reference togoogle::LogMessage::stream()'| /usr/local/include/kspp/sources/kafka_source.h|227|undefined reference to google::LogMessage::~LogMessage()'| /usr/local/include/kspp/sources/kafka_source.h|231|undefined reference togoogle::LogMessage::LogMessage(char const, int, int)'| /usr/local/include/kspp/sources/kafka_source.h|231|undefined reference to google::LogMessage::stream()'| /usr/local/include/kspp/sources/kafka_source.h|231|undefined reference togoogle::LogMessage::~LogMessage()'| /usr/local/include/kspp/sources/kafka_source.h|227|undefined reference to google::LogMessage::~LogMessage()'| /usr/local/include/kspp/sources/kafka_source.h|231|undefined reference togoogle::LogMessage::~LogMessage()'| /usr/local/lib/libkspp.so||undefined reference to curl_multi_assign'| /usr/local/lib/libkspp.so||undefined reference tocurl_easy_getinfo'| /usr/local/lib/libkspp.so||undefined reference to curl_easy_init'| /usr/local/lib/libkspp.so||undefined reference toboost::filesystem::detail::create_directories(boost::filesystem::path const&, boost::system::error_code)'| /usr/local/lib/libkspp.so||undefined reference to curl_slist_append'| /usr/local/lib/libkspp.so||undefined reference tocurl_multi_setopt'| /usr/local/lib/libkspp.so||undefined reference to curl_multi_add_handle'| /usr/local/lib/libkspp.so||undefined reference toavro::compileJsonSchema(std::istream&, avro::ValidSchema&)'| /usr/local/lib/libkspp.so||undefined reference to curl_multi_init'| /usr/local/lib/libkspp.so||undefined reference tocurl_easy_cleanup'| /usr/local/lib/libkspp.so||undefined reference to curl_multi_cleanup'| /usr/local/lib/libkspp.so||undefined reference toRdKafka::Conf::create(RdKafka::Conf::ConfType)'| /usr/local/lib/libkspp.so||undefined reference to curl_multi_remove_handle'| /usr/local/lib/libkspp.so||undefined reference tocurl_slist_free_all'| /usr/local/lib/libkspp.so||undefined reference to `RdKafka::KafkaConsumer::create(RdKafka::Conf*, std::cxx11::basic_string<char, std::char_traits, std::allocator >&)'| /usr/local/lib/libkspp.so||undefined reference to RdKafka::err2str[abi:cxx11](RdKafka::ErrorCode)'| /usr/local/lib/libkspp.so||undefined reference toboost::filesystem::detail::temp_directory_path(boost::system::error_code)'| /usr/local/lib/libkspp.so||undefined reference to `RdKafka::Producer::create(RdKafka::Conf, std::cxx11::basic_string<char, std::char_traits, std::allocator >&)'| /usr/local/lib/libkspp.so||undefined reference to curl_multi_info_read'| /usr/local/lib/libkspp.so||undefined reference tocurl_multi_socket_action'| /usr/local/lib/libkspp.so||undefined reference to curl_easy_setopt'| /usr/local/lib/libkspp.so||undefined reference toboost::filesystem::detail::status(boost::filesystem::path const&, boost::system::error_code)'| /usr/local/lib/libkspp.so||undefined reference to avro::ValidSchema::ValidSchema()'| /usr/local/lib/libkspp.so||undefined reference togoogle::LogMessage::LogMessage(char const, int)'| /usr/local/lib/libkspp.so||undefined reference to avro::ValidSchema::toJson(std::ostream&) const'| /usr/local/lib/libkspp.so||undefined reference togoogle::LogMessageFatal::~LogMessageFatal()'| /usr/local/lib/libkspp.so||undefined reference to boost::filesystem::path::operator/=(boost::filesystem::path const&)'| /usr/local/lib/libkspp.so||undefined reference togoogle::LogMessageFatal::LogMessageFatal(char const, int)'| /usr/local/lib/libkspp.so||undefined reference to RdKafka::TopicPartition::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)'| /usr/local/lib/libkspp.so||undefined reference toRdKafka::Topic::create(RdKafka::Handle, `

[skarlsson]

Aha, that explains - first - the examples are already built if you choose the ./rebuild-all-options.sh version. If you want to set this up using codeblocks (that I dont use) then you need to provide both the defines and libs that's in the CMakeLists.txt. Some of the settings are auto discovered.

You can find the link commands that are used on your system. cd build grep -r kspp_s *

will give me lots of stuff but for example:

./../lib/libkspp_s.a /usr/local/lib/libavrocpp.so -lcurl /usr/local/lib/librocksdb.so -lsnappy -lbz2 -lz -lrdkafka++ -lrdkafka -lboost_program_options -lboost_filesystem -lboost_system -lpthread -lrt -lc -lglog -lcrypto -lssl

I also suspect you should add defines to codeblocks, those are from CMakeLists.txt

add_definitions(-g -D_FILE_OFFSET_BITS=64 -D_REENTRANT -DEXTERNAL_LOCKS -DMULTITHREAD) add_definitions(-fPIC) add_definitions(-Wno-deprecated)

[skarlsson]

The part that says ../lib/libkspp_s.a /usr/local/lib/libavrocpp.so -lcurl /usr/local/lib/librocksdb.so should probably be replaced by -lkspp_s -lavrocpp -lrocksdb

[ahoora08]

Ok, suppose I want to compile it using terminal using something like g++ examole-filter.cpp -o .... can you say what compile option should I use.

[skarlsson]

cd kspp ./rebuild-all-options.sh

or

if you want to use an IDE use CLion since it understand CMake files

[ahoora08]

Why rebuilding ?

[ahoora08]

after executing ./rebuild-all-options.sh , into exmple-filter directory I commanded:

1. cmake .
2. make and here are some errors:

/usr/local/include/kspp/sinks/kafka_sink.h:124:45: error: expected ‘,’ or ‘...’ before ‘<’ token virtual int handle_event(std::shared_ptr<kevent<K, V>>) = 0; ^ /usr/local/include/kspp/sinks/kafka_sink.h:126:17: error: ‘shared_ptr’ in namespace ‘std’ does not name a template type inline std::shared_ptr<CODEC> codec() { ^ /usr/local/include/kspp/sinks/kafka_sink.h:130:10: error: ‘shared_ptr’ in namespace ‘std’ does not name a template type std::shared_ptr<CODEC> _codec; ^ /usr/local/include/kspp/sinks/kafka_sink.h:132:5: error: ‘partitioner’ does not name a type partitioner _partitioner; ^ /usr/local/include/kspp/sinks/kafka_sink.h: In member function ‘void kspp::kafka_sink_base<K, V, CODEC>::flush()’: /usr/local/include/kspp/sinks/kafka_sink.h:63:14: error: ‘ec’ does not name a type auto ec = _impl.flush(1000);

[skarlsson]

Thats not how you use CMake typical use is:

mkdir build cd build cmake .. make

the .. in cmake command must refer to your top level CMakeLists.txt dir in this specific case you probably want to enable at least the -DBUILD_SAMPLES=ON (look in "rebuild-all-options.sh" )

after you have run "rebuild-all-options.sh" the example binaries can be found in the kspp/bin

when running the examples they need to be able to find a cluster. If you define an env variable it will solve that problem

export KSPP_KAFKA_BROKER_URL=plaintext://localhost:9092

change the URL to something relevant

[ahoora08]

Thanks alot, Yes, they are in bin directory. You know I want to write my custom stream processing app. I can you tell me using what terminal command I can build my code correctly. (forget codeblocks).

For example, when I use cppkafka package in its instruction was specified that for building codes we should link both cppkafka and rdkafka in compile command. Is any similar rule for building codes here?

[skarlsson]

Easiest way is to just modify one of the examples and do a rebuild