search

confluentinc / bottledwater-pg

Change data capture from PostgreSQL into Kafka
http://blog.confluent.io/2015/04/23/bottled-water-real-time-integration-of-postgresql-and-kafka/
Apache License 2.0
2 stars 0 forks source link

avro_schema_union_append error #119

Open ImTheDeveloper opened 7 years ago

ImTheDeveloper commented 7 years ago

ERROR: could not load library "/usr/lib/postgresql/9.5/lib/bottledwater.so": /usr/lib/postgresql/9.5/lib/bottledwater.so: undefined symbol: avro_schema_union_append

Seem to be hitting this error when I run create extension bottledwater;

Any idea on how to fix this issue?

ImTheDeveloper commented 7 years ago

For information I've included what I run to install avro:

munkee@DE-3628-SDN:~$ git clone https://github.com/apache/avro.git
Cloning into 'avro'...
remote: Counting objects: 28333, done.
remote: Compressing objects: 100% (34/34), done.
remote: Total 28333 (delta 8), reused 0 (delta 0), pack-reused 28291
Receiving objects: 100% (28333/28333), 10.57 MiB | 2.07 MiB/s, done.
Resolving deltas: 100% (14481/14481), done.
Checking connectivity... done.
munkee@DE-3628-SDN:~$ cd avro/lang/c
munkee@DE-3628-SDN:~/avro/lang/c$ mkdir build
munkee@DE-3628-SDN:~/avro/lang/c$ cd build/
munkee@DE-3628-SDN:~/avro/lang/c/build$ cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local -DCMAKE_BUILD_TYPE=RelWithDebInfo
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found ZLIB: /usr/lib/i386-linux-gnu/libz.so (found version "1.2.8")
Enabled deflate codec
-- Found Snappy: /usr/lib/i386-linux-gnu/libsnappy.so
Enabled snappy codec
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1")
-- Checking for module 'liblzma'
--   No package 'liblzma' found
Disabled lzma codec. liblzma not found.
CMake Warning (dev) at CMakeLists.txt:169 (set):
  Policy CMP0053 is not set: Simplify variable reference and escape sequence
  evaluation.  Run "cmake --help-policy CMP0053" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  For input:

    '@ZLIB_PKG@ @LZMA_PKG@ @SNAPPY_PKG@'

  the old evaluation rules produce:

    'zlib  libsnappy'

  but the new evaluation rules produce:

    '@ZLIB_PKG@ @LZMA_PKG@ @SNAPPY_PKG@'

  Using the old result for compatibility since the policy is not set.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Checking for module 'jansson>=2.3'
--   Found jansson, version 2.7
-- Searching for asciidoc...
CMake Warning at docs/CMakeLists.txt:51 (message):
  asciidoc not found.  HTML documentation will *NOT* be built.

-- Configuring done
-- Generating done
-- Build files have been written to: /home/munkee/avro/lang/c/build
munkee@DE-3628-SDN:~/avro/lang/c/build$ make
Scanning dependencies of target avro-static
[  0%] Building C object src/CMakeFiles/avro-static.dir/allocation.o
[  1%] Building C object src/CMakeFiles/avro-static.dir/array.o
[  2%] Building C object src/CMakeFiles/avro-static.dir/codec.o
[  3%] Building C object src/CMakeFiles/avro-static.dir/consumer.o
[  4%] Building C object src/CMakeFiles/avro-static.dir/consume-binary.o
[  4%] Building C object src/CMakeFiles/avro-static.dir/datafile.o
[  5%] Building C object src/CMakeFiles/avro-static.dir/datum.o
[  6%] Building C object src/CMakeFiles/avro-static.dir/datum_equal.o
[  7%] Building C object src/CMakeFiles/avro-static.dir/datum_read.o
[  7%] Building C object src/CMakeFiles/avro-static.dir/datum_size.o
[  8%] Building C object src/CMakeFiles/avro-static.dir/datum_skip.o
[  9%] Building C object src/CMakeFiles/avro-static.dir/datum_validate.o
[ 10%] Building C object src/CMakeFiles/avro-static.dir/datum_value.o
/home/munkee/avro/lang/c/src/datum_value.c: In function ‘avro_datum_value_get_type’:
/home/munkee/avro/lang/c/src/datum_value.c:86:2: warning: #warning "Bug: EINVAL is not of type avro_type_t." [-Wcpp]
 #warning "Bug: EINVAL is not of type avro_type_t."
  ^
[ 11%] Building C object src/CMakeFiles/avro-static.dir/datum_write.o
[ 11%] Building C object src/CMakeFiles/avro-static.dir/dump.o
[ 12%] Building C object src/CMakeFiles/avro-static.dir/encoding_binary.o
[ 13%] Building C object src/CMakeFiles/avro-static.dir/errors.o
[ 14%] Building C object src/CMakeFiles/avro-static.dir/generic.o
[ 14%] Building C object src/CMakeFiles/avro-static.dir/io.o
[ 15%] Building C object src/CMakeFiles/avro-static.dir/map.o
[ 16%] Building C object src/CMakeFiles/avro-static.dir/memoize.o
[ 17%] Building C object src/CMakeFiles/avro-static.dir/resolved-reader.o
[ 18%] Building C object src/CMakeFiles/avro-static.dir/resolved-writer.o
[ 18%] Building C object src/CMakeFiles/avro-static.dir/resolver.o
[ 19%] Building C object src/CMakeFiles/avro-static.dir/schema.o
/home/munkee/avro/lang/c/src/schema.c: In function ‘avro_schema_from_json_t’:
/home/munkee/avro/lang/c/src/schema.c:854:3: warning: #warning "Bug: '0' is not of type avro_type_t." [-Wcpp]
  #warning "Bug: '0' is not of type avro_type_t."
   ^
[ 20%] Building C object src/CMakeFiles/avro-static.dir/schema_equal.o
[ 21%] Building C object src/CMakeFiles/avro-static.dir/st.o
[ 22%] Building C object src/CMakeFiles/avro-static.dir/string.o
[ 22%] Building C object src/CMakeFiles/avro-static.dir/value.o
[ 23%] Building C object src/CMakeFiles/avro-static.dir/value-hash.o
[ 24%] Building C object src/CMakeFiles/avro-static.dir/value-json.o
[ 25%] Building C object src/CMakeFiles/avro-static.dir/value-read.o
[ 25%] Building C object src/CMakeFiles/avro-static.dir/value-sizeof.o
[ 26%] Building C object src/CMakeFiles/avro-static.dir/value-write.o
[ 27%] Building C object src/CMakeFiles/avro-static.dir/wrapped-buffer.o
[ 28%] Linking C static library libavro.a
[ 28%] Built target avro-static
Scanning dependencies of target avroappend
[ 29%] Building C object src/CMakeFiles/avroappend.dir/avroappend.o
[ 29%] Linking C executable avroappend
[ 29%] Built target avroappend
Scanning dependencies of target avrocat
[ 30%] Building C object src/CMakeFiles/avrocat.dir/avrocat.o
[ 31%] Linking C executable avrocat
[ 31%] Built target avrocat
Scanning dependencies of target avro-shared
[ 31%] Building C object src/CMakeFiles/avro-shared.dir/allocation.o
[ 32%] Building C object src/CMakeFiles/avro-shared.dir/array.o
[ 33%] Building C object src/CMakeFiles/avro-shared.dir/codec.o
[ 34%] Building C object src/CMakeFiles/avro-shared.dir/consumer.o
[ 34%] Building C object src/CMakeFiles/avro-shared.dir/consume-binary.o
[ 35%] Building C object src/CMakeFiles/avro-shared.dir/datafile.o
[ 36%] Building C object src/CMakeFiles/avro-shared.dir/datum.o
[ 37%] Building C object src/CMakeFiles/avro-shared.dir/datum_equal.o
[ 38%] Building C object src/CMakeFiles/avro-shared.dir/datum_read.o
[ 38%] Building C object src/CMakeFiles/avro-shared.dir/datum_size.o
[ 39%] Building C object src/CMakeFiles/avro-shared.dir/datum_skip.o
[ 40%] Building C object src/CMakeFiles/avro-shared.dir/datum_validate.o
[ 41%] Building C object src/CMakeFiles/avro-shared.dir/datum_value.o
/home/munkee/avro/lang/c/src/datum_value.c: In function ‘avro_datum_value_get_type’:
/home/munkee/avro/lang/c/src/datum_value.c:86:2: warning: #warning "Bug: EINVAL is not of type avro_type_t." [-Wcpp]
 #warning "Bug: EINVAL is not of type avro_type_t."
  ^
[ 41%] Building C object src/CMakeFiles/avro-shared.dir/datum_write.o
[ 42%] Building C object src/CMakeFiles/avro-shared.dir/dump.o
[ 43%] Building C object src/CMakeFiles/avro-shared.dir/encoding_binary.o
[ 44%] Building C object src/CMakeFiles/avro-shared.dir/errors.o
[ 45%] Building C object src/CMakeFiles/avro-shared.dir/generic.o
[ 45%] Building C object src/CMakeFiles/avro-shared.dir/io.o
[ 46%] Building C object src/CMakeFiles/avro-shared.dir/map.o
[ 47%] Building C object src/CMakeFiles/avro-shared.dir/memoize.o
[ 48%] Building C object src/CMakeFiles/avro-shared.dir/resolved-reader.o
[ 48%] Building C object src/CMakeFiles/avro-shared.dir/resolved-writer.o
[ 49%] Building C object src/CMakeFiles/avro-shared.dir/resolver.o
[ 50%] Building C object src/CMakeFiles/avro-shared.dir/schema.o
/home/munkee/avro/lang/c/src/schema.c: In function ‘avro_schema_from_json_t’:
/home/munkee/avro/lang/c/src/schema.c:854:3: warning: #warning "Bug: '0' is not of type avro_type_t." [-Wcpp]
  #warning "Bug: '0' is not of type avro_type_t."
   ^
[ 51%] Building C object src/CMakeFiles/avro-shared.dir/schema_equal.o
[ 52%] Building C object src/CMakeFiles/avro-shared.dir/st.o
[ 52%] Building C object src/CMakeFiles/avro-shared.dir/string.o
[ 53%] Building C object src/CMakeFiles/avro-shared.dir/value.o
[ 54%] Building C object src/CMakeFiles/avro-shared.dir/value-hash.o
[ 55%] Building C object src/CMakeFiles/avro-shared.dir/value-json.o
[ 56%] Building C object src/CMakeFiles/avro-shared.dir/value-read.o
[ 56%] Building C object src/CMakeFiles/avro-shared.dir/value-sizeof.o
[ 57%] Building C object src/CMakeFiles/avro-shared.dir/value-write.o
[ 58%] Building C object src/CMakeFiles/avro-shared.dir/wrapped-buffer.o
[ 59%] Linking C shared library libavro.so
[ 59%] Built target avro-shared
Scanning dependencies of target avropipe
[ 60%] Building C object src/CMakeFiles/avropipe.dir/avropipe.o
[ 61%] Linking C executable avropipe
[ 61%] Built target avropipe
Scanning dependencies of target avromod
[ 62%] Building C object src/CMakeFiles/avromod.dir/avromod.o
[ 62%] Linking C executable avromod
[ 62%] Built target avromod
Scanning dependencies of target quickstop
[ 63%] Building C object examples/CMakeFiles/quickstop.dir/quickstop.o
[ 64%] Linking C executable quickstop
[ 64%] Built target quickstop
Scanning dependencies of target test_avro_1691
[ 65%] Building C object tests/CMakeFiles/test_avro_1691.dir/test_avro_1691.o
[ 66%] Linking C executable test_avro_1691
[ 66%] Built target test_avro_1691
Scanning dependencies of target test_avro_1034
[ 66%] Building C object tests/CMakeFiles/test_avro_1034.dir/test_avro_1034.o
[ 67%] Linking C executable test_avro_1034
[ 67%] Built target test_avro_1034
Scanning dependencies of target test_interop_data
[ 68%] Building C object tests/CMakeFiles/test_interop_data.dir/test_interop_data.o
[ 69%] Linking C executable test_interop_data
[ 69%] Built target test_interop_data
Scanning dependencies of target test_avro_984
[ 70%] Building C object tests/CMakeFiles/test_avro_984.dir/test_avro_984.o
[ 71%] Linking C executable test_avro_984
[ 71%] Built target test_avro_984
Scanning dependencies of target test_avro_schema_names
[ 72%] Building C object tests/CMakeFiles/test_avro_schema_names.dir/test_avro_schema_names.o
[ 73%] Linking C executable test_avro_schema_names
[ 73%] Built target test_avro_schema_names
Scanning dependencies of target test_avro_968
[ 74%] Building C object tests/CMakeFiles/test_avro_968.dir/test_avro_968.o
[ 74%] Linking C executable test_avro_968
[ 74%] Built target test_avro_968
Scanning dependencies of target test_avro_1165
[ 75%] Building C object tests/CMakeFiles/test_avro_1165.dir/test_avro_1165.o
[ 76%] Linking C executable test_avro_1165
[ 76%] Built target test_avro_1165
Scanning dependencies of target test_avro_1379
[ 76%] Building C object tests/CMakeFiles/test_avro_1379.dir/test_avro_1379.o
[ 77%] Linking C executable test_avro_1379
[ 77%] Built target test_avro_1379
Scanning dependencies of target generate_interop_data
[ 78%] Building C object tests/CMakeFiles/generate_interop_data.dir/generate_interop_data.o
[ 79%] Linking C executable generate_interop_data
[ 79%] Built target generate_interop_data
Scanning dependencies of target performance
[ 79%] Building C object tests/CMakeFiles/performance.dir/performance.o
In file included from /home/munkee/avro/lang/c/src/avro/legacy.h:33:0,
                 from /home/munkee/avro/lang/c/src/avro/io.h:31,
                 from /home/munkee/avro/lang/c/src/avro/consumer.h:30,
                 from /home/munkee/avro/lang/c/src/avro.h:28,
                 from /home/munkee/avro/lang/c/tests/performance.c:23:
/home/munkee/avro/lang/c/tests/performance.c: In function ‘populate_complex_record’:
/home/munkee/avro/lang/c/src/avro/value.h:407:13: warning: ‘field.iface’ may be used uninitialized in this function [-Wmaybe-uninitialized]
     ((value)->iface->method == NULL? (dflt): \
             ^
/home/munkee/avro/lang/c/tests/performance.c:535:16: note: ‘field’ was declared here
  avro_value_t  field;
                ^
[ 80%] Linking C executable performance
[ 80%] Built target performance
Scanning dependencies of target test_data_structures
[ 81%] Building C object tests/CMakeFiles/test_data_structures.dir/test_data_structures.o
[ 81%] Linking C executable test_data_structures
[ 81%] Built target test_data_structures
Scanning dependencies of target test_avro_1238
[ 81%] Building C object tests/CMakeFiles/test_avro_1238.dir/test_avro_1238.o
[ 82%] Linking C executable test_avro_1238
[ 82%] Built target test_avro_1238
Scanning dependencies of target test_avro_values
[ 82%] Building C object tests/CMakeFiles/test_avro_values.dir/test_avro_values.o
[ 83%] Linking C executable test_avro_values
[ 83%] Built target test_avro_values
Scanning dependencies of target test_avro_1087
[ 84%] Building C object tests/CMakeFiles/test_avro_1087.dir/test_avro_1087.o
[ 84%] Linking C executable test_avro_1087
[ 84%] Built target test_avro_1087
Scanning dependencies of target test_avro_1237
[ 85%] Building C object tests/CMakeFiles/test_avro_1237.dir/test_avro_1237.o
[ 86%] Linking C executable test_avro_1237
[ 86%] Built target test_avro_1237
Scanning dependencies of target test_avro_1279
[ 87%] Building C object tests/CMakeFiles/test_avro_1279.dir/test_avro_1279.o
[ 88%] Linking C executable test_avro_1279
[ 88%] Built target test_avro_1279
Scanning dependencies of target test_avro_1084
[ 89%] Building C object tests/CMakeFiles/test_avro_1084.dir/test_avro_1084.o
[ 90%] Linking C executable test_avro_1084
[ 90%] Built target test_avro_1084
Scanning dependencies of target test_avro_1405
[ 91%] Building C object tests/CMakeFiles/test_avro_1405.dir/test_avro_1405.o
[ 92%] Linking C executable test_avro_1405
[ 92%] Built target test_avro_1405
Scanning dependencies of target test_avro_data
[ 93%] Building C object tests/CMakeFiles/test_avro_data.dir/test_avro_data.o
[ 94%] Linking C executable test_avro_data
[ 94%] Built target test_avro_data
Scanning dependencies of target test_avro_1572
[ 95%] Building C object tests/CMakeFiles/test_avro_1572.dir/test_avro_1572.o
[ 95%] Linking C executable test_avro_1572
[ 95%] Built target test_avro_1572
Scanning dependencies of target test_refcount
[ 96%] Building C object tests/CMakeFiles/test_refcount.dir/test_refcount.o
[ 97%] Linking C executable test_refcount
[ 97%] Built target test_refcount
Scanning dependencies of target test_avro_schema
[ 97%] Building C object tests/CMakeFiles/test_avro_schema.dir/test_avro_schema.o
[ 98%] Linking C executable test_avro_schema
[ 98%] Built target test_avro_schema
Scanning dependencies of target test_cpp
[ 99%] Building CXX object tests/CMakeFiles/test_cpp.dir/test_cpp.o
[100%] Linking CXX executable test_cpp
[100%] Built target test_cpp
munkee@DE-3628-SDN:~/avro/lang/c/build$ sudo make install
[ 28%] Built target avro-static
[ 29%] Built target avroappend
[ 31%] Built target avrocat
[ 59%] Built target avro-shared
[ 61%] Built target avropipe
[ 62%] Built target avromod
[ 64%] Built target quickstop
[ 66%] Built target test_avro_1691
[ 67%] Built target test_avro_1034
[ 69%] Built target test_interop_data
[ 71%] Built target test_avro_984
[ 73%] Built target test_avro_schema_names
[ 74%] Built target test_avro_968
[ 76%] Built target test_avro_1165
[ 77%] Built target test_avro_1379
[ 79%] Built target generate_interop_data
[ 80%] Built target performance
[ 81%] Built target test_data_structures
[ 82%] Built target test_avro_1238
[ 83%] Built target test_avro_values
[ 84%] Built target test_avro_1087
[ 86%] Built target test_avro_1237
[ 88%] Built target test_avro_1279
[ 90%] Built target test_avro_1084
[ 92%] Built target test_avro_1405
[ 94%] Built target test_avro_data
[ 95%] Built target test_avro_1572
[ 97%] Built target test_refcount
[ 98%] Built target test_avro_schema
[100%] Built target test_cpp
Install the project...
-- Install configuration: "RelWithDebInfo"
-- Installing: /usr/local/include/avro.h
-- Installing: /usr/local/include/avro
-- Installing: /usr/local/include/avro/basics.h
-- Installing: /usr/local/include/avro/generic.h
-- Installing: /usr/local/include/avro/refcount.h
-- Installing: /usr/local/include/avro/msinttypes.h
-- Installing: /usr/local/include/avro/allocation.h
-- Installing: /usr/local/include/avro/errors.h
-- Installing: /usr/local/include/avro/data.h
-- Installing: /usr/local/include/avro/value.h
-- Installing: /usr/local/include/avro/resolver.h
-- Installing: /usr/local/include/avro/schema.h
-- Installing: /usr/local/include/avro/io.h
-- Installing: /usr/local/include/avro/msstdint.h
-- Installing: /usr/local/include/avro/legacy.h
-- Installing: /usr/local/include/avro/consumer.h
-- Installing: /usr/local/include/avro/platform.h
-- Installing: /usr/local/lib/libavro.a
-- Installing: /usr/local/lib/libavro.so.23.0.0
-- Up-to-date: /usr/local/lib/libavro.so
-- Installing: /usr/local/lib/pkgconfig/avro-c.pc
-- Installing: /usr/local/bin/avrocat
-- Installing: /usr/local/bin/avroappend
-- Installing: /usr/local/bin/avropipe
-- Installing: /usr/local/bin/avromod
munkee@DE-3628-SDN:~/avro/lang/c/build$ psql -h localhost -U ***
Password for user ***:
psql (9.5.4)
Type "help" for help.

***=# create extension bottledwater;
ERROR:  could not load library "/usr/lib/postgresql/9.5/lib/bottledwater.so": /usr/lib/postgresql/9.5/lib/bottledwater.so: undefined symbol: avro_schema_union_append
badboyd commented 7 years ago

Could you try sudo ldconfig ?

ImTheDeveloper commented 7 years ago

Strangely enough I tried with this fork and it works fine now:

https://github.com/wrmsr/bottledwater-pg

Unsure what the differences are it just looked like the maintainer had removed/updated a few dependencies

badboyd commented 7 years ago

Uhm, I've not ever run into this problem. Could you please build the main branch again then run sudo ldconfig so we can know if it can fix the problem ? :)

JamalSale commented 7 years ago

I have the same issue, but fork https://github.com/wrmsr/bottledwater-pg does not help

JamalSale commented 7 years ago

Problem solved by getting repo via git clone.. Looks like there is some bug in zip archive...