percona56-cluster broken on 2016Q4 base-64-lts (16.4.1) and 2017Q1

[siepkes]

Percona XtraDB Cluster 5.6 does not work correctly on 2016Q4 with the Base-64 LTS image. There seems to be a discrepancy between the MySQL version used and the Galera version used. The MySQL instance will start but replication will not work.

I also tried the more recent base-64 17.1.0 which uses pkgsrc 2017Q1 and the problem doesn't occur there.

2017-06-04 16:21:02 77017 [Note] WSREP: Read nil XID from storage engines, skipping position init
2017-06-04 16:21:02 77017 [Note] WSREP: wsrep_load(): loading provider library '/opt/local/lib/mysql/plugin/libgalera_smm.so'
2017-06-04 16:21:02 77017 [ERROR] WSREP: provider interface version mismatch: need '26', found '25'
2017-06-04 16:21:02 77017 [ERROR] WSREP: wsrep_load(/opt/local/lib/mysql/plugin/libgalera_smm.so) failed: Invalid argument (22). Reverting to no provider.
2017-06-04 16:21:02 77017 [Note] WSREP: Read nil XID from storage engines, skipping position init
2017-06-04 16:21:02 77017 [Note] WSREP: wsrep_load(): loading provider library 'none'
2017-06-04 16:21:02 77017 [Note] Plugin 'FEDERATED' is disabled.

[siepkes]

Should note that altough on 2017Q1 it can load the plugin correctly the bootstrap still fails with:

2017-06-04 23:46:52 69571 [Note] WSREP: protonet asio version 0
2017-06-04 23:46:52 69571 [Note] WSREP: Using CRC-32C for message checksums.
2017-06-04 23:46:52 69571 [Note] WSREP: backend: asio
2017-06-04 23:46:52 69571 [ERROR] WSREP: failed to open gcomm backend connection: 1: Failed to set thread schedparams other:0: 1 (Not owner)
         at galerautils/src/gu_thread.cpp:thread_set_schedparam():87
2017-06-04 23:46:52 69571 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -1 (Not owner)
2017-06-04 23:46:52 69571 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1380: Failed to open channel 'misc' at 'gcomm://': -1 (Not owner)
2017-06-04 23:46:52 69571 [ERROR] WSREP: gcs connect failed: Not owner
2017-06-04 23:46:52 69571 [ERROR] WSREP: wsrep::connect(gcomm://) failed: 7
2017-06-04 23:46:52 69571 [ERROR] Aborting

[mamash]

Which Percona package(s) exactly do you have installed from 2016Q4?

[siepkes]

@mamash This is the full list of all the percona I have installed in the 16.4.1 pkgsrc 2016Q4 image:

pkg_info  | grep percona
p5-DBD-percona56-cluster-4.041 Perl DBI/DBD driver for MySQL databases
percona-cluster-5.6.35 Percona Cluster Server 5.6
percona56-cluster-galera-25.3.19 Synchronous Multi-Master replication plugin for MySQL
percona56-cluster-xtrabackup-2.4.7 Percona XtraBackup
percona56-cluster-1.2 Meta-package for a Percona environment
quickbackup-percona-3.6 Percona quickbackup SMF service

For reference here is the entire list:

sudo-1.8.17p1       Allow others to run commands as root
libiconv-1.14nb2    Character set conversion library
py27-setuptools-32.1.0 New Python packaging system
p5-DBD-percona56-cluster-4.041 Perl DBI/DBD driver for MySQL databases
boost-libs-1.62.0nb1 Free, peer-reviewed portable C++ source libraries (binary libraries)
nodejs-7.2.1        V8 JavaScript for clients and servers
libgpg-error-1.25   Definitions of common error values for all GnuPG components
smtools-20160926    Joyent tools relevant to SmartOS and SmartMachines
percona-cluster-5.6.35 Percona Cluster Server 5.6
libgcrypt-1.7.4     GNU cryptographic library
zeromq-4.1.4nb1     The ZeroMQ messaging library
check-0.10.0        Unit test framework for C
py27-singledispatch-3.4.0.3 Library brings functools.singledispatch from Python 3.4
libfastjson-0.99.4  Fast json library for C
py27-yaml-3.12      Collection of libraries to process YAML with Python
tcp_wrappers-7.6.4  Monitor and filter incoming requests for network services
py27-backports_abc-0.4 Backport of recent additions to the 'collections.abc' module
xmlcatmgr-2.2nb1    XML and SGML catalog manager
ncurses-6.0nb3      CRT screen handling and optimization package
py27-jinja2-2.8     Small but fast and easy to use stand-alone template engine
grep-2.24nb1        GNU grep
py27-crypto-2.6.1nb3 Cryptographic and hash functions for Python
py27-markupsafe-0.23 Implements a XML/HTML/XHTML Markup safe string for Python
sqlite3-3.15.2nb2   SQL Database Engine in a C Library
cwrappers-20161125  pkgsrc compiler wrappers
libev-4.22          Full-featured and high-performance event loop
tmux-2.3nb2         BSD-licensed terminal multiplexer (GNU Screen alternative)
xz-5.2.2            XZ utilities
nawk-20121220nb1    Brian Kernighan's pattern-directed scanning and processing language
liblogging-1.0.5    Easy to use logging library
p5-DBI-1.636nb2     The database-independent Perl database access API
pcre-8.39           Perl Compatible Regular Expressions library
gsed-4.2.2nb4       GNU implementation of sed, the POSIX stream editor
pkg_alternatives-1.6 Generic wrappers for programs with similar interfaces
percona56-cluster-galera-25.3.19 Synchronous Multi-Master replication plugin for MySQL
percona56-cluster-xtrabackup-2.4.7 Percona XtraBackup
percona56-cluster-1.2 Meta-package for a Percona environment
py27-m2crypto-0.22.3nb1 Crypto and SSL toolkit for Python
py27-msgpack-0.4.6  MessagePack (de)serializer for Python
libyaml-0.1.7       YAML 1.1 parser and emitter written in C
readline-7.0        GNU library that can recall and edit previous input
gmp-6.1.2           Library for arbitrary precision arithmetic
pkgsrc-gnupg-keys-20160519 GnuPG keys for pkgsrc infrastructure
python27-2.7.12nb4  Interpreted, interactive, object-oriented programming language
bzip2-1.0.6nb1      Block-sorting file compressor
jq-1.5nb1           Command-line JSON processor
wget-1.18nb3        Retrieve files from the 'net via HTTP and FTP
py27-simplejson-3.8.2 Simplejson is a simple, fast, extensible JSON encoder/decoder
py27-apache-libcloud-0.17.0 Python library for interacting with cloud service providers
py27-curl-7.19.3.1nb2 Python module interface to the cURL library
db4-4.8.30          Berkeley DB version 4 from Oracle
libssh2-1.8.0       SSH2 protocol library
postfix-3.1.3nb1    Postfix SMTP server and tools
gawk-4.1.3          GNU awk
zlib-1.2.8nb3       General purpose data compression library
py27-backports.ssl_match_hostname-3.5.0.1 Python SSL support module for Python 2
nbsed-20120308      NetBSD-current's sed(1)
libidn-1.33         Internationalized Domain Names command line tool
diffutils-3.4       GNU diff utilities - find the differences between files
qpress-1.1          qpress file archiver
gettext-lib-0.19.8.1 Internationalized Message Handling Library (libintl)
nghttp2-1.17.0      Implementation of HTTP/2 in C
zoneinit-1.6.9      Joyent Instance initialization toolchain
gtar-base-1.29      The GNU tape archiver with remote magnetic tape support
py27-tornado-4.3    Fast and non-blocking web framework
bsdinstall-20160108 Portable version of the BSD install(1) program
perl-5.24.0         Practical Extraction and Report Language
py27-requests-2.12.4 HTTP library, written in Python, for human beings
mozilla-rootcerts-1.0.20160610 Root CA certificates from the Mozilla Project
libsodium-1.0.8     Library for build higher-level cryptographic tools
mpfr-3.1.5          GMP-based library for multiple-precision floating-point computations
pkg_install-info-4.5nb3 Standalone GNU info file installation utility
libuuid-2.28.2      Generate unique identifiers for objects
bmake-20150505      Portable (autoconf) version of NetBSD 'make' utility
libestr-0.1.10      Library for some string essentials
openssl-1.0.2jnb1   Secure Socket Layer and cryptographic library
less-481            Pager similar to more and pg
quickbackup-percona-3.6 Percona quickbackup SMF service
libevent-2.0.22nb1  Asynchronous event notification library
liblognorm-2.0.1    Tool to normalize log data
patch-2.7.5         Patch files using diff output
gcc49-libs-4.9.4nb1 The GNU Compiler Collection (GCC) support shared libraries
py27-certifi-2015.11.20.1 Python package for providing Mozilla's CA Bundle
findutils-4.6.0nb1  The GNU find, locate, updatedb, and xargs utilities
openldap-client-2.4.44nb3 Lightweight Directory Access Protocol libraries and client programs
icu-58.2            Robust and full-featured Unicode services
curl-7.52.1         Client that groks URLs
pkg_install-20160410nb1 Package management and administration tools for pkgsrc
lz4-1.7.4.2         Extremely Fast Compression algorithm
py27-expat-2.7.12   Python interface to expat
py27-zmq-16.0.2     Python bindings for zeromq
rsyslog-8.23.0      The enhanced syslogd for Unix
pkgin-0.9.4nb4      Apt / yum like tool for managing pkgsrc binary packages
changepass-1.3.3    Simple tool to set user passwords non-interactively
expat-2.2.0         XML parser library written in C
libxml2-2.9.4nb2    XML parser library from the GNOME project
coreutils-8.25nb1   GNU basic file, shell and text manipulation utilities
libarchive-3.2.1nb2 Library to read/create different archive formats
cyrus-sasl-2.1.26nb5 Simple Authentication and Security Layer
libffi-3.2.1nb2     Foreign function interface
bootstrap-mk-files-20160908 *.mk files for the bootstrap bmake utility

[mamash]

Thanks, looks like Percona finally diverged from the galera upstream, and is using/expecting a new API version that upstream doesn't have. I'll see how to tackle this best.

[mamash]

Oh and the there are now code diverging branches for 5.6 and 5.7 of the Percona fork, that's pretty ugly. Sounds like I'l have to re-create as separate packages.

[siepkes]

@mamash Is the (different) issue which I encountered when running on 2017Q1 actually the same problem?

[mamash]

I'm currently working on integrating the galera plugin into the PXC package directly, because Percona seems to release new forked Galera versions together with their PXC packages, and it's a versioning nightmare otherwise.

I'll leave the separate (upstream) Galera plugin for original MySQL and Percona Server (non-cluster) packages, though I plan to test that still, too.

[siepkes]

@mamash Let me know if I can help or test anything!

[mamash]

I believe I finally tracked down the problem to this commit. In short, it's not allowed to use pthread_setschedparam() to set policy and parameters in a non-global SunOS zone.

This commit landed in galera 25.3.16, which explains why the last PXC image available on JPC works fine, because it only uses 25.3.9.

[siepkes]

Good find! Is it as easy as patching the source and disabling that option? I doubt it has a real impact?

[mamash]

I'll patch out thread priority scheduling for now completely, and raise a ticket with upstream.

[mamash]

The Percona packages in 2016Q4 are now updated to the latest versions and refactored - they come with the WSREP/Galera plugin lib patched and included, so no need to the external percona56-galera package.

[mamash]

Opened https://github.com/codership/galera/issues/472