Training notes - using Jepsen Linearizability testers to find faults with Zookeeper

[michaelmaguire]

Zookeeper claims to offer "Linearizable Writes" with "Sequential Reads" consistency:

( "strongest" at the top, "weaker" at the bottom. https://aphyr.com/posts/313-strong-consistency-models )

It does not offer "Linearizable" or "Strong Serializable".

For fun, to illustrate the Jepsen framework, we tested Zookeeper against a higher level of consistency, namely "Linearizable", expecting it to fail.

Creating a test framework:

admin@ip-192-168-32-100:~$ lein new jepsen.zookeeper

Then edit

jepsen.zookeeper/project.clj and jepsen.zookeeper/src/zookeeper.clj

as show in this project's source.

Running a test:

admin@ip-192-168-32-100:~/jepsen.zookeeper master$ lein run test --nodes-file ~/nodes --username admin | tee output.log

[michaelmaguire]

See Gavin Lowe for paper on Linearizability Testing

http://www.cs.ox.ac.uk/people/gavin.lowe/LinearizabiltyTesting/paper.pdf

See:

https://github.com/jepsen-io/knossos

for implementation

[michaelmaguire]

Timeline diagram reads from top -> bottom in time:

http://ctolab.xtremeghost.com/student32/jepsen/files/zookeeper/20161214T110856.000Z/timeline.html

Failure of linearity shown in: http://ctolab.xtremeghost.com/student32/jepsen/files/zookeeper/20161214T110856.000Z/linear.svg

which reads left -> right in time.

[michaelmaguire]

Sample Zookeeper logs:

admin@ip-192-168-32-100:~/jepsen.zookeeper master$ ssh w5 less /var/log/zookeeper/zookeeper.log                                         
2016-12-13 14:54:32,162 - INFO  [main:QuorumPeerConfig@101] - Reading configuration from: /etc/zookeeper/conf/zoo.cfg                   
2016-12-13 14:54:32,169 - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3                                    
2016-12-13 14:54:32,169 - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0                                      
2016-12-13 14:54:32,169 - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.                                         
2016-12-13 14:54:32,170 - WARN  [main:QuorumPeerMain@113] - Either no config or no quorum defined in config, running  in standalone mode
2016-12-13 14:54:32,191 - INFO  [main:QuorumPeerConfig@101] - Reading configuration from: /etc/zookeeper/conf/zoo.cfg                   
2016-12-13 14:54:32,199 - INFO  [main:ZooKeeperServerMain@95] - Starting server                                                         
2016-12-13 14:54:32,211 - INFO  [main:Environment@100] - Server environment:zookeeper.version=3.4.5--1, built on 03/16/2014 21:36 GMT   
2016-12-13 14:54:32,211 - INFO  [main:Environment@100] - Server environment:host.name=w5                                                
2016-12-13 14:54:32,211 - INFO  [main:Environment@100] - Server environment:java.version=1.8.0_111                                      
2016-12-13 14:54:32,212 - INFO  [main:Environment@100] - Server environment:java.vendor=Oracle Corporation                              
2016-12-13 14:54:32,212 - INFO  [main:Environment@100] - Server environment:java.home=/usr/lib/jvm/java-8-oracle/jre                    
2016-12-13 14:54:32,212 - INFO  [main:Environment@100] - Server environment:java.class.path=/etc/zookeeper/conf:/usr/share/java/jline.ja
r:/usr/share/java/log4j-1.2.jar:/usr/share/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/netty.jar:/usr/share/ja
va/slf4j-api.jar:/usr/share/java/slf4j-log4j12.jar:/usr/share/java/zookeeper.jar                                                        
2016-12-13 14:54:32,212 - INFO  [main:Environment@100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/l
ib64:/lib:/usr/lib                                                                                                                      
2016-12-13 14:54:32,212 - INFO  [main:Environment@100] - Server environment:java.io.tmpdir=/tmp                                         
2016-12-13 14:54:32,212 - INFO  [main:Environment@100] - Server environment:java.compiler=<NA>                                          
2016-12-13 14:54:32,214 - INFO  [main:Environment@100] - Server environment:os.name=Linux                                               
2016-12-13 14:54:32,214 - INFO  [main:Environment@100] - Server environment:os.arch=amd64                                               
2016-12-13 14:54:32,214 - INFO  [main:Environment@100] - Server environment:os.version=3.16.0-4-amd64                                   
2016-12-13 14:54:32,215 - INFO  [main:Environment@100] - Server environment:user.name=zookeeper                                         
2016-12-13 14:54:32,215 - INFO  [main:Environment@100] - Server environment:user.home=/var/lib/zookeeper                                
2016-12-13 14:54:32,215 - INFO  [main:Environment@100] - Server environment:user.dir=/                                                  
2016-12-13 14:54:32,224 - INFO  [main:ZooKeeperServer@726] - tickTime set to 2000                                                       
2016-12-13 14:54:32,224 - INFO  [main:ZooKeeperServer@735] - minSessionTimeout set to -1                                                
2016-12-13 14:54:32,224 - INFO  [main:ZooKeeperServer@744] - maxSessionTimeout set to -1                                                
2016-12-13 14:54:32,253 - INFO  [main:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:2181                                   
2016-12-13 14:54:32,272 - INFO  [main:FileTxnSnapLog@240] - Snapshotting: 0x0 to /var/lib/zookeeper/version-2/snapshot.0   

[michaelmaguire]

Examples of Zookeeper "Linearizable" fails (using test code in this repo)

http://ctolab.xtremeghost.com/student7/jepsen/files/noop/20161214T112120.000Z/linear.svg

http://ctolab.xtremeghost.com/student7/jepsen/files/noop/20161214T112120.000Z/timeline.html

[michaelmaguire]

Another stale read example:

http://ctolab.xtremeghost.com/student4/jepsen/files/zookeeper/20161214T110938.000Z/timeline.html