DRBD command syntax based on module versions instead of drbd8-utils version

[GoogleCodeExporter]

> What software version are you running?

# gnt-cluster --version
gnt-cluster (ganeti v2.10.2) 2.10.2

# gnt-cluster version
Software version: 2.10.2
Internode protocol: 2100000
Configuration format: 2100000
OS api version: 20
Export interface: 0
VCS version: (ganeti) version v2.10.2

# hspace --version
hspace (ganeti) version v2.10.2
compiled with ghc 7.4
running on linux x86_64

Ganeti installed from this PPA: 
https://launchpad.net/~pkg-ganeti-devel/+archive/lts

> What distribution are you using?

Ubuntu Precise 12.04.4 64 bit

# uname -r
3.2.0-60-generic

> What steps will reproduce the problem?

Recently, Ubuntu updated the drbd8-utils tool to version 8.4.3 (see 
https://bugs.launchpad.net/bugs/1185756) but kept the same version of the DRBD 
kernel module. The new utils allow supporting newer kernels but it's still 
possible to use the original kernel (3.2).

When using the original kernel (3.2), the DRBD kernel module is at version 
8.3.11 and doesn't match the drbd8-utils version (8.4.3).

Since Ganeti decides which drbdsetup syntax to use based on the module version, 
it picks the wrong syntax in that specific case. This gives error like this 
when activating a disk:

ganeti-noded pid=4590 INFO RunCmd drbdsetup /dev/drbd5 disk
/dev/ganeti/eb6f7df7-4062-4703-9ddb-6f1de11f99d8.disk0_data
/dev/ganeti/eb6f7df7-4062-4703-9ddb-6f1de11f99d8.disk0_meta 0 -e detach
--create-device -d 1536m --no-md-flushes --no-disk-flushes --no-disk-barrier
2014-04-22 12:14:13,081: ganeti-noded pid=4590 ERROR drbd5: can't attach
local disk:
USAGE: drbdsetup command device arguments options

Device is usually /dev/drbdX or /dev/drbd/X.

Commands are:
    primary             secondary           attach
    disk-options        detach              connect
    net-options         disconnect          resize
    resource-options    new-current-uuid    invalidate
    invalidate-remote   pause-sync          resume-sync
    suspend-io          resume-io           outdate
    verify              down                state
    role                sh-status           cstate
    dstate              show-gi             get-gi
    show                check-resize        events
    wait-connect        wait-sync           new-resource
    new-minor           del-minor           del-resource

To get more details about a command issue 'drbdsetup help cmd'.

invalid command

2014-04-22 12:14:13,081: ganeti-noded pid=4590 ERROR Error while
assembling disk: drbd5: can't attach local disk:
USAGE: drbdsetup command device arguments options

> Please provide any additional information below.

By manually editing storage/drbd.py and forcing a call to 
drbd_cmdgen.DRBD84CmdGenerator the instance's disks can be attached and the 
instance started.

I'm attaching the node-daemon.log from before the above hack.

Original issue reported on code.google.com by simon.de...@gmail.com on 22 Apr 2014 at 7:04

Attachments:

• node-daemon.log

[GoogleCodeExporter]

Original comment by hel...@google.com on 23 Apr 2014 at 8:44

• Changed state: Accepted
• Added labels: Milestone-Release2.13, Priority-Medium, Type-Defect

[GoogleCodeExporter]

I thought this was a way to determine the version of the command line tools:

# drbdadm status
<drbd-status version="8.3.13" api="88">
<resources config_file="/etc/drbd.conf">
</resources>
</drbd-status>

but it seems this subcommand was dropped entirely from 8.4 :-(

Original comment by bcandler...@googlemail.com on 23 Apr 2014 at 8:24

[GoogleCodeExporter]

Ganeti uses drbdsetup directly and does not go through drbdadm, so we shouldn't 
use drbdadm to detect the version anyway.

On a quick look, I didn't find an obvious way to get the version of drbdsetup. 
So we probably have to fall back to parsing the output of `drbdsetup help`...

Original comment by thoma...@google.com on 24 Apr 2014 at 6:55

[GoogleCodeExporter]

Although drbdadm isn't used directly, you could assume that drbdadm and 
drbdsetup are from the same package.  Looking through source I found this 
option:

$ drbdadm -V
DRBDADM_BUILDTAG=GIT-hash:\ 83ca112086600faacab2f157bc5a9324f7bd7f77\ build\ 
by\ phil@fat-tyre\,\ 2012-05-07\ 10:21:24
DRBDADM_API_VERSION=88
DRBD_KERNEL_VERSION_CODE=0x08030b
DRBDADM_VERSION_CODE=0x08030d
DRBDADM_VERSION=8.3.13

$ drbdadm -V
DRBDADM_BUILDTAG=GIT-hash:\ 89a294209144b68adb3ee85a73221f964d3ee515\ build\ 
by\ phil@fat-tyre\,\ 2013-02-05\ 15:35:49
DRBDADM_API_VERSION=1
DRBD_KERNEL_VERSION_CODE=0x080402
DRBDADM_VERSION_CODE=0x080403
DRBDADM_VERSION=8.4.3

However I couldn't find anything like this in drbdsetup. Code to print version 
number is in print_usage_and_exit() but is commented out :-(

So if you want to use drbdsetup only, I think you'd have to parse the result of 
a malformed command.

Version 8.3:

# drbdsetup show

USAGE: drbdsetup device command arguments options

Device is usually /dev/drbdX or /dev/drbd/X.
General options: --create-device, --set-defaults
... snip

Version 8.4:

$ drbdsetup show
Missing first argument
USAGE:
 drbdsetup show {resource|minor|all} [--show-defaults]

Original comment by bcandler...@googlemail.com on 24 Apr 2014 at 9:26

[GoogleCodeExporter]

OK, the problem is simpler than I thought.

drbdsetup automatically defers to /lib/drdb/drbdsetup-83 if the kernel drbd 
version is 8.3. However it barfs on some of the arguments passed by ganeti 
before doing that. An example command line ganeti passes is:

drbdsetup /dev/drbd2 disk 
/dev/xenvg/a614fd3b-d016-44a1-a6d6-c51334190757.disk0_data 
/dev/xenvg/a614fd3b-d016-44a1-a6d6-c51334190757.disk0_meta 0 -e detach 
--create-device -d 1024m

However, it works if you pass this command line to /lib/drbd/drbdsetup-83 
directly.

So the workaround I suggest is: in the DRBD83 code, if /lib/drbd/drbdsetup-83 
exists, use that in preference to drbdsetup.

(I've also sent a mail to drbd-user with this example, so the compatibility of 
drbdsetup may eventually be improved)

Original comment by bcandler...@googlemail.com on 24 Apr 2014 at 3:18

[GoogleCodeExporter]

So you have a link to your mail in the archives? It would be great if we could 
track the process there and see what's happening on their end.

Original comment by thoma...@google.com on 25 Apr 2014 at 7:05

[GoogleCodeExporter]

http://lists.linbit.com/pipermail/drbd-user/2014-April/020784.html
(No replies as yet)

Related thread in ganeti google group:
https://groups.google.com/forum/#!topic/ganeti/MkCNmzF6hu8

Original comment by bcandler...@googlemail.com on 25 Apr 2014 at 10:22

[GoogleCodeExporter]

It seems like Ubuntu is coming up with a better package that will remove the 
need to workaround this problem in Ganeti.

The details of how it was fixed in Ubuntu are here 
https://bugs.launchpad.net/ubuntu/+source/drbd8/+bug/1185756/comments/82

For those who might need the packages before they hit Ubuntu mirrors: 
http://people.canonical.com/~smb/lp1185756/

Original comment by simon.de...@gmail.com on 25 Apr 2014 at 4:03

[GoogleCodeExporter]

Reply from the drbd people at
http://lists.linbit.com/pipermail/drbd-user/2014-April/020804.html

Basically they say that if you are calling drbdsetup directly, it's up to you 
to call the right binary yourself. drbdsetup is not intended to have a 
backwards-compatible interface (drbdadm does), so if you call drbdsetup 
directly you have to know what you're doing.

Original comment by bcandler...@googlemail.com on 9 May 2014 at 7:57

[GoogleCodeExporter]

workaround:
apt-get remove drbd8-utils
apt-get install drbd8-utils=2:8.3.11-0ubuntu1
apt-mark hold drbd8-utils

Original comment by atxm...@gmail.com on 10 May 2014 at 6:05