"munge cred decode: Socket communication error" on server side

[borkd]

Clients attempting to diodmount from a centos-6 server using munge auth fail. Without authentication diodmount succeeds.

Munge realm is working across all participating machines, both local tests munge -n | unmunge and remote ones munge -n | ssh host unmunge work fine.

I have tested with diod-1.0.24 and diod-1.0.23. No luck.

Client-side (debian 9.1):

client # munge -n | ssh 10.d.e.f unmunge
STATUS:           Success (0)
ENCODE_HOST:      ??? (0.0.0.0)
ENCODE_TIME:      2017-12-02 01:06:18 -0500 (1512194778)
DECODE_TIME:      2017-12-02 01:08:24 -0500 (1512194904)
TTL:              300
CIPHER:           aes128 (4)
MAC:              sha1 (3)
ZIP:              none (0)
UID:              root (0)
GID:              root (0)
LENGTH:           0

client # diodmount -o ro,noatime 10.d.e.f:/mnt /mnt
diodmount: attach: Operation not permitted

Server side (centos 6):

server # diod  --nwthreads 1 --foreground --export /mnt  -d 1
diod: P9_TVERSION tag 65535 msize 65536 version '9P2000.L'
diod: P9_RVERSION tag 65535 msize 65536 version '9P2000.L'
diod: P9_TAUTH tag 0 afid 0 uname '' aname '/mnt' n_uname 0
diod: P9_RAUTH tag 0 qid (0000000000000000 0 'A')
diod: P9_TWRITE tag 0 fid 0 offset 0 count 127
4d554e47 453a4177 51444141 446e5431 7154577a 5245482f 6d747565 6e48324f
79424749 41557250 51347043 49427836 56726174 582b756d 62526536 75477330
diod: P9_RWRITE tag 0 count 127
diod: P9_TATTACH tag 0 fid 1 afid 0 uname '' aname '/mnt' n_uname 0
diod: checkauth(root@10.a.b.c:/mnt): munge cred decode: Socket communication error
diod: attach(0@10.a.b.c:/mnt): checkauth: Operation not permitted
diod: P9_RLERROR tag 0 ecode 1
diod: P9_TCLUNK tag 0 fid 0
diod: P9_RCLUNK tag 0

Any suggestions what could be broken?

Thank you!

[garlick]

One possibility is that the diod server is trying to open a different path to the munge socket than munged is offering. See:

dun/munge#38

since that would be entirely a server problem, it might be interesting to see if the centos 6 system can diodmount its own file system with munge auth.

[borkd]

Indeed. After rebuilding and installing munge and diod by hand, the old fashioned way, client and server are happy to talk. My spack-fu is still weak. Thank you!

[garlick]

Ah, yeah it seems problematic to build libmunge in spack since it will have to match the installed munged, and you can't know for sure how the installed one was built.