search

SUSE / sesdev

CLI tool to deploy and manage SES clusters
MIT License
19 stars 32 forks source link

Creating a SES6 three node cluster results in chrony blocking forever #327

Closed p-se closed 4 years ago

p-se commented 4 years ago 
sesdev create ses6 --roles="[admin, prometheus, grafana, mon, mgr], \  
                 [storage, mon], [storage, mon]" \
                 --num-disks=2 --disk-size=10 ses6

[...]
 admin: 200 OK
    admin: + chronyc 'burst 4/4'
    admin: 503 No such source
    admin: + stdbuf -o0 chronyc waitsync
    admin: try: 1, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 2, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 2
    admin: , refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 3, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 3, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 4, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 4, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 5, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 5, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 6, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 6, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 7, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 7, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 8, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 8, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 9, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 9, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 10, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 10, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 11, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 11, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 12, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 12, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 13, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 13, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 14, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 14, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 15, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 15, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 16, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 16, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 17, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 17, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 18, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 18, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 19, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 19, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 20, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 20, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 21, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 21, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 22, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 22, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 23, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 23, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 24, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 24, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 25, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 25, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 26, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 26, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 27, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 27, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 28, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 28, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 29, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 29, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 30, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 30, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 31, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 31, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 32, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 32, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 33, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 33, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 34, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 34, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 35, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 35, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 36, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 36, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 37, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 37, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 38, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 38, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 39, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 39, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 40, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 40, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 41, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 41, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 42, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 42, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 43, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 43, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 44, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 44, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 45, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 45, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 46, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 46, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 47, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 47, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 48, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 48, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 49, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 49, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 50, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 50, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 51, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 51, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 52, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 52, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 53, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 53, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 54, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 54, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 55, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 55, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 56, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 56, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 57, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 57, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 58, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 58, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 59, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 59, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 60, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 60, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 61, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 61, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 62, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 62, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 63, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 63, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 64, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 64, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 65, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 65, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 66, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 66, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 67, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 67, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 68, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 68, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 69, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 69, refid: 7F7F0101, correction: 0.000000000, skew: 0.000

Second run:

[...]
    admin: 503 No such source
    admin: + stdbuf -o0 chronyc waitsync
    admin: try: 1, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: + chronyc makestep
    admin: 200 OK
    admin: + chronyc 'burst 4/4'
    admin: 503 No such source
    admin: + stdbuf -o0 chronyc waitsync
    admin: try: 1, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 2, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 2, refid: 7F7F0101
    admin: , correction: 0.000000000, skew: 0.000
    admin: try: 3, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 3, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 4, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 4, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 5, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 5, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 6, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 6, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 7, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 7, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 8, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 8, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 9, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 9, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 10, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 10, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 11, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 11, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 12, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 12, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 13, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 13, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 14, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 14, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 15, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 15, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 16, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 16, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 17, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 17, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 18, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 18, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 19, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 19, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 20, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 20, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 21, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 21, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 22, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 22, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 23, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 23, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 24, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 24, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 25, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 25, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 26, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 26, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 27, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 27, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 28, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 28, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 29, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 29, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 30, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
    admin: try: 30, refid: 7F7F0101, correction: 0.000000000, skew: 0.000
smithfarm commented 4 years ago

Yes, I can confirm this. It's essentially a duplicate of https://github.com/ceph/ceph-salt/issues/238 so let's try applying that fix.

smithfarm commented 4 years ago

This weird behavior was traced to the explicit admin role given on the command line. As of https://github.com/SUSE/sesdev/pull/330 sesdev raises an exception for that.