andrejohansson
commented
10 years ago Some additions....the minion stopped responding (running in console with -l debug) to test.ping calls from the master. I pinged three times without response, then when I should restarted I pressed CTRL+C on the minion and interestingly the minion didnt exit but some subprocess seemed to do and suddenly the minion started responding again.
So, could it be that the minion gets stuck in some sub process and while it is, it's not responsive to the master?
rs-sm1:
Minion did not return
root@saltmaster:/srv# salt rs-sm1 test.ping
rs-sm1:
Minion did not return
root@saltmaster:/srv# salt rs-sm1 test.ping
rs-sm1:
Minion did not return
root@saltmaster:/srv# salt rs-sm1 test.ping
rs-sm1:
TrueLogs for the same time:
[DEBUG ] Command details {'tgt_type': 'glob', 'jid': '20140528121010049293', '
tgt': 'rs-sm1', 'ret': '', 'user': 'sudo_andre', 'arg': [], 'fun': 'pkg.refresh_
db'}
[INFO ] User sudo_andre Executing command saltutil.find_job with jid 20140528
121020069146
[DEBUG ] Command details {'tgt_type': 'glob', 'jid': '20140528121020069146', '
tgt': 'rs-sm1', 'ret': '', 'user': 'sudo_andre', 'arg': ['20140528121010049293']
, 'fun': 'saltutil.find_job'}
[INFO ] User sudo_andre Executing command test.ping with jid 2014052812120573
0532
[DEBUG ] Command details {'tgt_type': 'glob', 'jid': '20140528121205730532', '
tgt': 'rs-sm1', 'ret': '', 'user': 'sudo_andre', 'arg': [], 'fun': 'test.ping'}
[INFO ] Returning information for job: 20140528121020069146
[INFO ] User sudo_andre Executing command saltutil.find_job with jid 20140528
121215753513
[DEBUG ] Command details {'tgt_type': 'glob', 'jid': '20140528121215753513', '
tgt': 'rs-sm1', 'ret': '', 'user': 'sudo_andre', 'arg': ['20140528121205730532']
, 'fun': 'saltutil.find_job'}
[INFO ] Returning information for job: 20140528121205730532
[INFO ] User sudo_andre Executing command test.ping with jid 2014052812122261
2574
[DEBUG ] Command details {'tgt_type': 'glob', 'jid': '20140528121222612574', '
tgt': 'rs-sm1', 'ret': '', 'user': 'sudo_andre', 'arg': [], 'fun': 'test.ping'}
[INFO ] Returning information for job: 20140528121215753513
[INFO ] User sudo_andre Executing command saltutil.find_job with jid 20140528
121232637500
[DEBUG ] Command details {'tgt_type': 'glob', 'jid': '20140528121232637500', '
tgt': 'rs-sm1', 'ret': '', 'user': 'sudo_andre', 'arg': ['20140528121222612574']
, 'fun': 'saltutil.find_job'}
[INFO ] Returning information for job: 20140528121222612574
[INFO ] Returning information for job: 20140528121232637500
[DEBUG ] Loaded minion key: c:\salt\conf\pki\minion\minion.pem
[DEBUG ] Decrypting the current master AES key
[DEBUG ] Loaded minion key: c:\salt\conf\pki\minion\minion.pem
[DEBUG ] Loaded minion key: c:\salt\conf\pki\minion\minion.pem
[DEBUG ] Decrypting the current master AES key
[DEBUG ] Loaded minion key: c:\salt\conf\pki\minion\minion.pem
[INFO ] Returning information for job: 20140528121010049293
[INFO ] User sudo_andre Executing command test.ping with jid 2014052812124914
7699
[DEBUG ] Command details {'tgt_type': 'glob', 'jid': '20140528121249147699', '
tgt': 'rs-sm1', 'ret': '', 'user': 'sudo_andre', 'arg': [], 'fun': 'test.ping'}
[INFO ] Returning information for job: 20140528121249147699
It was just before the line
[INFO ] Returning information for job: 20140528121222612574I pressed CTRL+C
basepi
UtahDave
olliewalsh
mechalaris
stale[bot]
First of all, I wanna say that my experience so far with Salt have been great. The architecture and simplicity of things when you get a hang of it is remarkable!
I've waited a bit to post this issue because it's hard to reproduce consistently, but unfortunately it's a major roadblock from using Salt in production environments using Windows unless I'm missing something big. Actually there is two problems but I describe the connection issues here and I will open another issue for the other problem (bootstrapping, se #13070).
We have our servers running in Azure, so everything here applies to Azure environments, also it will hopefully give you a good chance to replicate the environment. If not, contact me and you can have access to my environment
The Salt Master Is running Ubuntu 14.04 LTS (base image from Azure gallery) with only the Saltmaster installed.
The Salt Minion Is running Windows Server 2012 R2 64 Bit (base image from Azure gallery) with the SaltMinon installed and whatever software I've tried to install via salt.
The problem Connection drops quite often between the master and the minion, I'm working actively with the setup and maybe 1 out of 4 commands will get "the minion did not respond"/no output on the master. I am running the minion in console mode with -l debug.
Sometimes I see that the command reaches the minion and it works a bit. But many times it will not even reach the minions.
I get the feeling that there is two problems:
My workarounds so far:
Could #7159 be behind this maybe?
So, the stability in the connection prevents me from "trusting" that the minions do what I tell them to. Also the ability to call salt from scripts for automation is a no-go.
I wish I had more concrete information to give, but as I said, the environment is available to use if you want to see it for yourself.
Versions report