Closed devinnasar closed 5 years ago
Master logfile: master.log
Log tail from minion:
[root@OTDVAPVMXXXX dnasar]# tail -f /var/log/salt/minion
2018-05-04 14:10:41,092 [salt.minion :1854][INFO ][22040] Returning information for job: 20180504141041068653
2018-05-04 14:11:14,852 [salt.minion :1422][INFO ][28032] User sudo_dnasar Executingcommand state.apply with jid 20180504141114847682
2018-05-04 14:11:14,864 [salt.minion :1557][INFO ][22069] Starting a new job with PID 22069
2018-05-04 14:11:22,682 [salt.state :905 ][INFO ][22069] Loading fresh modules for state activity
2018-05-04 14:11:22,790 [salt.state :1762][INFO ][22069] Running state [/opt/otcli]at time 14:11:22.790948
2018-05-04 14:11:22,791 [salt.state :1795][INFO ][22069] Executing state file.absent for [/opt/otcli]
2018-05-04 14:11:22,802 [salt.state :290 ][INFO ][22069] {u'removed': u'/opt/otcli'}
2018-05-04 14:11:22,803 [salt.state :1943][INFO ][22069] Completed state [/opt/otcli] at time 14:11:22.803044 (duration_in_ms=12.095)
2018-05-04 14:11:22,803 [salt.state :1762][INFO ][22069] Running state [/opt/] at time 14:11:22.803776
2018-05-04 14:11:22,803 [salt.state :1795][INFO ][22069] Executing state archive.extracted for [/opt/]
2018-05-04 14:11:47,244 [salt.utils.schedule :1375][INFO ][28032] Running scheduled job: __mine_interval
2018-05-04 14:11:47,283 [salt.loaded.int.module.cmdmod :385 ][INFO ][22084] Executing command 'cat /etc/ssh/ssh_host_*_key.pub' in directory '/root'
2018-05-04 14:12:22,846 [salt.fileclient :1259][INFO ][22069] Fetching file from saltenv 'base', ** done ** 'opentempo/otcli/otcli-1.1-TEST.tar'
2018-05-04 14:12:22,941 [salt.state :290 ][INFO ][22069] {u'extracted_files': ['otcli-1.1-TEST', 'otcli-1.1-TEST/lib', 'otcli-1.1-TEST/lib/otcli-1.1-TEST.jar', 'otcli-1.1-TEST/lib/kotlin-stdlib-jdk8-1.2.10.jar', 'otcli-1.1-TEST/lib/kotlinx-coroutines-core-0.21.2.jar', 'otcli-1.1-TEST/lib/aws-java-sdk-cloudfront-1.11.257.jar', 'otcli-1.1-TEST/lib/aws-java-sdk-ecs-1.11.257.jar']}
2018-05-04 14:12:22,942 [salt.state :1943][INFO ][22069] Completed state [/opt/] at time 14:12:22.941979 (duration_in_ms=60138.202)
2018-05-04 14:12:22,942 [salt.state :1762][INFO ][22069] Running state [/opt/otcli] at time 14:12:22.942368
2018-05-04 14:12:22,942 [salt.state :1795][INFO ][22069] Executing state file.rename for [/opt/otcli]
2018-05-04 14:12:22,943 [salt.state :290 ][INFO ][22069] {u'/opt/otcli': u'/opt/otcli-1.1-TEST'}
2018-05-04 14:12:22,943 [salt.state :1943][INFO ][22069] Completed state [/opt/otcli] at time 14:12:22.943387 (duration_in_ms=1.019)
2018-05-04 14:12:22,943 [salt.state :1762][INFO ][22069] Running state [/opt/otcli/bin/otcli] at time 14:12:22.943832
2018-05-04 14:12:22,944 [salt.state :1795][INFO ][22069] Executing state file.exists for [/opt/otcli/bin/otcli]
2018-05-04 14:12:22,944 [salt.state :292 ][ERROR ][22069] Specified path /opt/otcli/bin/otcli does not exist
2018-05-04 14:12:22,944 [salt.state :1943][INFO ][22069] Completed state [/opt/otcli/bin/otcli] at time 14:12:22.944881 (duration_in_ms=1.049)
2018-05-04 14:12:22,946 [salt.minion :1854][INFO ][22069] Returning information for job: 20180504141114847682
^[2018-05-04 14:13:46,245 [salt.utils.schedule :1375][INFO ][28032] Running scheduled job: __mine_interval
2018-05-04 14:13:46,290 [salt.loaded.int.module.cmdmod :385 ][INFO ][22096] Executing command 'cat /etc/ssh/ssh_host_*_key.pub' in directory '/root'
2018-05-04 14:15:46,245 [salt.utils.schedule :1375][INFO ][28032] Running scheduled job: __mine_interval
2018-05-04 14:15:46,284 [salt.loaded.int.module.cmdmod :385 ][INFO ][22226] Executing command 'cat /etc/ssh/ssh_host_*_key.pub' in directory '/root
@devinnasar Thanks for the report. Using a simplified version of your state included above, I haven't been able to reproduce the issue. Are you able to comment everything in the state out and go through in stages to hopefully narrow down if a particular section is causing the issue with multiple minions? You mentioned using S3 but it appears you're using the Salt fileserver, are you using the s3fs Salt fileserver?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.
Description of Issue/Question
I'm installing a program to /opt/ from a tar archive which I have staged on s3. Periodically, archive.extracted states based on this archive fail when I target multiple minions, but not when I target them individually.
Setup
State:
Steps to Reproduce Issue
(Include debug logs if possible and relevant.)
Targeting multiple minions (unsuccessful, note how few files are extracted):
Targeting single minion (successful, note all files extracted):
Versions Report