search

xorpaul / g10k

my r10k fork in Go
Apache License 2.0
125 stars 50 forks source link

g10k does not deploy module correctly when -retrygitcommands is set #130

Closed maxenced closed 5 years ago

maxenced commented 5 years ago

We have a big environment with 380 modules. When deploying a new environment with -maxworker 2 -maxextractworker 2 -retrygitcommands on 2 nodes I don't get consistent, neither satisfying results :

When digging, it seems that when we use -retrygitcommands g10k will retry once, and then ignore silently if the second try fails too : 

2019/06/06 10:27:13 Executing git --git-dir /var/cache/g10k/modules/git@git.company.net-PP-Puppet_common_dnsmasq.git remote update --prune took 0.27376s
WARN: git command failed: git --git-dir /var/cache/g10k/modules/git@git.company.net-PP-Puppet_common_dnsmasq.git remote update --prune deleting local cached repository and retrying...
2019/06/06 10:27:13 DEBUG purgeDir(): Trying to remove: /var/cache/g10k/modules/git@git.company.net-PP-Puppet_common_dnsmasq.git called from doMirrorOrUpdate, because git command failed, retrying
2019/06/06 10:27:13 DEBUG executeCommand(): Executing git clone --mirror git@git.company.net:PP-Puppet/common_dnsmasq.git /var/cache/g10k/modules/git@git.company.net-PP-Puppet_common_dnsmasq.git
2019/06/06 10:27:14 Executing git clone --mirror git@git.company.net:PP-Puppet/common_dnsmasq.git /var/cache/g10k/modules/git@git.company.net-PP-Puppet_common_dnsmasq.git took 0.88200s
WARN: git repository git@git.company.net:PP-Puppet/common_dnsmasq.git does not exist or is unreachable at this moment!
2019/06/06 10:27:22 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.company.net-PP-Puppet_common_dnsmasq.git rev-parse --verify 'production^{object}'
2019/06/06 10:27:22 Executing git --git-dir /var/cache/g10k/modules/git@git.company.net-PP-Puppet_common_dnsmasq.git rev-parse --verify 'production^{object}' took 0.00156s

But at the end the run does not raise any error : 

Synced /etc/g10k.yaml with branch test_g10k_deploy with 313 git repositories and 68 Forge modules in 17.4s with git (6.9s sync, I/O 10.1s) and Forge (2.0s query+download, I/O 0.4s) using 2 resolv and 2 extract workers

but some modules are missing

maxenced commented 5 years ago

So, checked the code, it seems the behavior is not directly related to the retrygitcommands option.

As I understand it, issue is that https://github.com/xorpaul/g10k/blob/d9d50df1ad60c6da91b15d48a75c308363e453c5/git.go#L81 does not check return value from doMirrorOrUpdate

So we have no way to detect that environment has deployed correctly (unless we grep in logs)

xorpaul commented 5 years ago

Please check if v0.6.1 works for you: https://github.com/xorpaul/g10k/releases/tag/v0.6.1