dmurphy18
commented
5 years ago @gtmanfred @daks @javier Having looked at this, the issue is not really the scripts used to deploy but the way that updates are deployed. Currently, all the various scripts basically call a SLS state file which in turn calls an rsync command to update multiple web servers behind a load-balancer etc from a central image and it is this rsync update process which is disrupting the automated testing.
The correct method to use, is to update the multiple web servers with newer images etc, and flip on command from old images to new images, thereby minimizing interruption. Achieving this will take a while given SaltStack's IT department logistics etc.. There are plans to revise the hosting and architecture of repo.saltstack.com in the near future, and the IT department have been made aware of the issue and plan on addressing a solution as part of that revision.
Thanks for bringing this to SaltStack's attention and the issue will be addressed but no timeframe is available at the present time.
During the release of Salt 2019.2.0 and 2018.3.4 some users automated testing found some files unavailable, for example: gpg keys due to the way symbolic links are handled in release scripts. Need to minimize the disruption, hence examine scripts to override existing symbolic links rather than delete and replace:
Below captures some conversation on the matter from community slack general channel: daks [9:39 AM] https://repo.saltstack.com/apt/debian/9/amd64/2018.3/SALTSTACK-GPG-KEY.pub The debian GPG key seems unavailable
Javier Bértoli [9:40 AM] https://repo.saltstack.com/apt/debian/9/amd64/2018.3/SALTSTACK-GPG-KEY.pub
gtmanfred [9:40 AM] they are doing a release right now so it is all getting updated see #releases
daks [9:40 AM] ok, this break my tests! :slightly_smiling_face:
Javier Bértoli [9:40 AM] that's why it showed for me :slightly_smiling_face:
daks [9:41 AM] strange that the gpg key or its url change but ok will wait
gtmanfred [9:41 AM] it didn’t change it was unsymlinked as it was moved out of the way and the new latest repository was put in place the whole top level directory because 2018.3 is no longer the latest 2019.2 is
daks [9:42 AM] hum, still strange because the URL we use is not 'latest' but '2018.3'
gtmanfred [9:43 AM] so, 2018.3 is now linked to 2018.3.4 instead of 2018.3.3
daks [9:43 AM] but no problem, will go home, and run tests tomorrow :slightly_smiling_face: it's already back online
Daithi [9:57 AM] @Javier Bértoli just checked and the gpg key is there for https://repo.saltstack.com/apt/debian/9/amd64/2018.3/SALTSTACK-GPG-KEY.pub apologies for the temporary interruption, but did post in releases that these was about to be an update, in this case it was the update for 2018.3.3 -> 2018.3.4 as 2018.3.4 became the newest released version of the 2018.3 branch (edited) @daks ^^^
gtmanfred [9:59 AM] how are you doing the change of the symlink? you should just be able to do ln -sf , to overwrite the symlink, instead of deleting and then recreating.
daks [10:00 AM] @Daithi no problem, it's just that the interruption occurred just when I was running Kitchen tests on our code, and made highstate fail. It's OK now Congrats for the release
Daithi [10:05 AM] @gtmanfred use one of Dave Buscha's scripts, haven't ever dug into it, but perhaps I shall, add to the list and see what is going on
gtmanfred [10:06 AM] might want to update it
Daithi [10:06 AM] aye