[ansible/postgres] [ansible/artifactory] roles are not idempotent

[weakcamel]

Is this a request for help?:

Bug report.

---

Which installer:

Ansible

Which product and version:

Artifactory 7.12.5

What happened:

Running the playbook twice against an existing host restarts postgres and nginx despite no changes.

On second run of the playbook the following happens:

Restarted postgresql:

TASK [include_role : jfrog.installers.postgres] *************************************************************************************************************************************************************

TASK [jfrog.installers.postgres : define distribution-specific variables] ***********************************************************************************************************************************
ok: [artifactory.lab.speechmatics.io]

TASK [jfrog.installers.postgres : create directory for bind mount if necessary] *****************************************************************************************************************************
skipping: [artifactory.lab.speechmatics.io]

TASK [jfrog.installers.postgres : perform bind mount if necessary] ******************************************************************************************************************************************
skipping: [artifactory.lab.speechmatics.io]

TASK [jfrog.installers.postgres : perform installation] *****************************************************************************************************************************************************
included: /Users/waldekm/git/auto/galaxy-roles/ansible_collections/jfrog/installers/roles/postgres/tasks/Debian.yml for artifactory.lab.speechmatics.io

TASK [jfrog.installers.postgres : install python2 psycopg2] *************************************************************************************************************************************************
ok: [artifactory.lab.speechmatics.io]

TASK [jfrog.installers.postgres : install python3 psycopg2] *************************************************************************************************************************************************
ok: [artifactory.lab.speechmatics.io]

TASK [jfrog.installers.postgres : add postgres apt key] *****************************************************************************************************************************************************
ok: [artifactory.lab.speechmatics.io]

TASK [jfrog.installers.postgres : register APT repository] **************************************************************************************************************************************************
ok: [artifactory.lab.speechmatics.io]

TASK [jfrog.installers.postgres : install postgres packages] ************************************************************************************************************************************************
ok: [artifactory.lab.speechmatics.io]

TASK [jfrog.installers.postgres : extend path] **************************************************************************************************************************************************************
ok: [artifactory.lab.speechmatics.io]

TASK [jfrog.installers.postgres : initialize PostgreSQL database cluster] ***********************************************************************************************************************************
[WARNING]: Using world-readable permissions for temporary files Ansible needs to create when becoming an unprivileged user. This may be insecure. For information on securing this, see
https://docs.ansible.com/ansible/user_guide/become.html#risks-of-becoming-an-unprivileged-user
ok: [artifactory.lab.speechmatics.io]

TASK [jfrog.installers.postgres : install postgres configuration] *******************************************************************************************************************************************
[WARNING]: Using world-readable permissions for temporary files Ansible needs to create when becoming an unprivileged user. This may be insecure. For information on securing this, see
https://docs.ansible.com/ansible/user_guide/become.html#risks-of-becoming-an-unprivileged-user
ok: [artifactory.lab.speechmatics.io] => (item=pg_hba.conf)
ok: [artifactory.lab.speechmatics.io] => (item=postgresql.conf)

TASK [jfrog.installers.postgres : enable postgres service] **************************************************************************************************************************************************
ok: [artifactory.lab.speechmatics.io]

TASK [jfrog.installers.postgres : Hold until Postgresql is up and running] **********************************************************************************************************************************
ok: [artifactory.lab.speechmatics.io]

TASK [jfrog.installers.postgres : Create users] *************************************************************************************************************************************************************
[WARNING]: Using world-readable permissions for temporary files Ansible needs to create when becoming an unprivileged user. This may be insecure. For information on securing this, see
https://docs.ansible.com/ansible/user_guide/become.html#risks-of-becoming-an-unprivileged-user
ok: [artifactory.lab.speechmatics.io] => (item=None)
ok: [artifactory.lab.speechmatics.io]

TASK [jfrog.installers.postgres : Create a database] ********************************************************************************************************************************************************

TASK [jfrog.installers.postgres : Grant privs on db] ********************************************************************************************************************************************************

TASK [jfrog.installers.postgres : restart postgres] *********************************************************************************************************************************************************
changed: [artifactory.lab.speechmatics.io]

TASK [jfrog.installers.postgres : debug] ********************************************************************************************************************************************************************
ok: [artifactory.lab.speechmatics.io] => {
    "msg": "Restarted postgres service postgresql@9.6-main"
}

and later on nginx:

TASK [install nginx] ****************************************************************************************************************************************************************************************

TASK [jfrog.installers.artifactory_nginx : debug] ***********************************************************************************************************************************************************
ok: [artifactory.lab.speechmatics.io] => {
    "msg": "Attempting nginx installation without dependencies for potential offline mode."
}

TASK [jfrog.installers.artifactory_nginx : install nginx without dependencies] ******************************************************************************************************************************
ok: [artifactory.lab.speechmatics.io]

TASK [jfrog.installers.artifactory_nginx : configure main nginx conf file.] *********************************************************************************************************************************
ok: [artifactory.lab.speechmatics.io]

TASK [jfrog.installers.artifactory_nginx : configure the artifactory nginx conf] ****************************************************************************************************************************
ok: [artifactory.lab.speechmatics.io]

TASK [jfrog.installers.artifactory_nginx : restart nginx] ***************************************************************************************************************************************************
changed: [artifactory.lab.speechmatics.io]

What you expected to happen:

I'd expect this playbook to to idempotent, i.e. not take any actions if it's not making any changes to postgres or nginx. Especially an action as intrusive like service restart.

How to reproduce it (as minimally and precisely as possible):

https://github.com/jfrog/JFrog-Cloud-Installers/blob/master/Ansible/examples/playbook-rt.yml

Run it twice:

ansible-playbook -i inventory.yml playbook-rt.yml

Anything else we need to know:

n/a

[chukka]

@weakcamel Thanks for reporting , we already have internal tracking ticket for this - We are planning to release a new version in coming weeks with this support 👍

[weakcamel]

One more observation - which was probably already spotted during internal testing, but just in case: there is an idempotence issue with generating system.yaml file as well. Namely, there are 3 situations when it's being updated:

1. by this Ansible role when generating it for the 1st time
2. by installService.sh script ran from this Ansible role (probably one time only as well, but I'm not sure if that's so) - adding the user.shared
3. by Artifactory itself when it's encrypting the DB key

This means that re-running Ansible overwrites changes done by (2) and (3).

[weakcamel]

+ with nginx config when using SSL

[michaeloa]

xray has the same problem.

This is a pretty huge deal, because it makes it almost impossible to use these roles for any kind of effective system management. Any error while running the scripts means that you pretty much have to wipe everything and reinstall from scratch (and sometimes even that does not work), and these scripts are very error prone.

[chukka]

Agreed , Will be rolling out a new ansible release (post Artifactory 7.17.x is rolled out next week )

[michaeloa]

Would suggest looking through the role configs and make them more consistent with each other also.

E.g., we got tripped up recently because the artifactory role uses jdbc://postgres in its db host definition, but then the xray role uses just the postgres part (without jdbc://). Since the error messages are not very obvious (one of ansible's weaknesses), this led to a really quite unnecessary jfrog ticket.

[chukka]

@michaeloa Sure, we simplified installation across the products in the next release , BTW, Artifactory is java based hence jdbc in the URL and Xray go based , hence without jdbc , hope it helps

[weakcamel]

@chukka sorry to be a pain: is there a timeline for this 2.0 version of Ansible Collection?

My company has a large instance we need to re-install & migrate the data.

Automation of the soon-to-be-deployed instance is based on this existing Artifactory role (with several workarounds) but we're hesitant to roll it out - expecting that the 2.0 version may be significantly different and swapping the installation method may prove a challenge. Waiting for this release is blocking us at the moment - so could you shed some light on when it's going to be available?

[chukka]

@weakcamel Yes, we are planning to release post artifactory 7.18.x release rolls out (which will happen in couple of weeks ) . Currently we are certifying all our products to be postgresql 13 compatible as we want to ship 2.0 version of ansible collection with optional external postgresql 13 by default (hence the delay)

Also, This new plaform ansible installer supports

• Artifactory, xray, distribution and missioncontrol roles
• Optional (external) postgresql role
• All roles will be idempotent
• Supports ubuntu and centos (LTS) OS flavours

Hope this helps :)

[weakcamel]

@chukka Thank you, however this is still rather vague.

It was "in coming weeks" 3 months ago on 07.01 here and then "post Artifactory 7.17.x is rolled out next week" 3 weeks ago. Is there a specific date?

Currently we are certifying all our products to be postgresql 13 compatible as we want to ship 2.0 version of ansible collection with optional external postgresql 13 by default (hence the delay)

I completely understand and appreciate that this may be a large task. Sadly, current state of the installer is "utterly broken in many painful ways". As a paying customer, I honestly care less about the certification, what I care about is a working installation method. A hotfix release for existing collection is much more urgent than PG 13.0 support (I imagine most users run Postgres independently from Artifactory anyway).

If I knew It won't happen for months, I'd have forked this repository long time ago (or gave up on using this collection entirely). Does the license allow it? There's no LICENSE file at the root folder of this repo.

[michaeloa]

I completely understand and appreciate that this may be a large task. Sadly, current state of the installer is "utterly broken in many painful ways". As a paying customer, I honestly care less about the certification, what I care about is a working installation method.

I sadly have to concur with this. We have recently purchased an XRay license, and we can literally not install it using this collection. Which is a problem for us, because our Artifactory instance is installed using this collection, and we do not in general deploy anything without automating it. Which means we either have to wait for this to be fixed (we have a support case with JFrog support now) or roll our own ansible role that fixes the problem.

[chukka]

@michaeloa @weakcamel please review this PR - This new collection (jfrog.platform) will replace existing (jfrog.installers - implies v1.1.2 will be depreciated) . Please try and share feedback

[woyteck1]

Hi, @chukka,

I tried the new collection and we still get 4 changes on rerunning the jfrog.platform collection. These are the roles that keep having the "changed" status.

TASK [jfrog.platform.artifactory : Configure systemyaml] ************************************************************************************************************
changed: [artifacts.host]

TASK [jfrog.platform.artifactory : Configure join key] **************************************************************************************************************
changed: [artifacts.host]

TASK [jfrog.platform.artifactory : Create artifactory service] ******************************************************************************************************
changed: [artifacts.host]

RUNNING HANDLER [jfrog.platform.artifactory : restart artifactory] **************************************************************************************************
changed: [artifacts.host]

[weakcamel]

@chukka Sorry, I didn't even have the chance to review that - #106 was both posted and closed on the same day. Although to be honest I'm not sure I'd have been able to anyway (256 files changed).

We've had a chance to test the newest collection and there are still some problems with idempotency (as posted by my colleague above) - can this issue be reopened please?

[chukka]

@woyteck1 @weakcamel Interesting , Looks like systemyaml and joinkey are encrypted in the filesystem, Ansible reports it as changed , Let me look at other 2 tasks too. I prefer if we can have new issue raised for new collection (jfrog.platform)

[weakcamel]

@chukka Have you seen my earlier comment https://github.com/jfrog/JFrog-Cloud-Installers/issues/75#issuecomment-756621779 ?

[michaeloa]

We have just gotten xray set up with the old roles (after a turn through JFrog support - who were very helpful, to your credit) - so I don't think we will be updating this in the next couple of months. Maybe after the summer, assuming we can find some time.

[chukka]

@weakcamel Since this is an older issue for all roles and closed , can you please raise a new issue for artifactory role only . we are already tracking this internally . will plan to fix in upcoming releases

[weakcamel]

@chukka Raised again as https://github.com/jfrog/JFrog-Cloud-Installers/issues/116