HDP-3.0 with postgres: Schema registry (HDF) startup fails on centos-7 (latest) because default postgresql (v9.2) is too old

[lhoss]

from the Schema Registry Server Start log:

"migrate" option failed : org.flywaydb.core.internal.exception.FlywayEnterpriseUpgradeRequiredException: Flyway Enterprise or PostgreSQL upgrade required: PostgreSQL 9.2 is past regular support by PostgreSQL and no longer supported by Flyway Community Edition and Flyway Pro Edition, but still supported by Flyway Enterprise Edition.

The tested installation contains a (almost)full HDP-3.0.1 stack, plus NIFI and the 2 registries (REGISTRY_SERVER being the SchemaRegistry):

  - host_group: "hdp-master"
    services:
    - REGISTRY_SERVER
    - NIFI_REGISTRY_MASTER ...

Solution options

• using an external DB (to be installed+prepared outside this role), by configuring database_options.external_hostname ** con: DBs + Users needs to be populated (one could re-use the the existing database_options var tough)
• consider using an external role to install/configure the databases (which often have such flexibility builtin), like: galaxy.ansible.com/geerlingguy/postgresql
• added role support to specify an optional/custom pgdg repo * but I'm not sure howto to best support overriding the (included) postgres_ vars depending on the choosen version ** I will attach a patch, how I made it work, but where I 'hardcoded' existing vars to use 9.6. Of course different ways to make this configurable.. but first I went @alexandruanghel 's opinion if this could be a viable contribution

[lhoss]

• current patch, to solution °3 above (which isn't in a mergeable state): https://github.com/hortonworks/ansible-hortonworks/commit/18815af2f7b84c5d10a28b76541e8a05f77862ee
• deployment tested with the patch AND the SchemaRegistry issue was indeed fixed

[alexandruanghel]

Uh, great, nothing like default DB version being deprecated in the new software releases.

Sounds like we'd have to go on a similar route with MySQL / MariaDB as you correctly done (add a repo and install that non-OS-default version).

Also see my answer on https://github.com/hortonworks/ansible-hortonworks/issues/46

The only thing is that this new addition will have to be done for all OS-es. You can submit your PR for CentOS7 if you like and then I can add support for the rest of the OS-es.

[lhoss]

You can submit your PR for CentOS7 if you like and then I can add support for the rest of the OS-es.

I'm happy to contribute for centos/redhat-7!

Some quick questions/comments:

• (instead of using yum, as in patch-^) I'ld also use a yum_repository task (as already in the role for mysql)
• I was still wondering if it's a complete nogo for the role to start using external roles (at some point), for ex. to deploy the dbs ?
• Assuming the deploy is still expected to support (at least also) HDP-2.6.x, we should test if the older HDP stack already works with postgres v9.6 (I bet officially not supported!)
• OR we should consider, from the start, to support different postgres versions (dep. on hdp_version)
  • this could be solved using a dynamic fact postgres_maj_version , which is then used in the various vars like postgres_repo_url: 'https://download.postgresql.org/pub/repos/yum/{{postgres_maj_version}}/redhat/rhel-7-x86_64/'
• I wonder if one could make the config more hierarchical, so that in the 3 vars/redhat*yml config files, only the vars that differ, would override some defaults, common to any 'redhat' family distro .. but I can also see there's pros+cons
  • Actually the more problematic IMO issue with this included vars is that one cannot override them in host or group_vars (which have lesser priority), but that's a generic issue in ansible (unless one uses (dynamic)inventory-groups per distro for such vars, but not nice also)

[alexandruanghel]

Always use https://supportmatrix.hortonworks.com to check for any compatibility. There is a large matrix of what's supported or not so quite complex to do this in Ansible (at least in these playbooks, which or more "universal").

You're right in your assumptions, some versions might work (outside of that matrix), and for many things the difference between supported and not supported is "we haven't tested / certified that" rather than "it won't work". Which is also a reason why I've been relaxed a bit about versions in the database role/vars folder.

For the moment, the static way of defining the version would be preferred: https://github.com/hortonworks/ansible-hortonworks/blob/master/playbooks/roles/database/vars/redhat-7.yml#L9 Obviously that can be moved to the main group_vars but I'd like to keep that as simple as possible. It also doesn't help that some older db versions are not available in newer OS-es (like MySQL 5.6 in Ubuntu 16). And if really needed to overwrite that, you can use --extra-vars option.

To do these versions dynamically you would need to create some sort of internal matrix / variable file in Ansible, which will also need to be maintained, etc.

Anyway, the point of these playbooks is to provide a baseline. You can of course use other roles and prepare the database outside these playbooks. As long as database_options.external_hostname is set and the username/passwords are correct and the database is created in advance (essentially what the database role does) then the blueprint should work fine. This would actually be required if you use an external cloud service like RDS. And the reason why the entire database role is skipped if database_options.external_hostname is not empty: https://github.com/hortonworks/ansible-hortonworks/blob/master/playbooks/prepare_nodes.yml#L17

Another expectation that I have is people would customize this as much as possible for their "production". Of course things can be contributed back if they are "generic" but it's sometimes more efficient when you're only using 1 flavour of OS and 1-2 versions of Ambari/HDP. And by customizing I also mean changing and maintaining the few variable files that might be required for that environment, including matching the Ambari version with HDP and the database version, and maybe customizing the repo url of the db: https://github.com/hortonworks/ansible-hortonworks/blob/master/playbooks/roles/database/vars/redhat-7.yml#L10

[lhoss]

@alexandruanghel this should be a good base https://github.com/hortonworks/ansible-hortonworks/pull/67 👍

[alexandruanghel]

Great! Looks good at a first look!

I haven't tested yet, and before I do, can you integrate please this change: https://github.com/hortonworks/ansible-hortonworks/commit/de71ccf4918b6e17df4bb297c6b50f8a5b3603cf

This is for the repo toggle and I had the change ready as part of https://github.com/hortonworks/ansible-hortonworks/issues/46

[lhoss]

can you integrate please this change: de71ccf

I just rebased the branch to upstream master 👍