load-config command fails on first chef run.

[daveallen]

I have found the issue and have fixed it here: https://github.com/daveallen/chef-proxysql/blob/bf28b662378fae01686f14a3a08199a61157f200/libraries/proxysql_service.rb#L314-L325

The issue is that when an OS is using systemd it adds a systemd service when an init style service is added. So that when this is run: https://github.com/daveallen/chef-proxysql/blob/bf28b662378fae01686f14a3a08199a61157f200/libraries/proxysql_service.rb#L231-L242

systemctl is-active #{constructed_service_name} returns 'active' executing the load-config command before the service is in place and functioning.

The code I have added removes the initial systemd service that points to the init style service.

[ernestas-poskus]

I have checked rpm contents and there is not systemd unit in package only init.d which is deleted afterwards, what OS and version you are using?

[root@150d4e730449 ~]# rpm -ql proxysql
/etc/init.d/proxysql
/etc/logrotate.d/proxysql-logrotate
/etc/proxysql-admin.cnf
/etc/proxysql.cnf
/usr/bin/proxysql
/usr/bin/proxysql-admin
/usr/bin/proxysql-status
/usr/bin/proxysql_galera_checker
/usr/bin/proxysql_node_monitor
/usr/share/doc/proxysql-1.4.7
/usr/share/doc/proxysql-1.4.7/LICENSE
/var/lib/proxysql
/var/run/proxysql

[ernestas-poskus]

cc @daveallen

[daveallen]

This was on Ubuntu 16.04. So it would seem that this behavior isn't happening on that platform. There is the only_if guard that will prevent that from happening, if that file doesn't exist. So it should run fin on that one too. Or should I say 'not run' just fine. lol

[ernestas-poskus]

I wonder why CI didn't fail on this

[daveallen]

That is a good question. It is consistent for me.
It fails on the load-config command which gets run before proxysql is added as a systems service: def deriver_install install_proxysql create_directories(service_data_dir) install_config install_service

    service constructed_service_name do
      supports(
        status: true,
        restart: true
      )
      action %i[enable start]
    end
  end

And the only reason that could happen is if the service variable is set as true. In this instance systemctl is-active #{constructed_service_name} returns 'active' before the the cookbook adds the system service.

def install_config statements = proxysql_statements.map { |statement| "#{statement};" } cmd = %(echo "#{statements.join(' ')}" | #{mysql_cmd}) service = if new_resource.service_provider == :systemd "systemctl is-active #{constructed_service_name}" else "service status #{constructed_service_name}" end variables = config_variables

    execute 'load-config' do
      command cmd
      action :nothing
      only_if service
    end

That is because the install not only installs the init service it also installs a a systemd wrapper which makes it active for systemd. You remove the init part but leave the systemd wrapper in '/lib/systemd/system/proxysql.service' and leave the systemd service enabled. So what my code does is remove that systemd service if the OS supports systemd and if that wrapper file exists. It cleans out the leftovers before your code adds the systemd service and prevents the load-config command from executing when it will fail.

One explanation could be that the proxysql could be running from the package install. Which is something you don't want either as your code is about to start it again under a new systemd service.

By the way, thanks for your efforts on this cookbook!

[ernestas-poskus]

That is because the install not only installs the init service it also installs a a systemd wrapper which makes it active for systemd. You remove the init part but leave the systemd wrapper in '/lib/systemd/system/proxysql.service' and leave the systemd service enabled.

Hm, this is weird because whenever I converge ubuntu 16.x or C7

root@711e7f537a0b:~# uname -a 
Linux 711e7f537a0b 4.14.18-200.fc26.x86_64 #1 SMP Thu Feb 8 01:35:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@711e7f537a0b:~# find /etc/ -name '*proxysql*'
/etc/logrotate.d/proxysql-logrotate
/etc/rc0.d/K01proxysql
/etc/rc1.d/K01proxysql
/etc/rc2.d/S01proxysql
/etc/rc3.d/S01proxysql
/etc/rc4.d/S01proxysql
/etc/rc5.d/S01proxysql
/etc/rc6.d/K01proxysql
/etc/systemd/system/multi-user.target.wants/proxysql-first.service
/etc/systemd/system/proxysql-first.service
/etc/proxysql
/etc/proxysql/proxysql-first.cnf
root@711e7f537a0b:~# find /var/ -name '*proxysql*'
/var/lib/dpkg/info/proxysql.list
/var/lib/dpkg/info/proxysql.preinst
/var/lib/dpkg/info/proxysql.md5sums
/var/lib/dpkg/info/proxysql.prerm
/var/lib/dpkg/info/proxysql.postinst
/var/lib/dpkg/info/proxysql.postrm
/var/lib/dpkg/info/proxysql.conffiles
/var/lib/proxysql
/var/lib/proxysql/proxysql.pid
/var/lib/proxysql/proxysql.log
/var/lib/proxysql/proxysql_stats.db
/var/lib/proxysql/proxysql.db
/var/lib/proxysql/proxysql-first
/var/lib/proxysql/proxysql-first/proxysql_stats.db
/var/lib/proxysql/proxysql-first/proxysql.db
root@711e7f537a0b:~# 

There is no wrapper systemd service :thinking: .

You remove the init part but leave the systemd wrapper

Furthermore if user of cookbook is configuring proxysql service with 'proxysql' service name this part

        # Remove package defaults
        %w[
          /etc/proxysql.cnf
          /etc/proxysql-admin.cnf
          /etc/init.d/proxysql
          /lib/systemd/system/proxysql.service
        ].each do |f|
          file f do
            action :delete
          end

may be causing disruption in service since every chef-client run init service is deleted, I will try to find a workaround for package defaults and chef cookbook installed service. I think we could pass fictitious environment variable https://github.com/poise/poise-service#quick-start to poise_service and before removing service file check if that environment variable exist or not.

cc @daveallen

[ernestas-poskus]

Might be related https://github.com/ernestas-poskus/chef-proxysql/pull/30

[daveallen]

Okay

1. I spun up a Digital Ocean droplet with Ubuntu 16.04.
2. I added the percona repository.
3. I installed proxysql 'apt install proxysql'
4. I listed the contents of /lib/systemd/system/ and got this: -rw-r--r-- 1 root root 603 Mar 8 17:51 proc-sys-fs-binfmt_misc.mount -rw-r--r-- 1 root root 299 May 22 14:10 proxysql.service -rw-r--r-- 1 root root 568 Mar 8 17:51 quotaon.service

As you can see there is a wrapper there with this as the contents: [Unit] After=network.target

[Install] WantedBy=multi-user.target

[Service] Type=forking Restart=no TimeoutSec=5min IgnoreSIGPIPE=no KillMode=process GuessMainPID=no RemainAfterExit=yes LimitNOFILE=102400 LimitCORE=1073741824 ExecStart=/etc/init.d/proxysql start ExecStop=/etc/init.d/proxysql stop

If you run 'systemctl is-active proxysql' it will return 'active' even though you deleted all the other files.

So that is why it is necessary to remove it from the systemd service. Otherwise, the first time you run your cookbook it says it is active and runs 'load-config' and fails because it hasn't added the cookbooks systemd unit, and started proxysql yet.

If your tests aren't catching this they should.

[ernestas-poskus]

If your tests aren't catching this they should.

I wonder why not, I will try to catch this, thanks for more detailed info.

Also we should include:

LimitNOFILE=102400
LimitCORE=1073741824

[ernestas-poskus]

@daveallen please test 2.3.1 release and close this if issue is fixed

[daveallen]

I tested it several times and it has worked every time. Thanks!