[admin] add an option to trigger a graceful site reload

[signebedi]

[admin] add an option to trigger a graceful site reload I think that we touch a .reload_triggered file in the instance directory, remove the file, and then (maybe? I think this might be a buggy approach) run os.system and restart all the systemd daemons.

Originally posted by @signebedi in https://github.com/signebedi/gita-api/issues/48#issuecomment-1913666820

[signebedi]

So maybe a headless view that takes an API key (?) and validates it belongs to an admin user.

@app.route('/trigger-restart', methods=['POST'])
def trigger_restart():
    # Get API key from the POST data
    api_key = request.json.get('api_key')

    if not api_key:
        return jsonify(success=False, message="API Key is required."), 400

    # Find user by API key
    user = User.query.filter_by(api_key=api_key).first()

    # Check if user is a site admin
    if not user or not user.site_admin:
        # If user doesn't exist or is not a site admin, return an error
        abort(403)

      try:
          # Touch the .reload_triggered file
          instance_path = os.path.join('instance', '.reload_triggered')          
          with open(instance_path, 'a'):
              os.utime(instance_path, None)
          return jsonify(success=True)
      except Exception as e:
          return jsonify(success=False, error=str(e)), 500

An admin view would call this using eg. a javascript fetch call.

We also would need a celery(beat) task that periodically checks for the .reload_triggered file, removes it, and restarts.

@celery.task
def restart_services():
    instance_path = os.path.join('instance', '.reload_triggered')
    # Check if the .reload_triggered file exists
    if os.path.exists(instance_path):
        try:
                        # Restart Flask app service
            subprocess.check_call(['sudo', 'systemctl', 'restart', f'{app.config["ENVIRONMENT"]}-gita-api-gunicorn.service'])
            # Restart Celery worker service
            subprocess.check_call(['sudo', 'systemctl', 'restart', f'{app.config["ENVIRONMENT"]}-gita-api-celery.service'])
            # Restart Celery Beat service
            subprocess.check_call(['sudo', 'systemctl', 'restart',  f'{app.config["ENVIRONMENT"]}-gita-api-celerybeat.service'])
            # Remove the file after triggering the restart
            os.remove(instance_path)
        except subprocess.CalledProcessError as e:
            print(f'An error occurred while restarting services: {e}') # This needs to be a better method to log / capture the Exception

[signebedi]

I think that we need to add reload options to the systemd units for celery, celerybeat, and gunicorn.

See eg.

1. https://stackoverflow.com/a/9885625/13301284
2. https://serverfault.com/a/1013290
3. https://docs.gunicorn.org/en/stable/signals.html
4. https://docs.celeryq.dev/en/latest/userguide/daemonizing.html#service-file-celery-service

So this means adding eg. a ExecReload=/bin/kill -HUP $MAINPID line to gunicorn. Of course, this does not address a permissions issue for the service user trying to restart a systemd service without (rightfully) any root privileges.

[signebedi]

This probably also entails creating a celery configuration file, see https://docs.celeryq.dev/en/latest/userguide/daemonizing.html#generic-systemd-celery-example.

[signebedi]

Add a celery config We should create a celery configuration file, see https://docs.celeryq.dev/en/latest/userguide/daemonizing.html#generic-systemd-celery-example. We should also update the systemd unit file to use the celery config, and we should add an execreload option. Nb. because celery uses env files, we can probably integrate this with the config initialization in gita-init and write these to the application env file. This will of course require us to set env-specific config files in the systemd unit but, because we've already encapsulated much of this in a click interface that requires an environment option, this should be pretty easy to set.