Making Email Alerts more generic and scalable

[tbembersimeao]

Motivation and thoughts

My team is developing a module that needs a UI to configure and send emails. This will be the third time we are facing this sort of task in the past months. I would like to stop copying code between modules. I'd like to do something more reusable.

In researching this I was looking into Email Alerts. I'd really like to use the great UI this module provides. However, the only triggering event available is instrument submission. We need the ability to build a custom trigger.

It would be helpful to me and my work to extend/plugin Email Alerts functionality in terms of 1) triggers and 2) Piping variables. I imagine the technical features would work like this:

• External modules would be able to implement a hook that declares their own triggers - and Piping variables for each trigger.
• Trigger types can be:
  • Global: available on Control Center only.
  • Project: available on Project level only.
  • Record: available on Project level only. The existing Piping and Survey buttons configuration applies to it.
• As soon as trigger-declaring module is enabled, the new triggers and piping variables are automatically included in Email Alerts UI.
• The existing instrument submission trigger would be one option among many triggers that other modules may provide (e.g. project creation, project deletion, user creation, report generation, file upload, user role assignment, etc).
• Each trigger-declaring module is responsible for triggering its own events by calling an Email Alerts method.

User workflow modifications

The end User workflow would have these modifications:

• As this new architecture supports events outside of the project context, it will also present a link on the Control Center.
• Configure Email Alerts form is split up into 2 sections
  1. General settings: keeps generic settings like "Preload email addresses" (displayed on Project version only), "Define Sender Email Alerts", and "Send Failed Email Alerts".
  2. Data entry Piping settings (displayed on Project version only, applies for Record triggers only): carries the record context settings such as Piping and Survey links
• When the user presses Add New Email button, the user is prompted to select the type of trigger.
• The email creation modal is then presented. Any custom piping variables added by the trigger will automatically be presented as buttons on the New Alert page.
• The workflow proceeds normally at this point

Code snippets from a trigger-declaring module

This is a sketch of what a trigger-declaring module would look like - let's say we want to trigger notifications when a new project is created:

Trigger declaration hook

<?php

/**
 * @inheritdoc
 */
function redcap_email_alerts_triggers() {
    $info = array();

    $info['project_creation'] = array(
        'label' => 'Project creation',
        'type' => 'global',
        'buttons_color' => 'pink',
        'variables' => array(
            'project' => array(
                'name' => array(                     // To be available as [project][name]
                    'label' => 'Project name',       // Button label
                    'restrict_display_to' => 'body', // It could be also set as "address"
                ),
            ),
        ),
    );

    return $info;
}

Triggering an event

<?php

/**
 * @inheritdoc
 */
function redcap_every_page_top($project_id) {
    // Checking if Email Alert module is active.
    if (!class_exists('\Vanderbilt\EmailTriggerExternalModule')) {
        return;
    }

    // Checking if we are at the project creation page.
    if (PAGE != 'ProjectGeneral/create_project.php' || $_SERVER['REQUEST_METHOD'] != 'POST') {
        return;
    }

    // Triggering the event to be handled by Email Alerts module.
    \Vanderbilt\EmailTriggerExternalModule::trigger('project_creation', array(
        'project' => array(
            'name' => \Project::getValidProjectName($_POST['app_title']),
        ),
    ));
}

Architecture changes needed within email-alerts-module

Below is a sketch of a new method that checks for additional available triggers. The values returned by this method would be used to extend the list of available triggers in the UI and display any added piping variables available to the user.

<?php

/**
 * Looks for additional triggers provided by other external modules.
 */
static function getAvailableTriggers($project_id = null, $filter_by_context = true) {
    $triggers = array();
    $hook = 'redcap_email_alerts_triggers';

    $types = array('global', 'project', 'record');    
    if ($filter_by_context) {
        $types = empty($project_id) ? array('global') : array('project', 'record');
    }

    // Looks for enabled modules that implement Email Alerts hook in order to
    // include additional triggers.
    foreach (\ExternalModules::getEnabledModules($project_id) as $prefix => $version) {
        $module = \ExternalModules::getModuleInstance($prefix, $version);
        if (!method_exists($module, $hook)) {
            continue;
        }

        if (!$module_triggers = $module->$hook()) {
            continue;
        }

        if (!is_array($module_triggers)) {
            continue;
        }

        foreach ($module_triggers as $trigger_name => $info) {
            // Checking trigger type.
            if (!empty($info['type']) && in_array($info['type'], $types)) {
                $triggers[$trigger_name] = $info;
            }
        }
    }

    return $triggers;
}

Below is a example of how the emails could be sent by Email Alerts.

<?php

/**
 * Triggers an Email Alert event.
 * All registered emails that reference the given trigger will be sent.
 */
static function trigger($trigger_id, $vars = array()) {
    foreach (self::getEmailAlerts($trigger_id, $vars) as $email) {
        // The code below is simplified for presentation purposes.
        $mail = new \PHPMailer;
        $mail->addAddress($email['to']);
        $mail->Subject = $email['subject'];
        $mail->Body = $email['body'];
    }
}

/**
 * Gets email alerts for the given trigger, and pipe all the information.
 */
static function getEmailAlerts($trigger_id, $vars = array()) {
    if (!$trigger = self::getTriggerInfo($trigger_id)) {
        return;
    }

    // Get raw (non piped) email alerts for the given trigger.
    $emails = self::getRawEmailAlerts($trigger_id);

    // Apply the traditional REDCap Piping if this is an entry record context.
    if ($trigger['context'] == 'record' && !empty($_GET['id']) && !empty($_GET['event_id'])) {
        // Piping time!
        foreach ($emails as $i => $email) {
            foreach ($email as $key => $value) {
                $emails[$i][$key] = \Piping::replaceVariablesInLabel($value, $_GET['id'] . $entry_num, $_GET['event_id'], $_GET['instance'], array(), false, null, false);
            }
        }
    }

    // Apply generic piping if custom vars are available.
    if ($vars) {
        // Piping time!
        foreach ($emails as $i => $email) {
             foreach ($email as $key => $value) {
                $emails[$i][$key] = self::genericPiping($value, $vars);
            }
        }
    }

    return $emails;
}

/**
 * Gets info of a given trigger.
 */
static function getTriggerInfo($trigger_id) {
    $triggers = self:: getAvailableTriggers(null, false);
    if (isset($triggers[$trigger_id])) {
        return $triggers[$trigger_id];
    }

    return false;
}

/**
 * Applies Piping on the given subject string.
 *
 * Example: "Hello, [first_name]!" turns into "Hello, Joe Doe!".
 *
 * @param string $subject
 *   The string be processed.
 * @param array $data
 *   An array of source data. It supports nesting values, which are mapped to the
 *   subject string as nesting square brackets (e.g. [user][first_name]).
 *
 * @return string
 *   The processed string, with the replaced values from source data.
 */
static function genericPiping($subject, $data = array()) {
    preg_match_all('/(\[[^\[]*\])+/', $subject, $matches);

    foreach ($matches[0] as $wildcard) {
        $parts = substr($wildcard, 1, -1);
        $parts = explode('][', $parts);

        $value = '';
        if (count($parts) == 1) {
            // This wildcard has no children.
            if (isset($data[$parts[0]])) {
                $value = $data[$parts[0]];
            }
        }
        else {
            $child = array_shift($parts);
            if (isset($data[$child]) && is_array($data[$child])) {
                // Wildcard with children. Call function recursively.
                $value = self::genericPiping('[' . implode('][', $parts) . ']', $data[$child]);
            }
        }

        // Search and replace.
        $subject = str_replace($wildcard, $value, $subject);
    }

    return $subject;
}

/**
 * Fetches email alerts from the database for the given trigger.
 */
static function getRawEmailAlerts($trigger_id) {
    // TODO: Run SQL query to get email alerts, filtering by trigger.
}

Final comments

I know this would be a huge shift, but if the maintainers are open to move this project towards this direction, I'd be glad to work on it and submit a PR. It would be less work for me than building all of this functionality from scratch.

Anyway, your input and ideas would be really appreciated.

Thank you!

[knil-maloon]

Hi @tbembersimeao I was reading through your proposal and I think it is very interesting and I see people wanting to send emails on different levels than by project/record.

While checking on the proposal, the only thing I see is, for the second link in the control center, to go to the same code without duplicating it as they are going to be almost identical. I'm not sure if this was your idea but just wanted to point that out.

Also, how about if a project has the email alerts enabled, add a second tab named "Global Alerts" only for admins to see and maybe manage those global alerts? I thought it might be useful for admins managing alerts on a project to quickly see the global ones activated.

[tbembersimeao]

@knil-maloon thanks for your quick reply!

Answering your points:

I'm not sure if this was your idea but just wanted to point it out.

Yes, that's what I meant. My idea is having a common template that renders the config page according to the the given context (e.g. checks whether PROJECT_ID constant is defined).

Also, how about if a project has the email alerts enabled, add a second tab named "Global Alerts"

I think that would be great! A simpler alternative to that would be displaying a link to the Control Center's page ("Manage global alerts here").

I think it is very interesting and I see people wanting to send emails on different levels than by project/record

That's great! How do want to proceed? Do you want to coordinate the implementation from the beginning or you want me to create a PR so you review it from that point? Either way works for me.

[knil-maloon]

Feel free to make the changes, create a pull request and we will review it :)

[knil-maloon]

Hi @tbembersimeao just letting you know that I made a big update on the email alerts that will allow to schedule emails. We're just going to test it now so it's still not ready to go but I thought I should mention it :)

[tbembersimeao]

@knil-maloon thanks for letting me know!

[knil-maloon]

@tbembersimeao no problem. I will let you know once it's finished

[knil-maloon]

@tbembersimeao we just released the version 2.0 with the scheduled emails.