Feature Request: Improve scheduling?

[janboddez]

What

I was simply wondering; if you had a quite a few followers on different servers, that means a lot of POST requests are made for every new post.

These are made in a single WP-Cron call:

https://github.com/Automattic/wordpress-activitypub/blob/efd98acd0bf3f2c42f0ffc3d8f10920e74cb8512/includes/class-activity-dispatcher.php#L82-L84

Now I recently learned WP-Cron runs asynchronously even when WP_DISABLE_CRON is not set, and is normally triggered from a cron job when it is set.

The latter hopefully also means a longer (or even infinite) time-out limit.

Still, I wonder if it would be possible to either "spread these out" in time a bit more, make them ... less demanding.

Why

I imagine it can be quite taxing on smaller servers to send out a few dozen HTTP requests.

Note that I've only ever experienced real issues with incoming requests, after moving to an unmanaged VPS at least. (My previous (shared) web host seemed to have trouble even with outgoing requests.)

But I don't have very many followers either. :-P

How

It might be possible to either "slice" the $inboxes array and schedule only, e.g., 5 requests at a time, moving the remaining requests forward by a minute or so until they've all run. This might require keeping track of which inboxes were already "processed"---which would be a nice feature anyway, to allow for retries? Or the processed ones could simply be dropped from the array and the remainder could be passed onto the callback as an additional argument. Or something like that. :-D

Or allowing a true "job queue" or something to process them. (I guess a custom plugin could already replace the activitypub_send_activity callback with something "more efficient," or rather, less demanding. Based on Redis or whatever.)

Note that this likely creates race conditions too, i.e., the callback, when it eventually runs, should again verify that the post wasn't deleted or made "private," etc., in the meantime.

But: such an additional verification step might at one point anyhow become a necessity. In fact, the plugin seems to already be running one such check: https://github.com/Automattic/wordpress-activitypub/blob/efd98acd0bf3f2c42f0ffc3d8f10920e74cb8512/includes/class-activity-dispatcher.php#L64C8-L64C49. (I'd say even more such checks are in order. But that's a different matter.)

Come to think of it, if I had WP_DISABLE_CRON set right now and only ran cron once every 10 minutes or so, I'd have the same potential issue (that the post's visibility changed).

[janboddez]

If I'm overthinking this, please ignore. Or let me know, then I can forget about this. :-D

One possible, untested idea for the "slice thing":

public static function send_activity( WP_Post $wp_post, $type, $inboxes = array() ) {
    if ( is_user_disabled( $wp_post->post_author ) ) {
        return;
    }

    $object = Post::transform( $wp_post )->to_object();

    $activity = new Activity();
    $activity->set_type( $type );
    $activity->set_object( $object );

    if ( empty( $inboxes ) ) {
        $follower_inboxes  = Followers::get_inboxes( $wp_post->post_author );
        $mentioned_inboxes = Mention::get_inboxes( $activity->get_cc() );

        $inboxes = array_merge( $follower_inboxes, $mentioned_inboxes );
        $inboxes = array_unique( $inboxes );
    }

    $now = array_splice( $inboxes, 0, 3 );

    $json = $activity->to_json();

    foreach ( $now as $inbox ) { // Post to the first three inboxes.
        safe_remote_post( $inbox, $json, $wp_post->post_author );
    }

    if ( ! empty( $inboxes ) ) { // Schedule the rest later on.
        wp_schedule_single_event( time() + 30, 'activitypub_send_activity', array( $post, $type, $inboxes ) );
    }
}

The 3 could obviously also be 5 or 10, or 50 ...

[pfefferle]

Thanks for the suggestion, this should be better that it is now, but I am also working on an outbox custom post type. with more and more CRUD for more than Posts (Profiles, Comments, ...) this should help with the distribution and it will be simpler to send it in chunks when you have a real persistence and not only a scheduled object.

I hope to release a draft soon, to discuss this in more detail.

[baby-gnu]

Hello, I'm not sure it's the good place for my comment but I would like to be able to spread the toot submitted from wordpress.

Our use case is the following:

• we recently publish our magazine on our website(fr)
• the magazine is published all pages (~20) in bulk edit (35 pages)

I would prefer to push them on fediverse one by one every 15 or 30 minutes (maybe configurable?).

Thanks.

[janboddez]

Possibly related: how about rescheduling failed POST requests? (See also #559.)

I was thinking of something like this (in a separate plugin, for now), but never got around to it:

add_action( 'activitypub_safe_remote_post_response', function ( $response, $url, $body ) {
    if ( is_wp_error( $response ) || wp_remote_retrieve_response_code( $response ) >= 400 ) {
        // Reschedule?
        // Code to reschedule. ;-P
    }
}, 10, 3 );

[janboddez]

Just happened to come across https://github.com/deliciousbrains/wp-background-processing.

The accompanying article happens to mention a similar approach to what I'd described earlier (using wp_schedule_single_event() + a fixed batch size), but a "smarter" queuing system that decides on things like batch size for us is probably the better option.

[github-actions[bot]]

This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[janboddez]

Possibly related: how about rescheduling failed POST requests? (See also #559.)

I was thinking of something like this (in a separate plugin, for now), but never got around to it ...

Just got reminded of this issue ... I actually did implement something like this: https://github.com/janboddez/addon-for-activitypub/blob/main/includes/class-reschedule-requests.php

It currently seems to trigger on instances that are down and have been for a while only, so that's good I guess. :-)

[github-actions[bot]]

This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[pfefferle]

Keep