mailchimp-api Batch execute returning null - timeout?

[crmpicco]

I am using your mailchimp-api package in my application and I am utilise a large sync process which takes 56,000 user records from my database, batches it and sends it through to Mailchimp. My code:

$batch = $this->mailchimp->new_batch();
foreach ($subscribedUsers as $subscribedUser) { // 56k array
     $batch->put('batch'.$subscribedUser->getId(), $endpoint, [
         'email_address' => $email,
         'status_if_new' => 'subscribed',
         'merge_fields' => $mergeFields,
     ]);
}
$result = $batch->execute(); // this returned null
$batch = $this->mailchimp->new_batch($result['id']); // the script fell over here
$batch->check_status();

Here is some output from my script which shows that it took 522 seconds (8 mins) from creating the batch to queueing up the batch with 56785 records to executing it.

Created Mailchimp batch : timeSeconds=0.00026 Batched all Mailchimp users : timeSeconds=522.63815 Sent Mailchimp batch : timeSeconds=10.17836 Synced all Mailchimp user records : noOfPersons=56785 : executionTime=538.97

The batch appears to have made it through to Mailchimp as over the course of the next hour after running the script my mailing list was gradually updated every minute to reflect the changes I sent through.

What I am trying to understand is why did the call to execute return null when it should return an array which includes the all important batch id.

I notice there is a $timeout parameter to the execute method - would increasing this to something high, say 1000 seconds, solve this issue?

[RobWiddick]

You don't need this line $batch = $this->mailchimp->new_batch($result['id']); // the script fell over here if you're checking on the status of the batch previously posted. See this which shows the previously-set batch_id property being used. Oddly enough, however, $batch->execute() should be returning something... I would wonder if this is more of a cURL issue...

[crmpicco]

The call to new_batch isn't the problem here, it's the fact that execute returns null when it shouldn't. I've boosted the timeout parameter, but i'd like to know if that is likely to fix it in future runs.

[drewm]

That's still a very big batch. Does a smaller batch work? It does sound like you're hitting into configured or available resource limits.

[crmpicco]

Yes, smaller batches work with no error and execute returns as expected. Where would the configured/available resource limits be? In the PHP app? cURL? In the Mailchimp bundle?

[drewm]

Yes, could be any of those. This library only implements a cURL timeout option, which is configurable. Other than that, you could be hitting script timeout limits, memory limits, anything. That's not easy for someone without access to your server to investigate.

[justingivens]

@crmpicco as @drewm said you need to break up your batches into smaller ones.

Anything over 5000 operations might fail. That came straight from MailChimp. I tried to batch over 10k operations and received the null back.

At the end my foreach, I broke it down into small batches like this:

if( $counter === 500 ) {
    $counter = 0;
    $result = $batch->execute();
    $batch = $this->mailchimp->new_batch();
}
$counter++;

$result now contains the correct response data.

[paganm]

I have the same problem - curl calls returns null (at method makeRequest in Mailchimp class, $responseContent = curl_exec($ch); $responseContent is empty) - even if i split the batch op to contain only 400 ops per batch. And every batch is executed when previus is finished. That may occur not only in batch ops but to any curl call to mailchimp api.

I have asked the same question in [https://stackoverflow.com/questions/62784896/mailchimp-api-problem-with-dropped-curl-calls] where is explained the situation.

I cannot find any reason why that happens.

foreach($accounts as $idx => $email) { $members[] = $email; if($idx > 0 && (($idx % 400) == 0 || count($accounts) - 1 == $idx)){ if( ($status = $this->_addMembersInAList($listID, $members)) === 1) { $transfered += count($members); $this->updateProcessStatus($pid); $members = []; }else{ return $this->updateProcessStatus($pid, $api->getLastError(), true); } } }

where _addMemebersInAList is $batch = $api->new_batch(); foreach($members as $idx => $email){ $batch->post("m".$idx, "lists/$listID/members", [ 'email_address' => $email, 'status' => 'subscribed', ]); `} ` $batch_status = $batch->execute(3000); if($batch_status === false){ return 0; } while(1){ $status = $batch->check_status(); if(isset($status["status"]) && !empty($status["status"])){ if($status["status"] === "finished"){ return 1; }else{ sleep(1); } }else{ return 0; } }

It does not hit script timeout, memory limits or anything else.. One day may some calls returning null and the next day may run smooth, until i try to create a campaign with several emails (more than 20K for example).

Any thoughts? I have stuck here