Problem with feed-request per batch

[iparker]

Hello,

sorry for duplicate this - I already wrote about my problem at https://github.com/ChrisRAoW/mautic-rss-to-email-bundle/issues/42 but this issue is closed.

We use your rss-plugin very intensive to output different dynamic data in the emails.

The last thing we created was an rss-feed which generates a individual coupon code for each receiver. We realized this by an api-endpoint (in our system) which creates a new bonus-code and returns this code as rss-feed so we can output the code with your plugin. This endpoint get id-data from the contact.

Unfortunately we mentioned that we have very less new bonus code entries in our database - and with that I remembered the issue https://github.com/ChrisRAoW/mautic-rss-to-email-bundle/issues/42 and your comment with the batch-size.

I think this is the problem: In our usage/use case we expect that every single email will make an request to the feed url and output this individual data.

But this is not how it works because the feed url is just requested once for every batch size.

Do you have any idea how we can solve this and change this behaviour?

I found following comment in Parser.php:

// Replace tokens is only for e-mail send via the API. 
// The feed is only parsed once a batch. 
// So for multiple e-mails it will inconsistent results.
$feedUrl = $this->replaceTokens($this->getParam('url'));

Is there any way to make this call for every receiver? Or something like to change the batch-size to 1? Or do we need to subscribe to another event to have this behaviour for every single email?

Thanks for some ideas and help!

Best regards

Timo

[ChrisRAoW]

HI @iparker,

Nice to hear you are using the plugin intensively.

I think I can enable the replacement of the feed_tags on every individual e-mail based on a parameter. This then should be handled with care. If you have a list of 10000 subscribers, it will request the feed 10000 times, which can bring your server down if it is an heavy request.

Chris

[iparker]

Hi Chris,

thanks for your quick reply!

Yes you are right - the number of requests will heavily increase. But I think this is necessary for the way we want to use it. We have to monitor the performance/load of the server after the change.

It would be great if you can change this!

I think I can enable the replacement of the feed_tags on every individual e-mail based on a parameter.

Do you mean that it is possible to set a parameter next to the feed url so one feed is handled per batch and another feed is handled for each subscriber? Or is this more an global system parameter?

Thanks for your help!

Best regards,

Timo

[ChrisRAoW]

@iparker Already checked, I think i can implement this pretty easy. Will do it later today.

You will be able to set the parameter next to the feed url.

[iparker]

Great - thanks a lot!!!

[ChrisRAoW]

@iparker Can you try this feature-branche?

https://github.com/ChrisRAoW/mautic-rss-to-email-bundle/tree/feature/batch-mode

Documentation is in the readMe. Long story short, add batch="0" next to the url in the feed tag.

[iparker]

Thanks a lot for your quick fix!

I think the batch="0" is working - with this I have different output in the emails for different receivers.

But there is one wrong output in the feed-content.

I think the reason for this is the following line of code:

$feedParserContent = uniqid() . ' - ' . $feedParserContent;

There is this uniqid-Output before the feed content.

Maybe you can fix this?

Thanks a lot!

Timo

[ChrisRAoW]

@iparker whoops...was piece of debug code. Is removed now from the branch. I will merge in master too

[iparker]

Yes I think so. Again thanks a lot for your help!

So will you merge this batch-mode into master branch?

[ChrisRAoW]

@iparker nice addition the to plugin.

Already merged.

[iparker]

I'm exited to test the change tomorrow with an email to 20.000 receivers ;-)

[ChrisRAoW]

haha, let me know.

[iparker]

Hi Chris,

the send process is still running but I want to give you a first feedback about the performance.

Unfortunately the performance is very bad.

The email yesterday (with batch mode) handed about 1.000 in 2m30s. Now without the batch mode it takes about 30m for 1.000 emails.

So our use case and the way we want to output individual data based on rss feeds in mautic emails is not performant and it seems so slowly that we have to think about alternative approaches.

We have some ideas to improve the performance, for example to reduce the rss-content to the really necessary data. For example we have a long description field in the feed which is not in output and with that not necessary. But I think this will not make a big difference.

When I request the rss-feed directly for a single user in postman it takes about 180ms.

The load factor on the different server is not very high while the send process is running, so I think hardware-performance is not the main problem.

We send email in mautic in queue-mode with a queue-size of 500, a time limit of 60s and the send process is handled all 2m via cron.

Do you have any other ideas how we can speed up the send process in our (damned) use case?

Best regards,

Timo

[iparker]

Just reopened this issue because I don't know if you see my comment for the closed issue.

[ChrisRAoW]

Hi Timo,

This is exactly the reason why we didn't implemented this in the first place. I maybe have some ideas to optimize the process on the plugin side. On the other hand you could try to optimize the feed. If it would take 90ms in stead of 180 ms to request the feed, that would also help of course.

I will keep you posted on my findings.

Chris

[iparker]

Hi Chris,

we just reduced the feed-content which comes with a better response time (-50ms) and a much smaller request size (-75% size) but the performance is still the same.

I think we have to look for another approach for our use case, somethink like storing the feed-data as json-string in custom fields and output the json-data in the email with twig template extension.

I don't know if this will work but I think I have to try it.

Best regards,

Timo

[paulpiazza]

Hi, is there any news about this feature?

[ChrisRAoW]

@paulpiazza Unfortunatly i don't. I will ask my collegue to check it out next week.

In your case without batchmode it requires approx. 28min more with 1000 e-mails. That is 1,68sec per mail. For this 0,18s is fetching the new feed. So we are looking at an overhead of 1,5sec per e-mail without matchmode.

Hopefully we are able to find something to optimize.

[paulpiazza]

Thanks @ChrisRAoW