No way to set webjob logging retention policy

[sandeep-at-airlift]

Per this conversation in stackoverflow there is no way to set retention policy for different webjobs.

I have a frequently triggered webjob whose output cannot be indexed in time for me to view logs from other functions on the https://xxxx.scm.azurewebsites.net/azurejobs/#/functions page.

The page seems to be stuck trying to index the output from the frequently run job making it impossible for me to view output from the other jobs

[mathewc]

Are you asking for the ability to turn of logging/indexing for such a frequently executing job?

In addition to a frequent scheduled job, if you have a very high throughput queue triggered job, you'd see similar indexing issues.

[rustd]

The ask here is to delete logs older than 30 days or something similar. We had a backlog item for the Dashboard. I think we should try to get this in for the 1.1 timeframe. It's a dashboard only fix. We can use the same retention polices for Logging exposed in Antares portal.

[mathewc]

I definitely think retention policy would help quite a bit, but for very frequently running functions, I you'll still have the same problem - when you go to the Dashboard, there'll always be a bunch of indexing to perform, and your "recent invocations" will all be for the frequent function.

[rustd]

yeah, retention won't solve the issue with frequently running function. Those are separate issues.

[sandeep-at-airlift]

I think if we had the ability to set retention policy for job, it would work for me.

For the frequently run jobs I really don’t need more than a few hours of logging.

Hope that helps with the prioritization of this work.

Thanks,

Sandeep

From: Pranav Rastogi [mailto:notifications@github.com] Sent: Friday, August 14, 2015 11:14 AM To: Azure/azure-webjobs-sdk azure-webjobs-sdk@noreply.github.com Cc: sandeep-at-airlift sandeep@viaairlift.com Subject: Re: [azure-webjobs-sdk] No way to set webjob logging retention policy (#560)

yeah, retention won't solve the issue with frequently running function. Those are separate issues.

— Reply to this email directly or view it on GitHub https://github.com/Azure/azure-webjobs-sdk/issues/560#issuecomment-131198621 . https://github.com/notifications/beacon/ALZrD08gNbJxCJ0clqAS_wdn1eT0ipjEks5onid0gaJpZM4FpOV4.gif

[rustd]

Scenarios

• Using dashboard with high throughput triggers such as dashboard
• log retention policy -- Delete logs older than X time per storage account -- Delete logs older than X time per WebJob

[mathewc]

Will be enabled by https://github.com/Azure/azure-webjobs-sdk/issues/684

[kbrimington]

Hey, team:

684 was closed last October as part of the 2.0.0 preview. Does this mean we now have the ability to configure retention policies for webjobs? I cannot find information online describing how to do so.

It is becoming an increasing concern to my company, as the storage consumption of webjob logs appears to grow without bound.

Thanks!

[mathewc]

We are reducing our investments in the Dashboard going forward. The recommendation for logging going forward is to use Application Insights https://github.com/Azure/azure-webjobs-sdk/wiki/Application-Insights-Integration.

[shawndewet]

Can I improve the speed of indexing if I scale out my app service? I'm thinking to allow for a catch-up, to perhaps scale out to, say 20 instances, and leave it like that overnight, and scale back again in the morning)

[jcbrooks92]

Hi @shawndewet a little late to be commenting but yes scaling up (not sure about out) will help. https://github.com/Azure/azure-webjobs-sdk/issues/1050#issuecomment-359009675