search

Azure-Samples / azure-batch-samples

Azure Batch and HPC Code Samples
Other
260 stars 488 forks source link

No job statistics in BatchExplorer #131

Closed wirehell closed 8 years ago

wirehell commented 8 years ago

There is no statistics for jobs. They are visible for individual tasks though.

matthchr commented 8 years ago

Hi @wirehell can you expand more on this? Where are you looking for the JobStats and not seeing them? Have you tried turning on "use stats during list" in the options menu?

wirehell commented 8 years ago

For tasks the statistics is there: taskstatistics But not for jobs: jobstatistics

matthchr commented 8 years ago

@wirehell Are you refreshing your resources after enabling the "use stats during list" (either by refreshing the individual item or by refreshing the list of jobs?). If you do not refresh after checking the "use stats during list" box then the statistics don't show up.

Also, JobStatistics takes a bit to aggregate, you won't see it immediately after the first task in the job is run, it will take some time (I think 15m).

I looked into this a bit more on our server though and it seems your job doesn't have stats even now. We are looking into why that is.

matthchr commented 8 years ago

@wirehell Sorry for the long response -- there was a bug in the service where some jobs were not having statistics aggregated for them correctly (so you would see task level statistics but not job level statistics).

A fix has been rolled out, so all new jobs statistics will work (new jobs should have statistics). There is no issue in BatchExplorer, this was an issue with the underlying service.

wirehell commented 8 years ago

Ah, I see. Thanks.

Malondron commented 8 years ago

Hi,

the issue seems to persist. Both using the BatchExplorer and the usual batch rest API, the tasks have statistics, but the jobs don't.

nmontmarquette commented 7 years ago

@matthchr Probably not directly related to the original issue, how do you get statistics for jobs or tasks? I am using the Python API. When querying jobs with the list method, I feel like you have to supply something in the JobListOptions in the expand parameter. I have some hard time finding up-to-date documentation on that.

darylmsft commented 7 years ago

We will get you a clear answer in Python, but because it is late on a Friday I will add enough non-Python information in the hopes you can be unblocked. The <job, task> statistics are considered "associated entities" and such entities are more expensive to fetch and, thus, are not returned by default. In the context of C# this is discussed here. The REST API for ListJobs shows the expand clause here.

These sources implicate the JobListOptions in Python (with expand set to stats).

We will get you a sample fragment in Python to show how this is done but until then hopefully the above will help.

annatisch commented 7 years ago

Hi @nmontmarquette, In the Python SDK this can be done with the JobGetOptions (or taskGetOptions) where you can set the expand clause that @darylmsft described:

options = batch.models.JobGetOptions(expand='stats')
job = batch_client.job.get(job.id, job_get_options=options)
print(job.stats)

The same syntax is used for the select clause. If you want to select or expand multiple attributes, you can specify comma-separated values:

options = batch.models.JobGetOptions(expand='stats', select='id,state')
job = batch_client.job.get(job.id, job_get_options=options)