CSP Usage webjob throwing exception Microsoft.Rest.HttpOperationException

[ghost]

Hello Everyone, I am trying to deploy Peek in our environment and use it to abstract the usage data for different use cases like Enterprise Agreement, CSP. EA is working fine for us, however, we are facing issue while collecting the data for CSP. I have tried to increase the timeout but no luck. We are using the latest merge.

Suggestions to troubleshoot the problem are highly appreciated.

The log output is as follows: _[04/30/2017 09:01:00 > a84fe7: INFO] Starting job processing for Customer Type csp [04/30/2017 09:01:00 > a84fe7: INFO] Starting CSP Routine. Current Usage, Historic Usage and Historic Billing data will be updated.. [04/30/2017 09:01:00 > a84fe7: INFO] [04/30/2017 09:01:00 > a84fe7: INFO] Fetching records for Current Month's Usage from the API. This may take a while. If this operation is timimg out, consider increasing the TimeOut limit in Configuration file. [04/30/2017 09:01:00 > a84fe7: INFO] [04/30/2017 09:01:00 > a84fe7: INFO] Error encountered: [04/30/2017 09:01:00 > a84fe7: INFO] One or more errors occurred. [04/30/2017 09:01:00 > a84fe7: INFO] InnerException : [04/30/2017 09:01:00 > a84fe7: INFO] Exception of type 'Microsoft.Rest.HttpOperationException`1[System.Object]' was thrown. [04/30/2017 09:01:00 > a84fe7: INFO] StackTrace : [04/30/2017 09:01:00 > a84fe7: INFO] at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) [04/30/2017 09:01:00 > a84fe7: INFO] at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) [04/30/2017 09:01:00 > a84fe7: INFO] at System.Threading.Tasks.Task1.get_Result() [04/30/2017 09:01:00 > a84fe7: INFO] at BillingWebJob.Helpers.CspDataHelper.StartCspRoutine(Int32& status, Int32& totalRecordsCount, String& blobStorageUri) [04/30/2017 09:01:00 > a84fe7: INFO] at BillingWebJob.Program.CronJob(TimerInfo timer)__

Thanks, Sheel

[UtkarshVIT]

Hi Sheel Is this the complete error log? If yes, then can you run the project locally and see where the issue is happening?

Thanks Utkarsh

[alaniemi]

We are also having the same issue . Webjob tries to fetch data from CSP portal but fails and throws a HTTP exception/timeout error and is unable to transfer any information into database.

One way to bypass the error message is to comment the lines 529-537 in CspDataHelper.cs-file, which will skip the CSP usage data retrieval part. This will allow webjob task to continue and gather Historic billing information and insert data into database. Unfortunately only historic data will be available for powerbi-report, not actual usage data.

Issue #36 looks quite similar and multiple users have reported similar error messages.

[ghost]

Hi Utkarsh,

Thanks for your response. Yes, that is the complete log of error. I have checked running the project locally and on Azure as well, getting same error on both. Execution is getting timed out at below line of code in CspUsage.cs webjob class file.

HttpResponseMessage httpResponse = await this.Client.HttpClient.SendAsync(httpRequest, cancellationToken).ConfigureAwait(false);

Thanks, Sheel

[alaniemi]

Hi @UtkarshVIT , is there any way to modify that http request or pull usage information in another way? Have you managed to identify whats causing the http request to fail ?

[UtkarshVIT]

@alaniemi @sheelgohe You can verify if the CSP Usage API is timing out by calling it from the swagger interface as given in the deployment steps of the API project. If that is the case then we have implemented the APIs as async in the usageAsync branch. Can you try if that implementation of usage API is not timing out?

[ghost]

Hi @UtkarshVIT - CSP Usage API is timing out when called from Swagger interface as well.

I will take a look at usageAsync branch. and let you know.

Thanks, Sheel

[alaniemi]

Swagger/cspusage does indeed timeout . The request timed out.- The web server failed to respond within the specified time. Will try to redploy solution from async branch

[alaniemi]

@UtkarshVIT Redeployed solution from async branch and swagger now returns normal response. Webjob also runs without any errors.

Although we can now finally get usage data into database, webjob is only fetching partial data; database only contains usage data for companies that have a primary domain name starting with letter A e.g. aaaa.onmicrosoft.com .

Is there any easy way to log more detailed info about the API operations and write that to webjob console? How does one print full customer list fetched from the API to the console?

[UtkarshVIT]

@alaniemi glad to hear that it worked :) So we are currently accessing the custom made APIs which are define in the API Project. To access customer list we can utilize the customers namespace of the Partner Center SDK as given in this sample app.

[alaniemi]

@UtkarshVIT Solution deployed from the async branch has been running for multiple days. Webjob finished without any errors, but fetches only 198 records to usage table.

Same 198 records everyday. No changes whatsoever. Was expecting thousands of records and changes everyday. Something clearly isn't working properly.

How does that API request actually work? Does it try to fetch multiple pages of data and is only able to parse one page? Why are we only getting partial data to database?

How does one troubleshoot the issue and paste fetched information to console or storage log file in more detail?

In order to analyze the issue, would probably need to output the object info to the console and check if the actual list is longer than that 198 lines of data or if there is any error/timeout message somewhere

There was also discussion earlier about releasing newer version of the solution with utilization implementation. What's the status of the utilization branch? Could you please provide us an update on the status of the development work ?

[ghost]

Hi @UtkarshVIT - Any luck with identifying what's causing below CSPUsage http request to timeout?

CspUsage.cs: HttpResponseMessage httpResponse = await this.Client.HttpClient.SendAsync(httpRequest, cancellationToken).ConfigureAwait(false);

Thanks,

[UtkarshVIT]

@sheelgohe if you call the API from swagger UI, you'll see that it returns a timeout after approximately 4 minutes of execution. Any web service hosted on azure has a maximum execution time of 4 minutes after which it return a timeout error. The Usage API fetches the maximum amount of data and hence takes the maximum amount of time to process the request. Thus, if the partner has large amounts of usage data, the API may take over 4 minutes to respond and hence the timeout error.

@alaniemi The code for usage controller is here It iterates through each subscription for each customer of the partner. You can put a breakpoint on line 119 of the controller and see what is being returned.

[ingramleedy]

@UtkarshVIT has this fix been incorporated into the main branch?

[townendk]

@UtkarshVIT I'm using the async branch and I'm still running into this error unfortunately. Are there any other things I should be trying?

Many thanks