search

ServiceNow / sncicd-apply-changes

MIT License
12 stars 6 forks source link

apply-changes fails with "maximum execution time exceeded error" #8

Closed ramyaramdasan closed 3 years ago

ramyaramdasan commented 3 years ago

Hi @chiarnglin ,

Hope you are safe and doing well!

We face an issue with sncicd-apply-changes action where it fails within a few seconds after the action starts to run. It occurs intermittently and we could not relate it to any platform changes. I see the following message in the instance logs: _Cancelling transaction #20151 /api/sn_cicd/sc/apply_changes (maximum execution time exceeded): Thread APIINT-thread-3 (null, 44D65168DB5870102CE3172505961970), after 314838ms

Are there any timeout values set for the actions? Would it be possible to get a list of exceptions, their causes and resolutions for all the ServiceNow GitHub Actions which we can refer when we come across similar issues? Apply Changes

Thanks Ramya

chiarng commented 3 years ago

Hi Rayma, Hope you and your family are safe as well, we're lucky to have enough vaccines here in the US. This issue is new to me. What it sounds like is there's two levels of timeouts.

  1. The instance has a timeout for API calls and responses. I searched up some posts on Google, eventually got to https://docs.servicenow.com/bundle/quebec-platform-administration/page/administer/platform-performance/concept/c_DefaultQuotaRules.html. It doesn't look like the roughly 5 minutes in your example though, but that log message sounds like the instance was the one that cancelled the transaction for the API call the GitHub Action was using.
  2. There's a sleep function with a default timeout of 3000ms defined, but I think it's only used to rate limit the polling of the responses, and I don't see a cap in the number of retries. https://github.com/ServiceNow/sncicd-apply-changes/blob/master/src/App.ts#L17.

My guess is this is on the instance side. Something is going on, on the instance, and it's basically not completing the Apply Remote Changes operation in a timely manner. Is it because there's too many apps trying to Apply Remote Changes at once hitting a thread limit?

Thanks, Chiarng

ramyaramdasan commented 3 years ago

Hi @chiarnglin,

Thank you for the quick response. We just have two applications using the pipeline currently and both are less critical apps. They don't go through a lot of changes, so the deployments are less frequent. And like you said, the DefaultQuotaRule settings are not set to a value so low that the workflow fails in less than 30s from the trigger time. Please let me know if there is anything else that I could check on the instance.

Thanks Ramya

chiarng commented 3 years ago

@ramyaramdasan You have a good point, I would like to find out if there's a rate or thread count limit too on the performance side. I've asked our engineering team to help take a look, but to start with - is this on one of the DPP instances or one of the Shell instances? If it's the latter, we might need you to go help open a CS with support to start the conversation (so we can have HIHOP access into your instance via that support case). Feel free to post the case link here if you do go create one!

ramyaramdasan commented 3 years ago

Hi @chiarnglin,

We encountered this issue on the Shell Instance. I just wanted to get clarity on whether it was an issue with GitHub or instance. Now that you said it was most likely an instance related issue, I could create a HI request and have the support team investigate it. Thanks for the suggestion.

You take care!