bogdanst24 commented 1 week ago

Provide required information needed to triage your issue

For some users of our Event-based Outlook Web Add-in, the Fetch/XMLHttpRequest requests are failing with "A connection with the server could not be established". The triggering events are onMessageCompose and onMessageRecipientsChanged. The requests are made against a storage account using a valid Shared Access Signature. In all of the cases we were able to troubleshoot with the users, they were able to use the Shared Access Signature from the browser directly, so there is no problem with the SAS validity.

This is a problem we see happening randomly for many of our customers. Not all their users, only some. I am not talking about a local environment, but a production one with over 100k active users. General/obvious issues should be out of the question as everything still works as expected for the majority of users.

Your Environment

Platform [PC desktop, Mac, iOS, Office on the web]: Any
Host [Excel, Word, PowerPoint, etc.]: Outlook
Office version number: Unrelated
Operating System: Any
Browser (if using Office on the web): Any

Expected behavior

GET requests are working in all event-based actions.

Current behavior

GET requests are sometimes failing with [TypeError: Network request failed].

Steps to reproduce

Cannot reproduce locally without involving fiddler. In this case, after allowing fiddler to capture the traffic, it only works with it. However this might be a local issue as the local infrastructure is more comples.

Provide additional details

We have an automatic retrial mechanism on requests that have XMLHttpRequest as the last retry for gathering information. The request-response is the one shared above - "A connection with the server could not be established". I've been in several troubleshooting calls trying to figure it out and I was never able to put the finger on the issue. My observations:

accessing the URLs directly from the browser always works
users using a VPN or a proxy might be exposed to this issue, but it is not a rule
sometimes asking users to temporarily change the network to something like a mobile network hotspot, solves the issue - though this is not a fix, and it doesn't always work
the issue does not seem to be linked to a specific email client or device, as we can see it reproduced in our logs with Windows, macOS, Linux, iOS, and Android. It is mostly visible in Outlook Window, the legacy client, but that's also where most of our usage still comes from.

Fetch request error:

"stack":"TypeError: Network request failed\n at Anonymous function (Unknown script code:83:8045)\n at Anonymous function (Unknown script code:60:1235)\n at k (Unknown script code:60:498)\n at I.callTimers (Unknown script code:60:2655)\n at value (Unknown script code:41:3733)\n at Anonymous function (Unknown script code:41:660)\n at value (Unknown script code:41:2510)","message":"Network request failed"

 SignatureInsertionError: HTTP error 403: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

XMLHttpRequest error:

 XHR error: ', { UNSENT: 0,
  OPENED: 1,
  HEADERS_RECEIVED: 2,
  LOADING: 3,
  DONE: 4,
  readyState: 4,
  status: 0,
  timeout: 0,
  withCredentials: true,
  upload: {},
  _aborted: false,
  _hasError: true,
  _method: 'GET',
  _perfKey: 'network_XMLHttpRequest_https://VALIDSAS',
  _response: 'Error code = 15\n',
  _url: 'https://VALIDSAS',
  _timedOut: false,
  _trackingName: 'unknown',
  _incrementalEvents: false,
  _performanceLogger: 
   { _timespans: 
      { 'network_XMLHttpRequest_https:/VALIDSAS': { startTime: 1708438878302, startExtras: undefined },
        'network_XMLHttpRequest_BACKEND ERROR LOGGER API': { startTime: 1708438878444, startExtras: undefined } },
     _extras: {},
     _points: {},
     _pointExtras: {},
     _closed: false,
     _isLoggingForWebPerformance: false },
  responseHeaders: undefined,
  _requestId: null,
  _cachedResponse: undefined,
  _headers: { accept: 'application/json' },
  _responseType: '',
  _sent: true,
  _lowerCaseResponseHeaders: {},
  _subscriptions: [] }

pepso commented 1 week ago

We see below error locally and through centrally deployed LaunchEvent add-ins on Classic Outlook (2409): "stack":"TypeError: Network request failed\n at Anonymous function....","message":"Network request failed"

For us it is transient in nature and usually resolves itself in a restart or two after the Add-in bundle.js has been updated. This error have been present for ~1-2 months now.

Easiest to replicate this on local (for us) has been: 1) Have Outlook running in a way that you have triggered the Add-ins (we have local, dev, and production so have 3) 2) Update the add-in code in the WEF cache folder (for one of them is enough) 3) Restart Outlook to reload the bundle.js 4) See you get Network errors from the logs 5) Restart Outlook few times (while checking logs if the network errors are gone) 6) Eventually you will have all add-ins with network connectivity and the problem disappears

We have seen this problem appearing when: 1) Update the add-in bundle.js via file system copy 2) Update the add-in bundle.js via cache-expiry from the hosted address associated with manifest.xml

@bogdanst24 does above seem like a plausible scenario to you?

bogdanst24 commented 1 week ago

It could be plausible, but from what I can tell the issue is not only limited to Classic Outlook. I remember solving some customer issues in the past by simply removing the WEF cache folder and so forcing Outlook to fetch everything.

neprasad-microsoft commented 1 week ago

Hi @bogdanst24 Do you see this issue on New Outlook and Outlook Web also? Also, please capture logs and share with us, that will help us to debug further - https://github.com/OfficeDev/office-js/wiki/Collecting-log-files-for-Outlook-Add%E2%80%90ins-issues .

bogdanst24 commented 6 days ago

Do you see this issue on New Outlook and Outlook Web also?

Yes, we had such reports and we can see such failures in our logs from non desktop clients. I don't currently have a user I can get in a call with to collect the logs but I'll try to find one. What I remember from previous troubleshooting sessions, is that the requests are never registered in the network tab.

What about the Outlook Desktop?

neprasad-microsoft commented 6 days ago

Hi @bogdanst24 Can you help us with the errors on all platforms? Is it the same error everywhere?

bogdanst24 commented 4 days ago

Yes, that is the error _response: 'Error code = 15\n',

OfficeDev / office-js