search

OfficeDev / ews-java-api

A java client library to access Exchange web services. The API works against Office 365 Exchange Online as well as on premises Exchange.
MIT License
869 stars 560 forks source link

Perpetual 401(Unauthorized) after running fine for period of time #560

Open montehansen opened 8 years ago

montehansen commented 8 years ago

I have built a conference scheduling app that uses ews-java-api to browse and book meetings.

The app will run fine until a certain number of requests and mid-stream the Exchange EWS service will return only 401(Unauthorized). The only way to reset this state is to start the application again.

No ExchangeService instances are cached, that is, each request stream creates a new instance of ExchangeService. The request stream is basically fetching calendar items by impersonating the application user. I get the same behavior with both the Sept 2015 release and latest bits from the head.

Any ideas would be appreciated.

Monte

code to create a service instance for an impersonated request:

    // Create the EWS service instance.
    ExchangeService svc = new ExchangeService( ExchangeVersion.Exchange2010_SP2 /*server is actually Ex2013*/ );
    svc.setUrl( exchangeewsuri );
    svc.setCredentials( new WebCredentials( Property.getEwsUser(), Property.getEwsPassword() ) );

    // We use the context user to define what user to impersonate.
    String impersonatedaddress = contextusermailbox;

    // Impersonate the context user.
    svc.setImpersonatedUserId( new ImpersonatedUserId( ConnectingIdType.SmtpAddress, impersonatedaddress ) );

    if ( Property.getEwsTraceEnabled() )
    {
      svc.setTraceEnabled(true);
      svc.setTraceFlags( EnumSet.allOf(TraceFlags.class) );
      svc.setTraceListener( new ITraceListener() 
      {
        public void trace(String traceType, String traceMessage)
        {
          logger.debug( "Type:" + traceType + " Message:" + traceMessage );
        }
      } );
    }

    return svc;
montehansen commented 8 years ago

It's worth noting that I am parallelizing requests across a thread pool for performance reasons.

montehansen commented 8 years ago

I believe this is a result of throttling policies being applied on the Exchange server. I am investigating this further by changing or lifting those values for validation purposes.

Now, if this is so, I would suggest this response to be a failing of either the EWS client (my guess is the .NET version as well), or the EWS soap service. That is, I do not believe it to be appropriate to respond with a 401(Unauthorized) since the response has absolutely nothing to do with authentication. Some might think that the intent could be to be evasive for security purposes, but I think such evasiveness is secondary to the http model itself.

Given such a condition, there are other (appropriate) responses such as 403 Forbidden, or even 418 I'm a teapot =) is more suitable than 401(Unauthorized).

davster commented 8 years ago

Throttling on the exchange side will never result in a 401. Throttling will only result in either a 500 with a SOAP fault or a 503 ServerBusy.

From: montehansen [mailto:notifications@github.com] Sent: Tuesday, September 20, 2016 10:19 AM To: OfficeDev/ews-java-api ews-java-api@noreply.github.com Subject: Re: [OfficeDev/ews-java-api] Perpetual 401(Unauthorized) after running fine for period of time (#560)

I believe this is a result of throttling policies being applied on the Exchange server. I am investigating this further by changing or lifting those values for validation purposes.

Now, if this is so, I would suggest this response to be a failing of either the EWS client (my guess is the .NET version as well), or the EWS soap service. That is, I do not believe it to be appropriate to respond with a 401(Unauthorized) since the response has absolutely nothing to do with authentication. Some might think that the intent could be to be evasive for security purposes, but I think such evasiveness is secondary to the http model itself.

Given such a condition, there are other (appropriate) responses such as 403 Forbidden, or even 418 I'm a teapot =) is more suitable than 401(Unauthorized).

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgithub.com%2fOfficeDev%2fews-java-api%2fissues%2f560%23issuecomment-248369219&data=02%7c01%7cdavster%40exchange.microsoft.com%7c4e508266df094b9a2e3c08d3e17a261e%7c72f988bf86f141af91ab2d7cd011db47%7c1%7c0%7c636099887151353311&sdata=c63vjeBR%2b7YrU1eUZ0XBwd14g5AtuF%2fImUoXxIPFhNg%3d, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgithub.com%2fnotifications%2funsubscribe-auth%2fAILDSh-phayiXU5HSFhbvEnYt-VffZsOks5qsBVngaJpZM4J_-3-&data=02%7c01%7cdavster%40exchange.microsoft.com%7c4e508266df094b9a2e3c08d3e17a261e%7c72f988bf86f141af91ab2d7cd011db47%7c1%7c0%7c636099887151353311&sdata=a%2bcCe5B3ULmtFtiqpsgXqwwvhLMQoChM%2boXiJ7uiDvE%3d.

montehansen commented 8 years ago

Thanks Dave.

I am attempting to prove or disprove that theory. If not a result of throttling, I would love any thoughts from any in the know. The issue can manifest following a stream of successful requests. I have stepped the java-ews-api source and it is in fact receiving this 401 response. Same result with or without pre-authenticate.

davster commented 8 years ago

I wrote the throttling code in Exchange ☺ Aside from that, 401s can happen for a variety of reasons. Are you calling against O365 or on-prem?

From: montehansen [mailto:notifications@github.com] Sent: Tuesday, September 20, 2016 1:07 PM To: OfficeDev/ews-java-api ews-java-api@noreply.github.com Cc: David Sterling (EXCHANGE) davster@exchange.microsoft.com; Comment comment@noreply.github.com Subject: Re: [OfficeDev/ews-java-api] Perpetual 401(Unauthorized) after running fine for period of time (#560)

Thanks Dave.

I am attempting to prove or disprove that theory. If not a result of throttling, I would love any thoughts from any in the know. The issue can manifest following a stream of successful requests. I have stepped the java-ews-api source and it is in fact receiving this 401 response. Same result with or without pre-authenticate.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgithub.com%2fOfficeDev%2fews-java-api%2fissues%2f560%23issuecomment-248418395&data=02%7c01%7cdavster%40exchange.microsoft.com%7cbf28bbc9014c45b04ca908d3e1919fae%7c72f988bf86f141af91ab2d7cd011db47%7c1%7c0%7c636099987966803352&sdata=H6ZAa5Cz1k%2bu%2bGpxMvdE919%2fVr14gEMMeTqJcwc3Bns%3d, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgithub.com%2fnotifications%2funsubscribe-auth%2fAILDShxi6BzFWFYKie9Pa0SrQ3DQqjOpks5qsDzJgaJpZM4J_-3-&data=02%7c01%7cdavster%40exchange.microsoft.com%7cbf28bbc9014c45b04ca908d3e1919fae%7c72f988bf86f141af91ab2d7cd011db47%7c1%7c0%7c636099987966803352&sdata=8IuEYOFk8%2fUnQ1KP7w%2bxTaBebkMfruVQZOoTJt72bT4%3d.

montehansen commented 8 years ago

Thanks, Dave. I'd like to buy you a beer then =)

I am calling that via on-prem Ex2013. The extent of the questions being asked of Exchange are generally to acquire appointments for a date range (such as a work week) for a group of users, and the conference room. These requests are spread across a thread pool (the attendee questions are impersonated).

Monte

davster commented 8 years ago

Can you get access to the protocol logs on the on-prem server? Even better, the IIS logs. The 401 will often have a subcode and win32 status code that gives further info about why the auth failed. I assume Kerberos/NTLM auth?

From: montehansen [mailto:notifications@github.com] Sent: Tuesday, September 20, 2016 1:28 PM To: OfficeDev/ews-java-api ews-java-api@noreply.github.com Cc: David Sterling (EXCHANGE) davster@exchange.microsoft.com; Comment comment@noreply.github.com Subject: Re: [OfficeDev/ews-java-api] Perpetual 401(Unauthorized) after running fine for period of time (#560)

Thanks, Dave. I'd like to buy you a beer then =)

I am calling that via on-prem Ex2013. The extent of the questions being asked of Exchange are generally to acquire appointments for a date range (such as a work week) for a group of users, and the conference room. These requests are spread across a thread pool (the attendee questions are impersonated).

Monte

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgithub.com%2fOfficeDev%2fews-java-api%2fissues%2f560%23issuecomment-248423762&data=02%7c01%7cdavster%40exchange.microsoft.com%7c56c403d44bf2441115c708d3e1948fad%7c72f988bf86f141af91ab2d7cd011db47%7c1%7c0%7c636100000586266006&sdata=vFMj%2felIli2t9dAUJpx5hH4oZ5dJbWkXI%2f5zP27W6H4%3d, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgithub.com%2fnotifications%2funsubscribe-auth%2fAILDSvyM6Vi2I5tyDrZGjpQFvdOcCYaks5qsEG3gaJpZM4J-3-&data=02%7c01%7cdavster%40exchange.microsoft.com%7c56c403d44bf2441115c708d3e1948fad%7c72f988bf86f141af91ab2d7cd011db47%7c1%7c0%7c636100000586266006&sdata=FoD3NWz88u8Ga6hWyKK3JWAmgza4AFUr5wNZLtQuUig%3d.

montehansen commented 8 years ago

NTLM. Working on those logs...

montehansen commented 8 years ago

I managed to get my hands on some IIS logs that are in the format of HttpProxy. Fortunately, the java EWS client uses a unique user-agent so I was able to spot the http requests, and sure enough it reports some of them as 401 Unauthenticated. Not sure what to make of this because I could reproduce+step on the client and all requests appear authenticated (albeit impersonated). I am attaching an excel export showing some of the traffic. Unfortunately, HttpProxy doesnt include the soap envelope so kinda blind there.

My Exchange admin days are mui rusty and not sure what I can do on the server side to better correlate the client requests.

Thanks so much for your insight.

ExchangeServicesClient.xlsx

davster commented 8 years ago

Those look like Exchange CAFÉ logs (from v15\logging\httpproxy…) rather than IIS logs. If you find one of the 401s from that log, it will show you the server it tried to proxy to. You can then look in the iisLogs folder on that dest server to find the corresponding entries. Can probably find it roughly by datetime.

From: Monte Hansen [mailto:notifications@github.com] Sent: Wednesday, September 21, 2016 10:12 AM To: OfficeDev/ews-java-api ews-java-api@noreply.github.com Cc: David Sterling (EXCHANGE) davster@exchange.microsoft.com; Comment comment@noreply.github.com Subject: Re: [OfficeDev/ews-java-api] Perpetual 401(Unauthorized) after running fine for period of time (#560)

I managed to get my hands on some IIS logs that are in the format of HttpProxy. Fortunately, the java EWS client uses a unique user-agent so I was able to spot the http requests, and sure enough it reports some of them as 401 Unauthenticated. Not sure what to make of this because I could reproduce+step on the client and all requests appear authenticated (albeit impersonated). I am attaching an excel export showing some of the traffic. Unfortunately, HttpProxy doesnt include the soap envelope so kinda blind there.

My Exchange admin days are mui rusty and not sure what I can do on the server side to better correlate the client requests.

Thanks so much for your insight.

ExchangeServicesClient.xlsxhttps://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgithub.com%2fOfficeDev%2fews-java-api%2ffiles%2f485617%2fExchangeServicesClient.xlsx&data=02%7c01%7cdavster%40exchange.microsoft.com%7c593344fd13f54c7bcc1608d3e24269a4%7c72f988bf86f141af91ab2d7cd011db47%7c1%7c1%7c636100747285090784&sdata=sl9BYIIuLkbercEISI1wP3MV%2fZ%2f5L2lAEhXIn6x3Qbs%3d

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgithub.com%2fOfficeDev%2fews-java-api%2fissues%2f560%23issuecomment-248679096&data=02%7c01%7cdavster%40exchange.microsoft.com%7c593344fd13f54c7bcc1608d3e24269a4%7c72f988bf86f141af91ab2d7cd011db47%7c1%7c0%7c636100747285090784&sdata=eYHGOuhlc%2bToa4LA3tpxH%2bCtNO2lyWQgAztTK0raVDs%3d, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgithub.com%2fnotifications%2funsubscribe-auth%2fAILDStwWlYSeqYn3hXrILT5PkoOdVjp5ks5qsWVjgaJpZM4J_-3-&data=02%7c01%7cdavster%40exchange.microsoft.com%7c593344fd13f54c7bcc1608d3e24269a4%7c72f988bf86f141af91ab2d7cd011db47%7c1%7c0%7c636100747285090784&sdata=U1vXR7LFV7CZl0sXmNrvyN0eF5IQUT3nkGUns6gkwN0%3d.

montehansen commented 8 years ago

Thank you Davestr for your insight on this problem. I was ultimately able to resolve this issue by simply using the "official" release that is part of the Maven repository and it's artifacts/dependencies. I was not able to establish if the problem I was experiencing was a result of the library dependency version differences, or the fact that I was building my solution via the source code. I did this mainly because the published version in Maven was so old (and since there doesn't appear to be a an official release in the repository).

Thanks again!

Monte

Parthipan2111 commented 2 years ago

Hi Monte.. We are facing similar issue in spring boot openshift architecture.we use ews-java-api 2.1 version.

Can you please help me with ..which version it's resolved ur issue?

Parthipan2111 commented 2 years ago

Those look like Exchange CAFÉ logs (from v15\logging\httpproxy…) rather than IIS logs. If you find one of the 401s from that log, it will show you the server it tried to proxy to. You can then look in the iisLogs folder on that dest server to find the corresponding entries. Can probably find it roughly by datetime.

From: Monte Hansen [mailto:notifications@github.com] Sent: Wednesday, September 21, 2016 10:12 AM To: OfficeDev/ews-java-api ews-java-api@noreply.github.com Cc: David Sterling (EXCHANGE) davster@exchange.microsoft.com; Comment comment@noreply.github.com Subject: Re: [OfficeDev/ews-java-api] Perpetual 401(Unauthorized) after running fine for period of time (#560)

I managed to get my hands on some IIS logs that are in the format of HttpProxy. Fortunately, the java EWS client uses a unique user-agent so I was able to spot the http requests, and sure enough it reports some of them as 401 Unauthenticated. Not sure what to make of this because I could reproduce+step on the client and all requests appear authenticated (albeit impersonated). I am attaching an excel export showing some of the traffic. Unfortunately, HttpProxy doesnt include the soap envelope so kinda blind there.

My Exchange admin days are mui rusty and not sure what I can do on the server side to better correlate the client requests.

Thanks so much for your insight.

ExchangeServicesClient.xlsxhttps://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgithub.com%2fOfficeDev%2fews-java-api%2ffiles%2f485617%2fExchangeServicesClient.xlsx&data=02%7c01%7cdavster%40exchange.microsoft.com%7c593344fd13f54c7bcc1608d3e24269a4%7c72f988bf86f141af91ab2d7cd011db47%7c1%7c1%7c636100747285090784&sdata=sl9BYIIuLkbercEISI1wP3MV%2fZ%2f5L2lAEhXIn6x3Qbs%3d

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgithub.com%2fOfficeDev%2fews-java-api%2fissues%2f560%23issuecomment-248679096&data=02%7c01%7cdavster%40exchange.microsoft.com%7c593344fd13f54c7bcc1608d3e24269a4%7c72f988bf86f141af91ab2d7cd011db47%7c1%7c0%7c636100747285090784&sdata=eYHGOuhlc%2bToa4LA3tpxH%2bCtNO2lyWQgAztTK0raVDs%3d, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgithub.com%2fnotifications%2funsubscribe-auth%2fAILDStwWlYSeqYn3hXrILT5PkoOdVjp5ks5qsWVjgaJpZM4J_-3-&data=02%7c01%7cdavster%40exchange.microsoft.com%7c593344fd13f54c7bcc1608d3e24269a4%7c72f988bf86f141af91ab2d7cd011db47%7c1%7c0%7c636100747285090784&sdata=U1vXR7LFV7CZl0sXmNrvyN0eF5IQUT3nkGUns6gkwN0%3d.

Hi Davster..

We user inbuilt tomcat with springboot..Where i can get the traffic logs?