search

SpanningCloudApps / SB365-Powershell

Spanning Backup for Office 365 REST API PowerShell Module
https://spanning.com
Apache License 2.0
10 stars 5 forks source link

(504) Gateway Timeout. #41

Closed Denver80211 closed 2 years ago

Denver80211 commented 3 years ago

Running a script to refresh user licenses that's worked well for a while.. last few days getting the error below running Get-SpanningAssignedUsers

Get-SpanningTenantInfo works fine so I know I'm connected. Updated to 4.1 but same results

Invoke-WebRequest : The remote server returned an error: (504) Gateway Timeout. At C:\Windows\system32\WindowsPowerShell\v1.0\Modules\SpanningO365\Private\Invoke-SpanningRequest.ps1:112 char:23

Invoke-WebRequest : Cannot validate argument on parameter 'Uri'. The argument is null or empty. Provide an argument that is not null or empty, and then try the command again. At C:\Windows\system32\WindowsPowerShell\v1.0\Modules\SpanningO365\Private\Invoke-SpanningRequest.ps1:115 char:50

SPMatthewMcD commented 3 years ago

I'll take a look. In the mean time can you try Get-SpanningUser -UserType Assigned ? https://github.com/SpanningCloudApps/SB365-Powershell/blob/master/samples.md#listing-users

Denver80211 commented 3 years ago

get-spanninguser -usertype assigned

Invoke-WebRequest : The remote server returned an error: (504) Gateway Timeout. At C:\Windows\system32\WindowsPowerShell\v1.0\Modules\SpanningO365\Private\Invoke-SpanningRequest.ps1:128 char:23

Invoke-WebRequest : Cannot validate argument on parameter 'Uri'. The argument is null or empty. Provide an argument that is not null or empty, and then try the command again. At C:\Windows\system32\WindowsPowerShell\v1.0\Modules\SpanningO365\Private\Invoke-SpanningRequest.ps1:131 char:50

Denver80211 commented 3 years ago

I really want to blame the security team for blocking something but I don't think I would be able to query the Tenant Info if that was the case

SPMatthewMcD commented 3 years ago

Thanks for testing. I'll test it here in a few minutes.

sconstable1234 commented 3 years ago

I am running into the same problem, With both commands as well.

sconstable1234 commented 3 years ago

for the Get-spanninguser command I noticed that I get less timeouts with the -size 40 parameter, but It does not seem like its working for every page, some pages still timeout.

SPMatthewMcD commented 3 years ago

Thanks for the additional info. I'll do some more investigation tomorrow.

Lucidity-Ben-Williams commented 3 years ago

Hi There @SPMatthewMcD - Also seeings this! Seems to be timing out at the get-spanninguser command too.

SPMatthewMcD commented 3 years ago

Thanks folks, we're looking into it on our end. As I understand, the tenant info is OK, but it's happening when requesting users.

hemingsent commented 3 years ago

Seeing this as well, let me know if I can help in any way!

SPMatthewMcD commented 3 years ago

Thanks all. We're investigating. I confess that in this module we don't have any logic for throttling yet. I need to add that since we moved to a new and better API host I have been behind on improvements like that. I'll be working on that this week. That should allow the Module to handle this error "more gracefully".

At this time, know that we are investigating the backend and I'll be working on the module.

TRipka07 commented 3 years ago

Any update on this?

sconstable1234 commented 3 years ago

I have a ticket into spanning about this because I was able to reproduce the problem without the PowerShell Module. It seems to be something with the backend. If you change the page size to really low (like 10) it will work, but with 1000 Page size (The default in the PowerShell module) it chokes.

SPMatthewMcD commented 3 years ago

Thanks, we have escalated it to Engineering and they are investigating. I'll let you know what I hear.

brokencrow commented 3 years ago

Also having the same issues. TenantInfo comes through ok, but any other commands timeout. Any updates on a resolution ?

SPMatthewMcD commented 3 years ago

Our team is working on it. I hope to have an update early next week. In the mean time can you try the workaround of setting the -size parameter to a value under 40?

MortenRei commented 3 years ago

Same problem here. "Get-SpanningTenantInfo" seems to fail for some tenants, but not all. Seems like the problem is occasionally on some tenants, but one other tenants it always fails with the 504 error.

mtheule commented 3 years ago

The workaround using page size isn't working for me. "get-spanninguser -UserType Admins -size 10" throws the (504) Gateway Timeout error

brokencrow commented 3 years ago

I have tested and can confirm that size 10 - 40 does work, though not a sufficient work around in my case as I need to return the results for approximately 800 users.

SPMatthewMcD commented 3 years ago

Thanks for the confirmation. We are working on an optimization that should relieve the database load. I'll reply here when we deploy the update to production.

SPMatthewMcD commented 2 years ago

All,

I just ran this for a customer and though it took time, it did work for 8000+ user tenant. $users = Get-SpanningUser -UserType All -Size 20 -Verbose

The verbose flag will show you progress in the shell and let you know it's working.

We are planning to release an update next week to address the performance.

mtheule commented 2 years ago

Thanks for this update. I was able to retrieve user information using the sample command you supplied. Hooray!

Just a heads up, though. It doesn't appear that that Size parameter is being honored. I copy/pasted your sample command and let it run to completion, expecting 20 records. After about an hour, it showed results for more than 885 records. My screen buffer is set for 9000 rows and it was full.

It is better to have more information than not enough, so I thank you for the effort you have made to get the Powershell interface working again.

MATT

Denver80211 commented 2 years ago

Better-ish but, new error when adding licenses: Running enable-SpanningUserlist -UserPrincipalName $addFromADUsers where $addfromADUsers is an array of UPNs Error:

Invoke-WebRequest : {"code":"InternalServerError","message":"Non-2xx status returned from web service: 409. HttpResponseError: {\"url\":\" http://bo-a-p-aat.spanning.co/tenant_caches/2369590/users /assign\",\"headers\":{\"cache-control\":\"no-store\",\"content-encoding\":\"gzip\",\"content-type\":\"application/json\",\"date\":\"Fri, 23 Jul 2021 00:16:05 GMT\",\"pragma\":\"no-cache\",\"transfer-encoding\":\"chunked\",\"connection\":\"Close\"},\"responseBody\":{\"code\":\"ConflictError\",\"message\":\"Offline Users found in users to be assigned.\"},\"responseStatus\":409}"} At C:\Windows\system32\WindowsPowerShell\v1.0\Modules\SpanningO365\Private\Invoke-SpanningRequest.ps1:191 char:22

On Thu, Jul 22, 2021 at 5:26 PM mtheule @.***> wrote:

Thanks for this update. I was able to retrieve user information using the sample command you supplied. Hooray!

Just a heads up, though. It doesn't appear that that Size parameter is being honored. I copy/pasted your sample command and let it run to completion. After about an hour, it showed results for more than 885 records. My screen buffer is set for 9000 rows and it was full.

It is better to have more information than not enough, so I thank you for the effort you have made to get the Powershell interface working again.

MATT

From: Matthew McDermott @.> Sent: Thursday, July 22, 2021 3:18 PM To: SpanningCloudApps/SB365-Powershell @.> Cc: Matt Theule @.>; Comment @.> Subject: Re: [SpanningCloudApps/SB365-Powershell] (504) Gateway Timeout. (#41)

All,

I just ran this for a customer and though it took time, it did work for 8000+ user tenant. Get-SpanningUser -UserType All -Size 20 -Verbose

The verbose flag will show you progress in the shell and let you know it's working.

We are planning to release an update next week to address the performance.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub< https://github.com/SpanningCloudApps/SB365-Powershell/issues/41#issuecomment-885272203>, or unsubscribe< https://github.com/notifications/unsubscribe-auth/AJEWVOQVD53ODYAO2LLZC4TTZCKK7ANCNFSM463TTUQQ>.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SpanningCloudApps/SB365-Powershell/issues/41#issuecomment-885300472, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK5VQ7NAZY6L2GVGUJNUSZTTZCSLDANCNFSM463TTUQQ .

SPMatthewMcD commented 2 years ago

Clarification on the -size parameter. It regulates the page size of the response, not the total number of records received. Thus requesting -size 20 is going to return all the records in pages of 20. Use the -Verbose flag and you can watch the requests go by. I was successful using this on a tenant with 8000+ users but it did take longer than we'd like. We're planning an update to address performance next week.

Denver80211 commented 2 years ago

In my case I'm just trying to add 16 more users to licensing

On Fri, Jul 23, 2021 at 7:28 AM Matthew McDermott @.***> wrote:

Clarification on the -size parameter. It regulates the page size of the response, not the total number of records received. Thus requesting -size 20 is going to return all the records in pages of 20. Use the -Verbose flag and you can watch the requests go by. I was successful using this on a tenant with 8000+ users but it did take longer than we'd like. We're planning an update to address performance next week.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SpanningCloudApps/SB365-Powershell/issues/41#issuecomment-885639031, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK5VQ7LPLID2ZA4W434D7O3TZFU6RANCNFSM463TTUQQ .

SPMatthewMcD commented 2 years ago

Understood. Have you tried a for-each licensing each user. (I know it's a pain, just trying to find alternatives.)

Denver80211 commented 2 years ago

not a pain.. I'll try

On Fri, Jul 23, 2021 at 8:05 AM Matthew McDermott @.***> wrote:

Understood. Have you tried a for-each licensing each user. (I know it's a pain, just trying to find alternatives.)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SpanningCloudApps/SB365-Powershell/issues/41#issuecomment-885662983, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK5VQ7MOQYJGAUQX5JIDXTLTZFZKHANCNFSM463TTUQQ .

Denver80211 commented 2 years ago

same error with just one... instantaneous. I had been using the script for a while enable-SpanningUserlist -UserPrincipalName $addFromADUsers[0]

On Fri, Jul 23, 2021 at 8:16 AM JFB @.***> wrote:

not a pain.. I'll try

On Fri, Jul 23, 2021 at 8:05 AM Matthew McDermott < @.***> wrote:

Understood. Have you tried a for-each licensing each user. (I know it's a pain, just trying to find alternatives.)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SpanningCloudApps/SB365-Powershell/issues/41#issuecomment-885662983, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK5VQ7MOQYJGAUQX5JIDXTLTZFZKHANCNFSM463TTUQQ .

SPMatthewMcD commented 2 years ago

Thanks,

That cmdlet just wraps Enable-SpanningUserList can you try something like (untested pseudo code):

foreach ($user in $userlist) {
  Enable-SpanningUserList -UserPrincipalNames $user
}
Denver80211 commented 2 years ago

enable-SpanningUserlist -UserPrincipalName $addFromADUsers[0]does what you ask for one user, fails

On Fri, Jul 23, 2021 at 9:54 AM Matthew McDermott @.***> wrote:

Thanks,

That cmdlet just wraps Enable-SpanningUserList can you try something like (untested pseudo code):

foreach ($user in $userlist) { Enable-SpanningUserList -UserPrincipalNames $user }

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SpanningCloudApps/SB365-Powershell/issues/41#issuecomment-885735146, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK5VQ7JXOC6YJ6FTIB53RDLTZGGD3ANCNFSM463TTUQQ .

SPMatthewMcD commented 2 years ago

Thanks, we have an enhancement in QA that should release this week.

SPMatthewMcD commented 2 years ago

We released an update in all regions today. I tested on an 8000+ user tenant and it worked without error.

I need to test the module for performance and see if I can improve the processing time. If anyone wants to have a go I'll review PRs.

CRSMDGuy commented 2 years ago

I'm still facing the same errors after the updates have been released. Please see attached error. Capture Thanks, Mike from CRS

SPMatthewMcD commented 2 years ago

@CRSMDGuy,

I don't see a 504, I see an UnauthorizedError. Are you certain your token is still valid and that you are using the same account that generated the token? If you are please email me the info at matt.mcdermott@spanning.com.

TRipka07 commented 2 years ago

I've tested our script this morning and was able to license a user successfully. I think the changes on 7/28 are working. We will continue to monitor. Thanks.

SPMatthewMcD commented 2 years ago

@TRipka07 thanks for the confirmation. @Denver80211 is it working for you?

hemingsent commented 2 years ago

I tested mine as well and no errors anymore.

Thank you, I can't imagine assigning thousands of licenses manually!

SPMatthewMcD commented 2 years ago

@hemingsent Thanks for the confirmation. The team and I are working on a feature for early Q4 that will monitor an AAD Group. When you add users, we'll add a license and back them up. If you run low we'll notify you. Should take that pain away.

Denver80211 commented 2 years ago

Really excited to give this a try when I get back to the office on Monday. Do we need to load an updated module or is this all backend changes?

On Fri, Jul 30, 2021, 11:02 AM hemingsent @.***> wrote:

I tested mine as well and no errors anymore.

Thank you, I can't imagine assigning thousands of licenses manually!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SpanningCloudApps/SB365-Powershell/issues/41#issuecomment-890029339, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK5VQ7PVRS6O5WW342HXLNDT2LLKXANCNFSM463TTUQQ .

SPMatthewMcD commented 2 years ago

No update to the module (yet) this is purely our backed update. I am profiling the module now thanks to a customer with a ton of users, to see if I can find bottlenecks. If I do I'll post an update.

SPMatthewMcD commented 2 years ago

Open questions to the group here, clearly all of you are handling large numbers of users. In my profiling I am finding that PowerShell (specifically the Convert-FromJson cmdlet is consuming vast amounts of time and memory. Over the course of several tests I am seeing the requests taking 30 ish seconds and the Json conversion taking 2-3 times that long.

One solution I found is to move to PowerShell 7x. It natively uses NewtonSoft for Json parsing. This improves, but does not eliminate the issue. Times are cut in half, better but not great.

Changing the parser to Convert-FromJsonFast (which requires PoSh 7.x) adds an extra element of speed. Overall I was able to take the request from this morning that took 20 minutes down to 2.3 minutes.

Can I get an opinion on moving to PowerShell 7.5 and leaving this module as the last PowerShell 5 compatible version? I have a few things that I can do to improve the speed before I shift to PowerShell 7.

Denver80211 commented 2 years ago

This may be an ill informed opinion but, I'm reluctant to move to anything not natively available in windows to keep distributed administration predictable. I just went through the hassle of updating a ton of 2012 machines at the new job to run powershell 5 scripts. So installing 7 makes me nervous about building solutions that won't work across my already messy environment. That being said, I understand the attraction to new and better software.... you just might be leaving some of us behind

On Fri, Jul 30, 2021 at 3:00 PM Matthew McDermott @.***> wrote:

Open questions to the group here, clearly all of you are handling large numbers of users. In my profiling I am finding that PowerShell (specifically the Convert-FromJson cmdlet is consuming vast amounts of time and memory. Over the course of several tests I am seeing the requests taking 30 ish seconds and the Json conversion taking 2-3 times that long.

One solution I found is to move to PowerShell 7x. It natively uses NewtonSoft for Json parsing. This improves, but does not eliminate the issue. Times are cut in half, better but not great.

Changing the parser to Convert-FromJsonFast (which requires PoSh 7.x) adds an extra element of speed. Overall I was able to take the request from this morning that took 20 minutes down to 2.3 minutes.

Can I get an opinion on moving to PowerShell 7.5 and leaving this module as the last PowerShell 5 compatible version? I have a few things that I can do to improve the speed before I shift to PowerShell 7.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SpanningCloudApps/SB365-Powershell/issues/41#issuecomment-890151004, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK5VQ7NIT344CGOF2YQFKX3T2MHHBANCNFSM463TTUQQ .

SPMatthewMcD commented 2 years ago

@Denver80211 Great feedback.

I think I'll create a "beta branch" for the version 7 stuff. There is only so much I can do in 5 based on what I see in the PowerShell repo and other complaints about how slow Convert-FromJson is in V5.

SPMatthewMcD commented 2 years ago

Closing this issue as most have reported that it's working now. I'll be releasing an update soon that improves the Module performance.

SPMatthewMcD commented 2 years ago

All, I just posted an update to the development branch that has a few optimizations. Feel free to clone or download to test. It should run about 30% faster than before.