Unable to refresh the Lithnet connector in one specific environment since few days

[s0987]

Been using this Lithnet connector in our projects in different environments till now. But getting below error in one specific environment.

The specific error occurs in Google MA when the connector tries to refresh the schema:

---

The extensible extension returned an unsupported error. The stack trace is:

"System.ArrayTypeMismatchException: Attempted to access an element as a type incompatible with the array. at System.Collections.Generic.List1.Add(T item) at Lithnet.GoogleApps.GoogleJsonSerializer..ctor() at Lithnet.GoogleApps.SchemaRequestFactory.<>c__DisplayClass1_0.<.ctor>b__0() at Lithnet.GoogleApps.Pool1.CreateAndAddItem() at Lithnet.GoogleApps.Pool1.Take() at Lithnet.GoogleApps.BaseClientServicePool1.Take(NullValueHandling nullValueHandling) at Lithnet.GoogleApps.SchemaRequestFactory.ListSchemas(String customerID) at Lithnet.GoogleApps.MA.SchemaBuilderUsers.AddUserCustomSchema(MASchemaType type, IManagementAgentParameters config) in D:\dev\git\lithnet\googleapps-managementagent\src\Lithnet.GoogleApps.MA\Schema\SchemaBuilderUsers.cs:line 876 at Lithnet.GoogleApps.MA.SchemaBuilderUsers.BuildBaseSchema(MASchemaType type, IManagementAgentParameters config) in D:\dev\git\lithnet\googleapps-managementagent\src\Lithnet.GoogleApps.MA\Schema\SchemaBuilderUsers.cs:line 368 at Lithnet.GoogleApps.MA.SchemaBuilderUsers.d__2.MoveNext() in D:\dev\git\lithnet\googleapps-managementagent\src\Lithnet.GoogleApps.MA\Schema\SchemaBuilderUsers.cs:line 27 at Lithnet.GoogleApps.MA.SchemaBuilder.GetSchema(IManagementAgentParameters config) in D:\dev\git\lithnet\googleapps-managementagent\src\Lithnet.GoogleApps.MA\Schema\SchemaBuilder.cs:line 25 at Lithnet.GoogleApps.MA.ManagementAgent.GetSchema(KeyedCollection`2 configParameters) in D:\dev\git\lithnet\googleapps-managementagent\src\Lithnet.GoogleApps.MA\ManagementAgent.cs:line 528 Forefront Identity Manager 4.6.258.0"

---

Installed "Lithnet GoogleApps Management Agent 2.1.7774.34944"

Please note that no password nor any other settings been changed on server. Was able to Import, export, so on till last week.

[ryannewington]

@s0987

That's a strange one. Have their been any other ECMA2 MAs installed or updated in this environment since the issue started? Check for a NewtonSoft.Json.dll file in the extensions folder - My guess would be the two apps are using different versions of this DLL and are in conflict.

[s0987]

Your prompt response is well appreciated!! When replaced the DLL file with NewtonSoft.JSON v13.0.1:

· Schema Refresh works

· Delta Import Works

· Export works (with some errors)

· Full Import fails – with a new error (see below)

Google.API – new error:

The extensible extension returned an unsupported error.

The stack trace is:

"System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> Google.GoogleApiException: Google.Apis.Requests.RequestError

Invalid Input: query [400]

Errors [

           Message[Invalid Input: query] Location[ - ] Reason[invalid] Domain[global]

]

at Google.Apis.Requests.ClientServiceRequest`1.d__31.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Google.Apis.Requests.ClientServiceRequest`1.d__27.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Google.Apis.Requests.ClientServiceRequest`1.d__26.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()

at Lithnet.GoogleApps.ApiExtensions.ExecuteWithRetry[T](ClientServiceRequest1 request, Func2 shouldRetry, Int32 retryAttempts, Int32 consumeTokens)

at Lithnet.GoogleApps.ApiExtensions.ExecuteWithRetry[T](ClientServiceRequest`1 request, RetryEvents policy, Int32 retryAttempts, Int32 consumeTokens)

at Lithnet.GoogleApps.ApiExtensions.ExecuteWithRetry[T](ClientServiceRequest`1 request, RetryEvents policy)

at Lithnet.GoogleApps.UserRequestFactory.d__4.MoveNext()

at System.Collections.Concurrent.Partitioner.DynamicPartitionerForIEnumerable1.InternalPartitionEnumerable.GrabChunk_Buffered(KeyValuePair2[] destArray, Int32 requestedChunkSize, Int32& actualNumElementsGrabbed)

at System.Collections.Concurrent.Partitioner.DynamicPartitionEnumerator_Abstract`2.MoveNext()

at System.Threading.Tasks.Parallel.<>c__DisplayClass42_0`2.b__1()

at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)

at System.Threading.Tasks.Task.<>c__DisplayClass176_0.b__0(Object )

--- End of inner exception stack trace ---

at Lithnet.GoogleApps.MA.ManagementAgent.<>c__DisplayClass40_0.b_0(Task`1 ) in D:\dev\git\lithnet\googleapps-managementagent\src\Lithnet.GoogleApps.MA\ManagementAgent.cs:line 380

at System.Threading.Tasks.Task.Execute()

--- End of inner exception stack trace ---

at Lithnet.GoogleApps.MA.ManagementAgent.GetImportEntriesFull() in D:\dev\git\lithnet\googleapps-managementagent\src\Lithnet.GoogleApps.MA\ManagementAgent.cs:line 481

at Lithnet.GoogleApps.MA.ManagementAgent.GetImportEntries(GetImportEntriesRunStep importRunStep) in D:\dev\git\lithnet\googleapps-managementagent\src\Lithnet.GoogleApps.MA\ManagementAgent.cs:line 399

---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> Google.GoogleApiException: Google.Apis.Requests.RequestError

Invalid Input: query [400]

Errors [

           Message[Invalid Input: query] Location[ - ] Reason[invalid] Domain[global]

]

at Google.Apis.Requests.ClientServiceRequest`1.d__31.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Google.Apis.Requests.ClientServiceRequest`1.d__27.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Google.Apis.Requests.ClientServiceRequest`1.d__26.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()

at Lithnet.GoogleApps.ApiExtensions.ExecuteWithRetry[T](ClientServiceRequest1 request, Func2 shouldRetry, Int32 retryAttempts, Int32 consumeTokens)

at Lithnet.GoogleApps.ApiExtensions.ExecuteWithRetry[T](ClientServiceRequest`1 request, RetryEvents policy, Int32 retryAttempts, Int32 consumeTokens)

at Lithnet.GoogleApps.ApiExtensions.ExecuteWithRetry[T](ClientServiceRequest`1 request, RetryEvents policy)

at Lithnet.GoogleApps.UserRequestFactory.d__4.MoveNext()

at System.Collections.Concurrent.Partitioner.DynamicPartitionerForIEnumerable1.InternalPartitionEnumerable.GrabChunk_Buffered(KeyValuePair2[] destArray, Int32 requestedChunkSize, Int32& actualNumElementsGrabbed)

at System.Collections.Concurrent.Partitioner.DynamicPartitionEnumerator_Abstract`2.MoveNext()

at System.Threading.Tasks.Parallel.<>c__DisplayClass42_0`2.b__1()

at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)

at System.Threading.Tasks.Task.<>c__DisplayClass176_0.b__0(Object )

--- End of inner exception stack trace ---

at Lithnet.GoogleApps.MA.ManagementAgent.<>c__DisplayClass40_0.b_0(Task`1 ) in D:\dev\git\lithnet\googleapps-managementagent\src\Lithnet.GoogleApps.MA\ManagementAgent.cs:line 380

at System.Threading.Tasks.Task.Execute()

---> (Inner Exception #0) The service admin has thrown an exception: Google.GoogleApiException: Google.Apis.Requests.RequestError

Invalid Input: query [400]

Errors [

           Message[Invalid Input: query] Location[ - ] Reason[invalid] Domain[global]

]

at Google.Apis.Requests.ClientServiceRequest`1.d__31.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Google.Apis.Requests.ClientServiceRequest`1.d__27.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Google.Apis.Requests.ClientServiceRequest`1.d__26.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()

at Lithnet.GoogleApps.ApiExtensions.ExecuteWithRetry[T](ClientServiceRequest1 request, Func2 shouldRetry, Int32 retryAttempts, Int32 consumeTokens)

at Lithnet.GoogleApps.ApiExtensions.ExecuteWithRetry[T](ClientServiceRequest`1 request, RetryEvents policy, Int32 retryAttempts, Int32 consumeTokens)

at Lithnet.GoogleApps.ApiExtensions.ExecuteWithRetry[T](ClientServiceRequest`1 request, RetryEvents policy)

at Lithnet.GoogleApps.UserRequestFactory.d__4.MoveNext()

at System.Collections.Concurrent.Partitioner.DynamicPartitionerForIEnumerable1.InternalPartitionEnumerable.GrabChunk_Buffered(KeyValuePair2[] destArray, Int32 requestedChunkSize, Int32& actualNumElementsGrabbed)

at System.Collections.Concurrent.Partitioner.DynamicPartitionEnumerator_Abstract`2.MoveNext()

at System.Threading.Tasks.Parallel.<>c__DisplayClass42_0`2.b__1()

at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)

at System.Threading.Tasks.Task.<>c__DisplayClass176_0.b__0(Object )<---

<---

Forefront Identity Manager 4.6.258.0"

[s0987]

Downloaded the Lithnet custom Google connector code but unable to debug the solution on local. Unable to find reference for lithnet.licensing.Core in solution nor in Github nor in Nuget. Is there anyway to debug the solution to resolve the Full Import issue on Lithnet Google MA?

[ryannewington]

Can you delete the newtonsoft.json.dll file from the folder?

All the required DLLs are embedded in the MA assembly itself. But if .NET finds a newer version on the disk, its using that instead of the specific version it needs.

This one huge problem with the ECMA model. All assemblies sharing a folder means we are back to DLL hell. I've tried to make this problem not as bad for Lithnet assemblies by embedding their dependencies, but if someone else drops DLLs in the folder we need, then things go haywire. The Google SDK is extremely fragile with this stuff, so the weird behaviour your seeing makes sense in that respect.

Due to the IL merging we do, debugging is really difficult, and I dont think is going to help you unpack this one.

[s0987]

Thanks for reply! We see DLL files in the MIM Extensions folder, and also multiple copies in the MIM ExtensionsCache subfolders. Should we delete just the primary on in Extensions, or all of them, including the Extension Cache?

[ryannewington]

You can actually stop the MIM sync service, anmd delete all the folders in the extensions cache.

Each time an MA binary changes, FIM takes a copy of the folder and puts it in cache and loads DLLs from there. When you drop new files, MIM detects this, creates a new cache, and copies the updated files from there.

It just never cleans up the old copies.

[s0987]

Ok. Is there any chance (or plans to in the near future) compile the LithNet Google connector with the latest NuGet version of Google.Apis (1.52.0) and Newtonsoft.JSON?

[ryannewington]

I'm committed to making sure it works in as many customer environments as possible, as opposed to always running the latest version. As mentioned, the Google SDK is very prone to breaking changes, so I update when necessary but not for the fun of it.

Can you describe what you have found in your environment, so I can see what options we have going forward?

1. What DLLs did you find in the extensions folder? If I can get a full list, with versions, that would be very useful.
2. Was Newtonsoft present, and do you know which MA dropped this DLL?
3. Were any of the Google SDK DLLs, and do you know which MA dropped this DLL?
4. Is they are from some other MA, what versions does that MA use?
5. Did deleting the DLLs allow the Google MA to start working? I assume doing so broke something else.

The more information I have the better. These scenarios are always complex, so I need to get a clear picture of what you have discovered.

Can you also provide me the assembly binding redirect section of your miiserver.exe.config file?

Once I understand the specifics of what is going on in your environment, I can look to see what can be done to get it working.

[s0987]

I need to check with my team and get back to you on this. We narrowed the issue down to the [User API Query Parameter] on the [Global Parameters] page in the MA config settings. Trying to use a value like [orgUnitPath=/Students] or [domain=test.domain] breaks. However, setting the query to [email:*@test.domain] works fine. We will use this setting for now to finish the Full Import, but would prefer to use the orgUnitPath query instead. We started experiencing issues with OrgUnit parsing around June 17, which is why we updated to the latest Google.Apis and Newtonsoft.JSON NUGet packages.

[ryannewington]

ah ok. yeah, take those packages out of the folder. Forcing the MA to load versions of those DLLs that it wasn't compiled with is guaranteed to cause issues.

So was the query parameter working before, and then stopped working? Or are you trying to do a different query and unable to get that to work?

[s0987]

yes, the query parameter used to work, but stopped working some time around mid June. (Same query: rgUnitPath=/PathToSubOU)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.