Azure / azure-documentdb-datamigrationtool

Azure DocumentDB Data Migration Tool
MIT License
387 stars 183 forks source link

Using Azure CosmosDB as a source fails with "Server could not parse the Url" #88

Closed bcrobinson closed 5 years ago

bcrobinson commented 5 years ago

This happens with both dt.exe and dtui.exe, full exception stack at end.

Repro steps: 1) Set Source Information to Azure Cosmos DB 1) Enter valid connection string, click Verify, this shows a success dialog. 1) Set Query to select * from c 1) Set Target Information to a JSON file (it doesn't seem to have thing to do with target & Json file should rule out any errors here) 1) Start Import 1) At this point an error diaglog will appear with the following error System.AggregateException: One or more errors occurred. ---> Microsoft.Azure.Documents.DocumentClientException: Server could not parse the Url., documentdb-dotnet-sdk/2.2.1 Host/64-bit MicrosoftWindowsNT/6.2.9200.0 (full stack trace at end)

This consistently happens with multiple collections using V1.8.2, but the export runs successfully with V1.7 (as I happened to have that version lying around).

The only thing that could be unusual about my setup is that I just created a new database in the Cosmos account with database provisioning. So the account contains 2 databases, the original one with per-collection resource provisioning and the new one with database provisioning.

After telling the export to connect using HTTPS & turning on Fiddler I can see it successfully making requests to Cosmos (looks like fetching metadata about the collection) then fails with the following request

GET https://###redacted###.documents.azure.com/dbs/+alYAA==/colls/+alYAA== HTTP/1.1

Which returns

HTTP/1.1 404 Not Found
Content-Type: application/json
Content-Location: https:///###redacted###.documents.azure.com/dbs/+alYAA==/colls/+alYAA==
Server: Microsoft-HTTPAPI/2.0
x-ms-activity-id: b4bb64ca-398a-49a1-9968-0f16774a45d2
Strict-Transport-Security: max-age=31536000
x-ms-gatewayversion: version=2.2.0.0
Date: Mon, 01 Apr 2019 11:23:12 GMT
Content-Length: 63

{"code":"NotFound","message":"Server could not parse the Url."}

From the successful export using V1.7 I can see the following request at this point instead

GET https://###redacted###.documents.azure.com/dbs/+alYAA==/colls/+alYAPQ2AMA= HTTP/1.1

followed by

GET https://###redacted###.documents.azure.com/dbs/+alYAA==/colls/+alYAPQ2AMA=/pkranges HTTP/1.1

Full Stack trace

System.AggregateException: One or more errors occurred. ---> Microsoft.Azure.Documents.DocumentClientException: Server could not parse the Url., documentdb-dotnet-sdk/2.2.1 Host/64-bit MicrosoftWindowsNT/6.2.9200.0
   at Microsoft.Azure.Documents.Client.ClientExtensions.<ParseResponseAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.GatewayStoreModel.<>c__DisplayClass21_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<>c__DisplayClass1_0.<<ExecuteAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.GatewayStoreModel.<InvokeAsync>d__21.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.GatewayStoreModel.<ProcessMessageAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Routing.ClientCollectionCache.<ReadCollectionAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<>c__DisplayClass1_0.<<ExecuteAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Common.AsyncCache`2.<GetAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Common.CollectionCache.<ResolveCollectionAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextFactory.<CreateDocumentQueryExecutionContextAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Linq.DocumentQuery`1.<CreateDocumentQueryExecutionContextAsync>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Linq.DocumentQuery`1.<ExecuteNextPrivateAsync>d__36`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.DataTransfer.DocumentDb.Client.Enumeration.AsyncEnumeratorBase`2.<GetCurrentChunkCursor>d__11.MoveNext() in d:\a\1\s\DocumentDb\Microsoft.DataTransfer.DocumentDb\Client\Enumeration\AsyncEnumeratorBase.cs:line 0
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.DataTransfer.DocumentDb.Client.Enumeration.AsyncEnumeratorBase`2.<MoveNextAsync>d__9.MoveNext() in d:\a\1\s\DocumentDb\Microsoft.DataTransfer.DocumentDb\Client\Enumeration\AsyncEnumeratorBase.cs:line 0
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.DataTransfer.DocumentDb.Source.DocumentDbSourceAdapter.<ReadNextAsync>d__4.MoveNext() in d:\a\1\s\DocumentDb\Microsoft.DataTransfer.DocumentDb\Source\DocumentDbSourceAdapter.cs:line 34
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.DataTransfer.Core.Service.DataTransferAction.<ExecuteAsync>d__0.MoveNext() in d:\a\1\s\Core\Microsoft.DataTransfer.Core\Service\DataTransferAction.cs:line 35
   --- End of inner exception stack trace ---
   at Microsoft.DataTransfer.Core.Service.DataTransferAction.<ExecuteAsync>d__0.MoveNext() in d:\a\1\s\Core\Microsoft.DataTransfer.Core\Service\DataTransferAction.cs:line 0
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.DataTransfer.Core.Service.DataTransferService.<TransferAsync>d__6.MoveNext() in d:\a\1\s\Core\Microsoft.DataTransfer.Core\Service\DataTransferService.cs:line 71
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.DataTransfer.WpfHost.Steps.Import.ImportStep.<Execute>d__10.MoveNext() in d:\a\1\s\Wpf\Microsoft.DataTransfer.WpfHost\Steps\Import\ImportStep.cs:line 73
---> (Inner Exception #0) Microsoft.Azure.Documents.DocumentClientException: Server could not parse the Url., documentdb-dotnet-sdk/2.2.1 Host/64-bit MicrosoftWindowsNT/6.2.9200.0
   at Microsoft.Azure.Documents.Client.ClientExtensions.<ParseResponseAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.GatewayStoreModel.<>c__DisplayClass21_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<>c__DisplayClass1_0.<<ExecuteAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.GatewayStoreModel.<InvokeAsync>d__21.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.GatewayStoreModel.<ProcessMessageAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Routing.ClientCollectionCache.<ReadCollectionAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<>c__DisplayClass1_0.<<ExecuteAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Common.AsyncCache`2.<GetAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Common.CollectionCache.<ResolveCollectionAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextFactory.<CreateDocumentQueryExecutionContextAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Linq.DocumentQuery`1.<CreateDocumentQueryExecutionContextAsync>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.Linq.DocumentQuery`1.<ExecuteNextPrivateAsync>d__36`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.DataTransfer.DocumentDb.Client.Enumeration.AsyncEnumeratorBase`2.<GetCurrentChunkCursor>d__11.MoveNext() in d:\a\1\s\DocumentDb\Microsoft.DataTransfer.DocumentDb\Client\Enumeration\AsyncEnumeratorBase.cs:line 0
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.DataTransfer.DocumentDb.Client.Enumeration.AsyncEnumeratorBase`2.<MoveNextAsync>d__9.MoveNext() in d:\a\1\s\DocumentDb\Microsoft.DataTransfer.DocumentDb\Client\Enumeration\AsyncEnumeratorBase.cs:line 0
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.DataTransfer.DocumentDb.Source.DocumentDbSourceAdapter.<ReadNextAsync>d__4.MoveNext() in d:\a\1\s\DocumentDb\Microsoft.DataTransfer.DocumentDb\Source\DocumentDbSourceAdapter.cs:line 34
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.DataTransfer.Core.Service.DataTransferAction.<ExecuteAsync>d__0.MoveNext() in d:\a\1\s\Core\Microsoft.DataTransfer.Core\Service\DataTransferAction.cs:line 35<---
ngeor-sendasyou commented 5 years ago

I have the same problem with 1.8.2 but the problem does not exist in 1.8.1 (just run both versions)

peterson1 commented 5 years ago

I'm having the exact same problem with 1.8.2. I ran 1.8.1, as ngeor-sendasyou suggested, and it ran successfully.

japicket commented 5 years ago

I apparently have two different "versions" of 1.8.2. One of them does not work and gets the above error message, while the earlier 1.8.2 version works. A colleague downloaded 1.8.2 about a week before I did; the earlier download is the one that worked.

The only difference in the drops folder is that the version that doesn't work includes Microsoft.Azure.Documents.Client.TransientFaultHandling.dll and Microsoft.Azure.Documents.Client.TransientFaultHandling.xml.

haitsongmsft commented 5 years ago

Please take the latest binary drop for 1.8.2.