search

Azure / azure-documentdb-datamigrationtool

Azure DocumentDB Data Migration Tool
MIT License
388 stars 182 forks source link

779 Transferred, but 0 visible in Data Explorer #168

Closed mmontana-bag closed 2 years ago

mmontana-bag commented 2 years ago

We use dt.exe to Migrate Data:

dt.exe tells us "Transferred: 779" and "Failed 0", but after that when i look inside Cosmos DB in Portal with Data Explorer there is 0 Data.

Used dt.exe command:

dt.exe /ErrorLog:C:\Sys\Cosmosdb\data.csv /OverwriteErrorLog /s:AzureTable /s.ConnectionString:DefaultEndpointsProtocol=https;AccountName=xxx;AccountKey=xxx /s.Table:data /t:DocumentDB /t.ConnectionString:AccountEndpoint=https://xxx.documents.azure.com:443/;AccountKey=xxx;Database=TablesDB /t.CollectionThroughput:400 /t.Collection:data /t.ParallelRequests:1

I double checked that I used the correct Endpoint - This should not be the problem. This Tenant only got one CosmosDB Tenant-wide. There is not an other DB where the Data could go.

MS Support told me that I "migrated data to CosmosDB Table API account but in SQL Api format" and should raise an issue here, becuase it is maintained by open-source community.

What are we doing wrong?

PaulCheng commented 2 years ago

Instead of /t:DocumentDB, you should use /t:TableAPIBulk. /t.ConnectionString should be obtained from Portal. /t.TableName:data

chshihMSFT commented 2 years ago

You can use https://cosmos.azure.com/sunset with SQL format connectionString (AccountEndpoint=https://xxx.documents.azure.com:443/;AccountKey=xxx;) to check transferred data, but you'll see those are ill-format against Cosmos DB Table API. Please check /t:TableAPIBulk as @PaulCheng mentioned, and below command sample for migrating data from Azure Storage Table Service to Cosmos DB Table API. dt.exe /s:AzureTable /s.ConnectionString:DefaultEndpointsProtocol=https;AccountName=<Azure Table storage account name>;AccountKey=<Account Key>;EndpointSuffix=core.windows.net /s.Table:<Table name> /t:TableAPIBulk /t.ConnectionString:DefaultEndpointsProtocol=https;AccountName=<Azure Cosmos DB account name>;AccountKey=<Azure Cosmos DB account key>;TableEndpoint=https://<Account name>.table.cosmos.azure.com:443 /t.TableName:<Table name> /t.Overwrite ref doc: https://docs.microsoft.com/en-us/azure/cosmos-db/table/table-import#sample-command-source-is-table-storage

OR AzCopy v7.3.1 can be the help as well, as the above doc describe.

  1. Export Data from Azure Table Storage to local file .\AzCopy /Source:https://tablestoragename.table.core.windows.net/tablename/ /SourceKey:tablestoragekey /Dest:C:\Temp\tableoutput

  2. Import local file to Azure Cosmos DB Table API .\AzCopy /V:C:\Temp\AzCopyLog.log /Source:C:\Temp\tableoutput\ /Dest:https://cosmosdbaccountname.table.cosmos.azure.com/dsttable/ /DestKey:cosmosdbkey /Manifest:"cosmosdbaccountname_collectionname_timestamp.manifest" /EntityOperation:InsertOrSkip Note: you can specify EntityOperation for more detail migration behavior, lists parameter definition for your reference InsertOrSkip - Skips an existing entity or inserts a new entity if it does not exist in the table. InsertOrMerge - Merges an existing entity or inserts a new entity if it does not exist in the table. InsertOrReplace - Replaces an existing entity or inserts a new entity if it does not exist in the table.