InsertStream/UpsertStream

[morikat1509]

Hi, I upgraded the old tool to the new repo with release2.1.4. When trying to run the tool to import records from json to comsos db insert/upsert is working while insert stream and upsert stream are not doing any import(the records count still the same). I am using the CLI command and not the settings file.

Here is the command args for the tool:

string source = "json";
string sink = "cosmos-nosql";

string sourceSettingJson = "https://raw.githubusercontent.com/azure-samples/cosmos-db-migration-sample-data/main/nosql-data.json";

string targetDbConnnectionString = "AccountEndpoint=https://localhost:8081/;blabla";
string targetDatabaseName = "Database";
string targetCollectionName = "Prod8";

int retry = 2;
int batchSize = 100;

string cmdArgs = 
             $"--source {source} --sink {sink} " +
             $"--SourceSettings:FilePath={sourceSettingJson} " +
             $"--SinkSettings:ConnectionString=\"{targetDbConnnectionString}\" "+
             $"--SinkSettings:Database={targetDatabaseName} " +
             $"--SinkSettings:Container={targetCollectionName} " +
             $"--SinkSettings:PartitionKeyPath=\"/partitionKey\" " +
             $"--SinkSettings:MaxRetryCount= {retry}" +
             $"--SinkSettings:BatchSize={batchSize} " +
             $"--SinkSettings:WriteMode=InsertStream";

The tool is not failing but no documents are imported when using streams, what could be wrong with this setup if the insert and upsert are working? Are the streams logic ready to use?

[bowencode]

The -Stream modes can be slightly more performant but also less compatible. Depending on network specifics, ConnectionMode=Direct and WriteMode=InsertStream should be the fastest, but may also fail to transmit data. At the opposite end, ConnectionMode=Gateway and WriteMode=Upsert should be the most compatible set of options but also the least performant.

Outside of actual transfer performance, Stream modes also require different JSON serialization during transfer so there may be an issue specific to your data that is only causing problems with that serialization.