Azure / blobporter

Highly concurrent data transfer tool for Azure Blob Storage.
Other
85 stars 29 forks source link

Invalid input. You can't start a transfer with duplicate sources. #104

Closed kratz00 closed 5 years ago

kratz00 commented 5 years ago

Environment:

Test: BlobPorter.exe -f "1GB:10" -c perf -t perf-blockblob

Current result: 2018/09/26 15:25:29 Invalid input. You can't start a transfer with duplicate sources. First duplicate detected:&{1GB_10503766000.dat 1073741824 1GB_10503766000.dat}

Expected result: Ten 1GB files should be uploaded.

Remark: The culprit seems to be: https://github.com/Azure/blobporter/blob/febba38ac00ce6bbbeb8d0f9b388e23dfe82f593/sources/perfsource.go#L54 The precision of time.Now() is not high enough on Windows.

Here is the output of the ten generated names on Windows: [1GB_10662266000.dat 1GB_10662266000.dat 1GB_10662266000.dat 1GB_10662266000.dat 1GB_10662266000.dat 1GB_10662266000.dat 1GB_10662266000.dat 1GB_10662266000.dat 1GB_10662266000.dat 1GB_10662266000.dat]

And the same for GNU/Linux: [1GB_10815303341.dat 1GB_10815305919.dat 1GB_10815306463.dat 1GB_10815306881.dat 1GB_10815307274.dat 1GB_10815307643.dat 1GB_10815308007.dat 1GB_10815308424.dat 1GB_10815308781.dat 1GB_10815309148.dat]

Would it be a good idea to replace time.Now().Nanosecond() with a function generating a UUID?