search

Azure / azure-storage-azcopy

The new Azure Storage data transfer utility - AzCopy v10
MIT License
613 stars 222 forks source link

Filtering out empty files in azure-storage-azcopy sync? #2472

Open parisiannoelia opened 11 months ago

parisiannoelia commented 11 months ago

AzCopy version: azcopy version 10.21.2 Platform: Mac

I am using azure-storage-azcopy sync to sync between source and destination. This already works ok so far but often I receive empty files on source folder and that also gets copied to destination. How can I check and filter out blank/empty files and skip them from being copied to destination azure storage?

Below command is the one am using so far:

azcopy sync "/sftp/sourceFolder" "https://mypersonal.blob.core.windows.net/payment" --recursive=false --include="PPP*.*.txt"

How can we avoid or skip syncing empty files from source to destination? Please help.

gapra-msft commented 11 months ago

Hi @parisiannoelia it is not currently possible to filter based on size of file. If your empty files have a naming pattern you could use exclude-pattern or exclude-regex CLI parameters.

parisiannoelia commented 11 months ago

Hi @parisiannoelia it is not currently possible to filter based on size of file. If your empty files have a naming pattern you could use exclude-pattern or exclude-regex CLI parameters.

No. There is no fixed pattern of empty files coming through as the source bucket is not going to be in my or my team's control.

mweibel commented 8 months ago

FTR this would be interesting in our use case as well - certain software we don't control write as a sort of a locking mechanism a 0 byte file first. Excluding those would be great.