search

exasol / cloud-storage-extension

Exasol Cloud Storage Extension for accessing formatted data Avro, Orc and Parquet, on public cloud storage systems
MIT License
7 stars 11 forks source link

FilesMetadataEmitter reports wrong file count when importing multiple files #270

Closed kaklakariada closed 2 months ago

kaklakariada commented 10 months ago

When importing files from a folder (s3a://bucket/path/folder) or using a glob (s3a://bucket/path/file-0*), class FilesMetadataEmitter logs a wrong file count. The count is always one more than the actual file count:

[main] INFO com.exasol.cloudetl.emitter.FilesMetadataEmitter - Found total of '101' files in path 's3a://bucket/path*'.

When importing a single file, FilesMetadataEmitter logs correctly Found total of '1' files in path 's3a://bucket/path/file.parquet'.

### Tasks
- [ ] Reproduce the problem
Shmuma commented 2 months ago

Cannot reproduce this issue. Even on s3 bucket, connector shows proper amount of files in the logs