TFRecord to support S3 index URIs - Githubissues

NVIDIA / DALI

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

https://docs.nvidia.com/deeplearning/dali/user-guide/docs/index.html

Apache License 2.0

5.09k stars 615 forks source link

TFRecord to support S3 index URIs #5515

Closed jantonguirao closed 3 months ago

jantonguirao commented 3 months ago

Category:

Bug fix

Description:

TFRecord index file didn't use the FileStream:Open abstraction, therefore it did not support S3 storage like advertised.
Fixes https://github.com/NVIDIA/DALI/issues/5513
Added a FileStreamBuf abstraction (useful to use with std::istream)

Additional information:

Affected modules and functionalities:

Indexed file loader (TFRecord reader)

Key points relevant for the review:

Tests:

[ ] Existing tests apply
[ ] New tests added
- [ ] Python tests
- [ ] GTests
- [ ] Benchmark
- [ ] Other
[x] N/A Need to figure out the S3 storage infrastructure for testing first.

Checklist

Documentation

[ ] Existing documentation applies
[ ] Documentation updated
- [ ] Docstring
- [ ] Doxygen
- [ ] RST
- [ ] Jupyter
- [ ] Other
[x] N/A

DALI team only

Requirements

[ ] Implements new requirements
[ ] Affects existing requirements
[x] N/A

REQ IDs: N/A

JIRA TASK: DALI-3995

dali-automaton commented 3 months ago

CI MESSAGE: [15735775]: BUILD STARTED

dali-automaton commented 3 months ago

CI MESSAGE: [15735775]: BUILD FAILED

dali-automaton commented 3 months ago

CI MESSAGE: [15741123]: BUILD STARTED

dali-automaton commented 3 months ago

CI MESSAGE: [15741123]: BUILD PASSED