Enforce dataset new file msg verbosity

foundation-model-stack / fms-fsdp

🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash attention v2.

https://pytorch.org/docs/stable/fsdp.html

Apache License 2.0

116 stars 18 forks source link

Enforce dataset new file msg verbosity #43

Closed daviswer closed 3 months ago

daviswer commented 3 months ago

Previous StreamingDocDataloader passed a logging.INFO level message when opening new files in all cases, ignoring the verbosity arg in constructor. This PR fixes this bug.