Future of torchdata and dataloading

[laurencer]

As of July 2023, we have paused active development on TorchData and have paused new releases. We have learnt a lot from building it and hearing from users, but also believe we need to re-evaluate the technical design and approach given how much the industry has changed since we began the project. During the rest of 2023 we will be re-evaluating our plans in this space.

We want to hear from users on their use-cases and the pain-points they have (with data loading in general or torchdata specifically). Please reply on this issue to help inform our future roadmap.

[erip]

Thanks for the update, @laurencer. Does this mean torchdata as a domain library or the entire concept of datapipes/DataLoader2 is under review?

[laurencer]

The short answer is we need to look at both. More holistically there's lots of benefits with datapipes & DataloaderV2, however we've seen some limitations in a few use-cases which indicate we may need to tweak them a bit (or they're not the one-stop solution that we were hoping for). Overall the data loading space is really important and we hear about a lot of pain-points so we want to make sure we get the core abstractions right.

[josiahls]

I've personally liked the idea behind data pipes and the newer data loader. It'll be helpful to know what are some examples of use cases where this concept/API broke down?

I think this would be helpful for people about to jump into torchdata to know the breaking limitations, because it is not obvious to me at least.

[andrew-bydlon]

I too would like to hear what limitations you are referencing.

If it is performance oriented, I believe there's an argument. You could make something compatible with a compiled framework, especially something like torch.compile. Someone recently talked about speedups loading tar files with rust as an example.

My question is what you recommend as an alternative platform to torchdata for flexible and fast dataloading in pytorch?

[nairbv]

Does the design re-evaluation apply only to TorchData, or also to the portion of the datapipes API that was upstreamed to PyTorch core?

https://github.com/pytorch/pytorch/tree/main/torch/utils/data/datapipes

[BlueskyFR]

IMO the first thing that comes to mind is TensorFlow's tf.data.Dataset API, which is super cool to use. It is deeply integrated in the framework and with Keras, and the different operations you "pipe"/chain together can be fused at runtime so that the input pipeline is more optimized. However there are still some solutions such as NVIDIA DALI that is waaay faster if it applies to your use case.

I don't know TorchData in much details but I'd say building a nice-looking pipeline is easy, but making a pipeline optimized for high performance while still making it look cool is the real challenge. By cool-looking I mean nice/easy to read as code, but also easily extensible, like users could share "databricks" together or something.

I hope this helps!

[BlueskyFR]

Also, do you still recommend to use torchData in the meantime or will the compatibility with torch break at some point so we should avoid using it?

[rbavery]

Our ML team have been avid users of torchdata. We have used it to build datapipes that fetch large raster datasets from cloud providers to support training and inference. Currently we have a couple projects using torchdata, the most robust is https://zen3geo.readthedocs.io/en/latest/walkthrough.html a library managed by my colleague @weiji14 for fetching and batching large satellite images, with steps organized as functional datapipe ops.

While the API makes it easier to reuse custom data operations, we've been running into some consistent pain points when integrating datapipes with Dataloader V1 or Dataloader V2. We've had to switch back to Dataloader V1 and Datasets.

It doesn't seem like there is a clear set of documented rules for prefetching, shuffling, buffer sizes, memory pinning that results in good performance, or even any performance gains that beats single process dataloading when using torchdata with either Dataloader. All the configurations we have tried result in hangups, out of memory errors, or slower performance than a single process. It's also unclear how these parameters interact with different reading services.

I would love to see better docs and functionality for setting prefetching, shuffling, etc. with different kinds of reading services. Being able to profile datapipes and inspect RAM and cpu consumption of each operation would also be invaluable.

[biphasic]

I started to prototype a new version of my event data library on top of torchdata. The API is very clean and easy to understand, which is a strong plus point, even if it has a minor performance impact (I didn't verify that). I remember struggling with DataLoader2 and making multithreading work.

[sehoffmann]

Hey, thanks for the update. Does that mean that torchdata will become obsolete in the future?

As I already indicated in older issues, what I see as the biggest weakness right now, is the lack of control and flexibility with regards to:

• Shuffling
• Sharding
• Multiprocessing (dispatching to other processes etc.)

These things are right now tightly integrated into the torchdata core, and not easily accessible from user code. Giving User code the same Power and flexibility is paramount in my opinion to facilitate more complex pipelines than the vanilla cv pipeline. Or contrary, these functionalities should be implemented in user land without privileged handling from torchdata.

You can have a look at my repository (sehoffmann/atmodata) for some monkey patches that were necessary to facilitate my pipeline.

[npuichigo]

As for me, the ideal data pipeline should be ergonomic, flexible and efficient.

Chainable iterator already shows its power in iterator algorithm libraries like itertools and more-itertools. torchdata choose to enhance that with a functional programming API, that's good. Actually rust Iterator algorithms provide a good list for common used pipeline, besides Filter and Map, there're also FilterMap and FlatMap and so on.

But flexibilty still has room for improvement. At least, torchdata should be comparible to pypeln which has good flexibility to switch or mix thread/process/coroutine based tasks.

For performance, can torchdata be comparible with huggingface dataset? Can we easily leverage arrow or something else to build a high-performance data pipeline? It's better to show more benchmarks on production data.

[BarclayII]

DGL team is currently studying what should the UX be for scaling deep learning on graphs, namely the sampling strategies. More specifically, we want to support customization on

• Different graph storages (in-memory, disk, graph databases, etc.)
• Different node/edge feature storages (in-memory, disk, etc.)
• Sampling algorithms (online neighbor/subgraph sampling, offline sampling, etc.)
• Downstream tasks (node classification, link prediction with negative sampling, graph classification, etc.)
• Orchestration (whether to put sampling and feature fetching in multiprocessing/multithreading, how to schedule different stages, etc.).

So our current design depends on the composability of torchdata's DataPipes to allow for maximum extensibility, expressing the graph storages/feature storages/sampling algorithms/etc. as a composition of iterables and their transforms.

That being said, we are currently not pursuing active usages on DataLoader2 due to concerns on compatibility to existing packages depending on PyTorch DataLoader (e.g. PyTorch Lightning). That being said, we borrowed some ideas from ReadingService (namely the in-place editing of DataPipes).

We already have some demo in https://github.com/dmlc/dgl/tree/master/tests/python/pytorch/graphbolt.

Happy to discuss further.

[nairbv]

@npuichigo pypeln looks interesting. Based on there being multiple single-threaded queues between stages I assume this is designed for a single-node setup? PyTorch users would need multi-node support. To insert a queue between stages of a pipeline with multi-node stages, presumably we'd want to use some kind of purpose-built stand alone message queue. I'm not sure if that kind of setup is desirable -- once the training data reaches GPU hosts, I'd think we usually don't want to send it back elsewhere, so that architecture might make more sense for pre-processing.

@BarclayII

our current design depends on the composability of torchdata's DataPipes not pursuing active usages on DataLoader2

I'd like to use DataPipes for some NLP problems for similar reasons, and have some prototypes. I'd like to get confirmation, but from what I can tell it seems like it may only be TorchData that has paused development, whereas the DataPipes API is already part of PyTorch core.

For my use-cases I don't need DataLoader2 or readingservice/adapter. I think there are other ways to solve the problems addressed by those additional APIs -- I think there are ways to do it with just DataPipes that would address the concerns @sehoffmann raises around shuffling/sharding being "tightly integrated into the torchdata core, and not easily accessible from user code."

I also wonder whether we actually need a separate API focused on composability of datasets. The original dataset API could be used with composition too, and I'm not sure exactly what challenges we'd face in doing so. I know we wouldn't have the functional helper functions but that seems minor, and not sure what else we'd be missing.

[bryant1410]

however we've seen some limitations in a few use-cases which indicate we may need to tweak them a bit

@laurencer can you elaborate on which are those use cases?

[BarclayII]

@npuichigo I checked pypeln as well. It seems that the user needs to specify how to organize the queues in low-level (e.g. multiprocessing, multithreading, asyncio, etc.). Normally our UX shouldn't involve such a low-level specification unless the developers want to implement their own pipeline scheduling.

The original dataset API could be used with composition too, and I'm not sure exactly what challenges we'd face in doing so. I know we wouldn't have the functional helper functions but that seems minor, and not sure what else we'd be missing.

@nairbv Other than the functional helper functions, I find the in-place editing of DataPipe (namely torchdata.dataloader2.graph namespace) useful. For instance, with a single-processing DataPipe. I can have a DataLoader that changes the DataPipe for multi-processing and the process will be transparent to users. We also intend to apply the same idea for coordinate graph sampling, feature prefetching, and CPU-GPU transfer. https://github.com/dmlc/dgl/blob/master/python/dgl/graphbolt/dataloader.py#L58 shows an example.

Happy to discuss further.

[hhoeflin]

As for data pipelining solution, it would be nice if this could be developed without a dependency on a deep learning framework (torch, tensorflow etc),

[vincenttran-msft]

Thanks for the update @laurencer Laurence.

This is unfortunate news to hear as we over here at Microsoft's Developer Experience team have seen a lot of interest in cloud computing and seamless integration between Azure Storage and PyTorch from our customers. Our summer intern's project was building out a custom FileLoader and FileLister DataPipe that allowed easily interacting with datasets that are stored on Azure Storage, and so the news of a halt in development and an uncertain future of the torchdata repo makes for a difficult situation in regard to planning our future in terms of continuing development of integration with PyTorch.

With that being said, I am hopeful that this is a necessary step back in order to re-strategize and refine the future roadmap to ultimately end up with a better user experience for all. As the field of AI/ML continues to develop in the near future, it is really a matter of when (and not if) we will revisit building direct support for Azure Storage with PyTorch workflows, and so we will likely reach out sometime when the future of torchdata and dataloading is clearer. However, there are still some questions (and many were raised by previous posters above) that I would like to echo which would greatly help us developers in the interim of no new releases:

1. What are the expectations of compatibility / should we even consider continuing to build out torchdata support, or is that something that is becoming obsolete in the near future?
2. Any ETAs for when we will be updated on the status of the future of torchdata? While I understand this may be difficult, it would be great if any information could be shared in regard to timing to keep any developers from continuing to build out their infrastructure with torchdata, or if they should begin the migration process etc.

Thanks in advance, and please feel free to reach out if necessary!

[talmo]

1. What are the expectations of compatibility / should we even consider continuing to build out torchdata support, or is that something that is becoming obsolete in the near future?
2. Any ETAs for when we will be updated on the status of the future of torchdata? While I understand this may be difficult, it would be great if any information could be shared in regard to timing to keep any developers from continuing to build out their infrastructure with torchdata, or if they should begin the migration process etc.

+1, any updates on the roadmap since September?

[npuichigo]

I found an alternative to use Ray Data for data loading. It's framework-agnostic and performant, also providing a chainable API but with more fine-grained parallism control. I also write a tutorial to use together with HuggingFace Dataset here https://github.com/npuichigo/blazing-fast-io-tutorial.

[coufon]

I'd like to share our work https://github.com/google/space that supports materialized views in ML data pipelines. It ingests more metadata (versioning, column stats, logical plan) to ML datasets and pipelines, to provide a database/lakehouse like experience. Materialized views have the benefits of incrementally processing (and go back to old versions) and data lineage. Hope it will be useful.

[jaanli]

Hi @laurencer - any update on this?

[bhack]

I am also interested about the torchdata and datloading vision/roadmap related to audio/video API fragmentation https://github.com/pytorch/pytorch/issues/81102

[seanmcc-msft]

Any update here? My team is interested in creating an Azure Storage extension for PyTorch, similar to S3, but we cannot proceed with planning and implementation until we know what the future of PyTorch extensions will look like.

[bhack]

I don't know if @laurencer is still active on this or he is working for pytorch/Meta more in general. /cc @ejguan as he is the codeowner of pytorch dataloader.

[nairbv]

@bhack I know @ejguan isn't still on torchdata (I think he's in ads now)

[bhack]

@nairbv I think that the codeowners need to be updated https://github.com/pytorch/pytorch/blob/main/CODEOWNERS#L113-L114

[nairbv]

Agreed. I don't believe dataloader currently has a specific owner but I'm not still at Meta.

[bhack]

I hope it will be one of the topics at https://events.linuxfoundation.org/pytorch-conference/

[andrewkho]

Hi everyone, we’d like to share an update about how we plan to use the pytorch/data repo going forward. We will be focusing our efforts on a more iterative approach to Dataloader v1 (torch.utils.data), and we plan to use torchdata as a place to iterate on new features, such as the StatefulDataLoader. To avoid confusion in our support and offering, we will be deleting DataLoader2 and DataPipes from this repo. We plan to restart the torchdata release cycle, beginning with 0.8.0 in July 2024 (following PyTorch 2.4.0). DataLoader2 and DataPipes will be marked deprecated. In the October 0.9.0 release, they will no longer be part of the torchdata package, and existing users will need to either migrate away or pin to an older release such as 0.8.0. We welcome feedback to these plans, and will be responding in this Issue to any questions and concerns.

[zhitaoli]

Is it possible to publish some whitepaper / vision showcasing where the current steering committee's thought is, and consider an RFC process?

[talmo]

Thanks for the update @andrewkho!

1. Are there any places we learn more about the roadmap? A design doc for StatefulDataLoader or anything we can use to help inform our decisions on how to engineer future-oriented performant data pipelines? We've been in a bit of a holding pattern waiting for clarity before needing to sink tons of eng hours on reimplementing complex data pipelines...
2. What does the removal of DataPipes from this repo mean for the corresponding implementation in core torch (torch.util.data.datapipes)?

[josiahls]

@andrewkho Glad to hear work will restart on this project. Glaring concerns:

1.This plan talks about removing a bunch of stuff, especially datapipes

2. The plan fails to explain what they are being replaced with.
   • StatefulDataLoader looks like a very small part of a much bigger picture.
   • It would be helpful give a plan / vision of that is going to proceed.
   • It would be helpful to know what was the original problem with torchdata/datapipes so other's can learn/avoid whatever antipatterns/scalability issues/API issues caused this project to get paused and completely redone.

[bhack]

It would be nice in this roadmap also to consider how to coordinate/consolidate with some other projects in the ecosystem for their specific data domain. See https://github.com/pytorch/pytorch/issues/81102

[andrewkho]

Thanks everyone for the questions and comments. I'd like to acknowledge @josiahls 's comment that we are removing stuff without saying what we're replacing it with, which is completely fair criticism. We want to get feedback and see if there is significant pushback on something that we are sure about (deleting datapipes/dataloader2) as soon as possible.

Regarding roadmaps and plans, we're currently iterating on some ideas on how best to serve the community, but it's not quite in a state that is ready to discuss yet. I'll be able to share more in the next couple of months, please bear with us as we get our act together :)

Re: datapipes in torch.utils.data, we're going to be taking a hard look at that as well.

[talmo]

Thanks @andrewkho! We'll be looking forward to hearing more.

Just to +1 what has been pointed out before in this thread: the tf.data.Dataset is a majorly performant system to consider, allowing you to build IterDataPipe-style chains of transformations, each with autographed functions (tf.function) and auto-tunable level of parallelism. It's awesome to be able to control concurrency at each part of the data pipeline, and it makes for super composable data pipeline blocks.

[nairbv]

There are a couple of different approaches to stateful / savable datasets in IBM repos that may be of interest. These have been used to stop and restart long-running training jobs when training LLMs.

SavableDataset: https://github.com/foundation-model-stack/foundation-model-stack/blob/main/fms/datasets/util.py#L65

StatefulDataset: https://github.com/foundation-model-stack/fms-fsdp/blob/main/fms_fsdp/utils/dataset_utils.py#L68

[andrew-bydlon]

It's understandable to remove datapipes from this repo, as it is just a pass through to the torch version. I do like the functional datapipes approach, allowing for dp.someiterable nomenclature. I hope it can be ported.

For data loader 2, is there a concrete idea for how to do a data sampler approach in dl1? I typically don't have a length in iterable datapipes, or they are infinite cycles. I'm unaware if a standard was created. The analogous reading services for distributed and multiprocessing are very nice here.

Glad to hear it is coming back in some form. To me, it is a useful repo for datapipes and accessing cloud providers (e.g. S3). I wonder what the big vision is after removing dp and dl2 🤣

[jaanli]

We have been using dbt: @dbt-labs helps us transform our pre-train data and upload it automatically to S3 for streaming - I think a similar pipeline should be feasible for DataLoader2-type tasks? Not sure...

Here's an example for health care data: https://github.com/onefact/healthcare-data/

[rgtjf]

StatefulDataLoader may be a prerequisite for stateful data pipes.

[andrewkho]

Hi everyone, our roadmap for the second half of the 2024 is available publicly alongside many other PyTorch efforts and I wanted to share this here as well in case you haven't seen it yet. The KRs are subject to change as we learn more, and there is still more work to do to get clarity on the overall effort. If you have feedback or thoughts, we're very interested to hear from you folks :)

Also another update, we'll be at PyTorch Conference in San Francisco this September, if anyone is attending we'd be happy to meet up and talk about anything!

[josiahls]

@andrewkho Thanks for the update! So if I am understanding correctly:

The replacement for torchdata datapipes themselves might look something like:

• huggingface datasets (which actually are built on apache arrow what it looks like?) and others such as WebDatasets, Mosaic, Ray, Nvidia DALI).
• torchdata will attempt using learnings from what huggingface et al have implemented and move the basic functionality into a core API.
• Problem with original datapipe API was it did not get buy in from the larger OSS space.

Using huggingface API as an example (there could be other examples to pull from though):

I have only worked a little bit with huggingface datasets. From going through their documentation and my memory working with them. Maybe someone more knowledgeable can enlighten me:

Pros:

• Huggingface dataset API is intended as a scaffold for custom pipelines.
• Pretty good docs.
• Core methods / functions / processes being just methods of a common object are easier to step through when debugging compared to datapipes.
• Custom functions such as transforms are just input / output callable.

Cons (could be my ignorance of what hg datasets can do):

• Custom core behaviors for shuffling / sharding / dataloading / mapping can only be done via inheritance.
  • Personal taste inheritance is hard to reason with when objects have many abstractions.
  • Was a fan of the datapipe API being very horizontal. The question of "what does this pipe do?" could be answered (generally) on a single page and a single object, as opposed to jump through several hierarchies or scrolling for 1000s of lines of code.
• Transforms with internal state / params would need to be separate callable objects that are instantiated separately and then passed to the .map. This is as opposed to just using a custom datapipe.
  • Additionally transforms that need multiple inputs or generate multiple outputs would need a dataset subclass
• In general "scaffolds" can be limiting, and trying to customize them isn't fun. Datapipes in the end were just iterators, so doing "weird" stuff was more straightforward.
• Played around with making an RL framework in torchdata which was doable since the API itself was pretty horizontal. Trying to make more hierarchical pipeline frameworks work for RL was difficult.

These cons aren't a condemnation of the decision to look into HuggingFace Datasets, WebDatasets, Mosaic, Ray, Nvidia DALI though since they are popular frameworks for a reason. I'm looking forward to seeing how torchdata leverages their work and make it more flexible. As for me personally, I'll also look at HG and try out their API (for transforms / multimodel / RL training) see if the cons above aren't really cons at all.

edited: Make it more clear in my comment that torchdata plans to generally look at the space of dataloading / datasets, not just huggingface specifically.

[andrewkho]

@josiahls we definitely do not want to force users to take dependencies on HuggingFace which is more NLP focused. We do want to ensure we're compatible and working well together, since they depend on PyTorch and LLM fine-tuning is such an important use case today. However the use cases we're supporting are more broad

[npuichigo]

@andrewkho Please also take litdata into consideration

[mfbalin]

@andrewkho Are the datapipes in torch.utils.data going to be removed as well?

[frozenbugs]

Are you going to remove the datapipe related code from torch.util.data? If so, can you announce the plan in advance and give us several months to take action?

[KBlansit]

I'd like to chime in and say that it is very unfortunate that the official online documentation make almost no mention of this planned deprecation. In fact, I only was made aware of this planned of this issue this morning when I updated my version of torchdata.

Adding a banner warning of deprecation and linking to this github issue would better communicate to ML developers the current status of this project. Keeping the official documents up to date with the current project status is an important issue as otherwise developers may be unaware and waste crucial time and effort into projects that will not be supported in the future.

[keunwoochoi]

as a user of torchdata, i was very happy to see the resurrection of the project.

i have a question about the development plan. from the README, i see:

torchdata repo to be an iterative enhancement of torch.utils.data.DataLoader

this is somewhat surprising. although the current Datapipes seem to have various issues underneath the shell, so far, Datapipes ARE torchdata. the current API reference:

API Reference:

Stateful DataLoader Iterable-style DataPipes Map-style DataPipes Utility Functions DataLoader2 ReadingService

and this is it; i.e., until ver 0.7, torchdata == the datapipes and other necessary utilities (dataloader2 and reading service).

that's why it is surprising for me, that while the development of torchdata has re-started, it is being done in a way it discards everything it had.

so, can i ask for a bit more details about what the new direction (enhancement of torch.utils.data.DataLoader)? or am i missing something here?

thanks.

[keunwoochoi]

to add more context: i love datapipes and want them improved, not deleted.

---

although alternatives seem to exist, none of these (HuggingFace Datasets, WebDatasets, Mosaic, Ray, Nvidia DALI) are a general-purpose, scalable data loading solution that is native to torch i.e., expected to be compatible with other libraries such as Accelerator, Lightning, etc. E.g., MosaicML dataset is not compatible with Lightning's multi-node training. in short, none of them enabled us to load multimodal data from remote (s3 buckets) with multi-node training. datapipes are the only working solution. even if they are somehow compatible in some cases, they are not expected so. they can always become incompatible with something they're not responsible for.

why then the low adoption of datapipes? i believe, perhaps for these reasons, based on my experience.

• lack of documentation and examples.
• difficulty in debugging and profiling.
  • when it's slow, it's really hard to figure out why
  • with some unexpected timeout, i had to monitor & re-start the training quite often.
• lack of automatic optimization such as tf.data.AUTOTUNE (yes.. tensorflow..)

even so, i still believe

• the API is great! functional datapipes for loading and .map() with a distributed setup.. what else?
• 100% that this should be done on torch, natively
• improving it would lead to much happier and wider adoption.

[andrewkho]

I'd like to chime in and say that it is very unfortunate that the official online documentation make almost no mention of this planned deprecation. In fact, I only was made aware of this planned of this issue this morning when I updated my version of torchdata.

Adding a banner warning of deprecation and linking to this github issue would better communicate to ML developers the current status of this project. Keeping the official documents up to date with the current project status is an important issue as otherwise developers may be unaware and waste crucial time and effort into projects that will not be supported in the future.

Thanks @KBlansit for calling out the gap in notice in that document, we'll be updating that this week. cc @gokulavasan

[andrewkho]

as a user of torchdata, i was very happy to see the resurrection of the project.

i have a question about the development plan. from the README, i see:

torchdata repo to be an iterative enhancement of torch.utils.data.DataLoader

this is somewhat surprising. although the current Datapipes seem to have various issues underneath the shell, so far, Datapipes ARE torchdata. the current API reference:

API Reference: Stateful DataLoader Iterable-style DataPipes Map-style DataPipes Utility Functions DataLoader2 ReadingService

and this is it; i.e., until ver 0.7, torchdata == the datapipes and other necessary utilities (dataloader2 and reading service).

that's why it is surprising for me, that while the development of torchdata has re-started, it is being done in a way it discards everything it had.

so, can i ask for a bit more details about what the new direction (enhancement of torch.utils.data.DataLoader)? or am i missing something here?

thanks.

Hi @keunwoochoi , thanks for the feedback! Unfortunately we aren't quite ready to share any more details about the new direction quite yet (we are actively iterating on this), but we do really appreciate the feedback and will update this thread as soon as we can.