docs: data splitting for distributed training

Nixtla / nixtla

TimeGPT-1: production ready pre-trained Time Series Foundation Model for forecasting and anomaly detection. Generative pretrained transformer for time series trained on over 100B data points. It's capable of accurately predicting various domains such as retail, electricity, finance, and IoT with just a few lines of code 🚀.

https://docs.nixtla.io

Other

2.31k stars 187 forks source link

docs: data splitting for distributed training #530

Closed ngupta23 closed 1 week ago

ngupta23 commented 1 week ago

Description

It is great that TimeGPT supports distributed training. But it is not clear how the data splitting happens. For example

In Spark, how is the data gettig distributed when forecassting with Spark?
In Dask & Ray, the example shows partition = 2 with equal number of points. What if there are multiple IDs and it can not be split equally? How is the data split being handled under the hood.

Thanks!

Link

No response

elephaint commented 1 week ago

On the first page of Computing at Scale: