Add a collate_lists_fn - Githubissues

uber / petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

Apache License 2.0

1.76k stars 281 forks source link

Add a collate_lists_fn #772

Open selitvin opened 1 year ago

selitvin commented 1 year ago

The function allows user to specify their own lists collate logic. Before the change we would always raise an error if length of lists in a single batch would not be the same (when using make_batch_reader)

CLAassistant commented 1 year ago

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

Yevgeni Litvin seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}