Fixups that simplify working with transforms and WeightedSamplingReader

uber / petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

Apache License 2.0

1.8k stars 284 forks source link

Fixups that simplify working with transforms and WeightedSamplingReader #436

Closed selitvin closed 5 years ago

selitvin commented 5 years ago

Don't fail transform if unknown fields are encountered. Warn instead.
Fixing ngram equivalence check in weighted_sampling_reader.
Adding ngram eq operator implementation