uber / petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

Apache License 2.0

1.78k stars 285 forks source link

Bug/weighted sampling reader #626

Closed gueguenster closed 3 years ago

gueguenster commented 3 years ago

petastorm.pytorch.DataLoader expects the field _is_batchedreader in the reader. WeightedSamplingReader does not have this field. It seems to be replaced by the property: _batchedoutput.
petastorm.pytorch.DataLoader expects the functions stop() and join() from the reader. WeightedSamplingReader does not have these functions. This PR adds these functions.

CLAassistant commented 3 years ago

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

Robert Gruener seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

codecov[bot] commented 3 years ago

Codecov Report

Merging #626 (e458546) into master (f8c427c) will increase coverage by 0.01%. The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #626      +/-   ##
==========================================
+ Coverage   85.31%   85.32%   +0.01%     
==========================================
  Files          85       85              
  Lines        4929     4933       +4     
  Branches      783      783              
==========================================
+ Hits         4205     4209       +4     
  Misses        584      584              
  Partials      140      140

Impacted Files	Coverage Δ
petastorm/pytorch.py	`90.72% <100.00%> (ø)`
petastorm/weighted_sampling_reader.py	`84.78% <100.00%> (+1.44%)`	:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update f8c427c...e458546. Read the comment docs.

selitvin commented 3 years ago

@gueguenster , can you please sign the CLA?

selitvin commented 3 years ago

@gueguenster : seems that the problem is not CLA but that we have Robert appears as author of one of the unrelated comments in the commit stack. Can you please rebase and remove Robert's commits + push-force?

gueguenster commented 3 years ago

@selitvin , thanks I look in fixing that.

gueguenster commented 3 years ago

I had difficulty removing the commits.