Using the new tables_io functions for the parquet file iterator

[joselotl]

Problem & Solution Description (including issue #)

Code Quality

• [x] My code follows the code style of this project
• [x] I have written unit tests or justified all instances of #pragma: no cover; in the case of a bugfix, a new test that breaks as a result of the bug has been added
• [ ] My code contains relevant comments and necessary documentation for future maintainers; the change is reflected in applicable demos/tutorials (with output cleared!) and added/updated docstrings use the NumPy docstring format
• [ ] Any breaking changes, described above, are accompanied by backwards compatibility and deprecation warnings

[joselotl]

@sidneymau If you have time, maybe you could try to run directly one estimator using directly your parquet files to see if this fix works.

[eacharles]

I think that tables_io.io.getInputDataLength just decides function to call based on the type of data. I.e. it calls tables_io.io.getInputDataLengthPq or Hdf5, or whatever.On May 1, 2024, at 1:58 AM, hangqianjun @.***> wrote: @hangqianjun commented on this pull request.

In src/rail/core/data.py:

@@ -367,6 +367,9 @@ class PqHandle(TableHandle):

 suffix = "pq"

• @classmethod
• def _size(cls, path, **kwargs):
• return tables_io.io.getInputDataLengthPq(path, **kwargs) Looks good. One question: it seems tables_io.io.getInputDataLength covers all hdf5 and parquet files, but why do we still need tables_io.io.getInputDataLengthPq specifically on line 372?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because your review was requested.Message ID: @.***>

[hangqianjun]

I think that tables_io.io.getInputDataLength just decides function to call based on the type of data. I.e. it calls tables_io.io.getInputDataLengthPq or Hdf5, or whatever.

Yes exactly. I thought getInputDataLength already has the functionality so we can use it instead of getInputDataLengthPq. However it's no big issue, as the code still works! It just made me go look up the difference between the two when I saw this, and that's why I raised the question! Happy to approve the PR.

[eacharles]

So, to be clear, if you know what type of data you have you should probably call the appropriate function directly.   But if you aren’t sure, then you should call the generic function.-eOn May 1, 2024, at 8:48 AM, hangqianjun @.***> wrote:

I think that tables_io.io.getInputDataLength just decides function to call based on the type of data. I.e. it calls tables_io.io.getInputDataLengthPq or Hdf5, or whatever.

Yes exactly. I thought getInputDataLength already has the functionality so we can use it instead of getInputDataLengthPq. However it's no big issue, as the code still works! It just made me go look up the difference between the two when I saw this, and that's why I raised the question! Happy to approve the PR.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because your review was requested.Message ID: @.***>

[hangqianjun]

So, to be clear, if you know what type of data you have you should probably call the appropriate function directly.   But if you aren’t sure, then you should call the generic function.-e

I see! Thanks for the explanation!