legend-exp / pygama

Python package for data processing and analysis
https://pygama.readthedocs.io
GNU General Public License v3.0
18 stars 56 forks source link

DataLoader: handling of non-scalar data #448

Closed gracesong312 closed 1 year ago

gracesong312 commented 1 year ago

Main updates to functionality:

  1. WaveformTable (and general Table) loading
    • Used to: manually explode columns of a WaveformTable so that one WaveformTable got loaded as three Arrays (for dt, t0, and values)
    • Now: Load Tables as Tables, only explodes columns if the user requests a DataFrame, which it does in get_dataframe
  2. ArrayOfEqualSizedArray and VectorOfVector loading AOESA will get loaded as VoV because the size of an array across multiple channels may be different. Added a constructor to VoV that builds a VoV out of a list of lists in order to accommodate this.
  3. Support sortby in FileDB config Currently, FileDB sorts the dataframe by the key "timestamp," which is fine for L200, but test stands may not use a naming convention that includes "timestamp", so the user can now specify which key should be used for sorting. It still defaults to "timestamp" if no sortby is specified.

For testing, I'm not entirely sure what's failing in the codecov and docs, I really don't think I changed anything. Also, I've tested VoV loading separately, but there isn't really good VoV data in the test suite, so I haven't added that in the tests folder.

codecov-commenter commented 1 year ago

Codecov Report

Base: 45.94% // Head: 46.12% // Increases project coverage by +0.18% :tada:

Coverage data is based on head (2404383) compared to base (ece3b82). Patch coverage: 85.36% of modified lines in pull request are covered.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main legend-exp/pygama#448 +/- ## ========================================== + Coverage 45.94% 46.12% +0.18% ========================================== Files 93 93 Lines 10894 10924 +30 ========================================== + Hits 5005 5039 +34 + Misses 5889 5885 -4 ``` | [Impacted Files](https://codecov.io/gh/legend-exp/pygama/pull/448?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=legend-exp) | Coverage Δ | | |---|---|---| | [src/pygama/lgdo/vectorofvectors.py](https://codecov.io/gh/legend-exp/pygama/pull/448?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=legend-exp#diff-c3JjL3B5Z2FtYS9sZ2RvL3ZlY3Rvcm9mdmVjdG9ycy5weQ==) | `76.80% <81.48%> (+0.84%)` | :arrow_up: | | [src/pygama/lgdo/table.py](https://codecov.io/gh/legend-exp/pygama/pull/448?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=legend-exp#diff-c3JjL3B5Z2FtYS9sZ2RvL3RhYmxlLnB5) | `92.62% <83.33%> (-0.54%)` | :arrow_down: | | [src/pygama/flow/file\_db.py](https://codecov.io/gh/legend-exp/pygama/pull/448?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=legend-exp#diff-c3JjL3B5Z2FtYS9mbG93L2ZpbGVfZGIucHk=) | `74.26% <87.50%> (-0.10%)` | :arrow_down: | | [src/pygama/flow/data\_loader.py](https://codecov.io/gh/legend-exp/pygama/pull/448?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=legend-exp#diff-c3JjL3B5Z2FtYS9mbG93L2RhdGFfbG9hZGVyLnB5) | `80.94% <88.57%> (+1.75%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=legend-exp). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=legend-exp)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

gipert commented 1 year ago

pre-commit.ci run

gipert commented 1 year ago

The pre-commit.ci failure seems an upstream problem to me, let's see if they fix it by today.

jasondet commented 1 year ago

@gipert can you check the error?

jasondet commented 1 year ago

I figured it out.