opendatacube / odc-stats

Statistician is a framework of tools for generating statistical summaries of large collections of EO data managed in an ODC instance.
Apache License 2.0
9 stars 4 forks source link

remove early dask unpack #101

Closed emmaai closed 10 months ago

emmaai commented 10 months ago

It's to test/address one of my theories on issue #100: the early unpack triggered something unexpected in dask graph.

My second theory is that the node id in dask graph is not unique when there are many chunks.

codecov[bot] commented 10 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (b4a4eee) 76.39% compared to head (1dc5369) 76.41%. Report is 2 commits behind head on develop.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## develop #101 +/- ## =========================================== + Coverage 76.39% 76.41% +0.02% =========================================== Files 34 34 Lines 3211 3210 -1 =========================================== Hits 2453 2453 + Misses 758 757 -1 ``` | [Files](https://app.codecov.io/gh/opendatacube/odc-stats/pull/101?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opendatacube) | Coverage Δ | | |---|---|---| | [odc/stats/proc.py](https://app.codecov.io/gh/opendatacube/odc-stats/pull/101?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opendatacube#diff-b2RjL3N0YXRzL3Byb2MucHk=) | `25.77% <ø> (+0.13%)` | :arrow_up: | | [tests/test\_sqs.py](https://app.codecov.io/gh/opendatacube/odc-stats/pull/101?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opendatacube#diff-dGVzdHMvdGVzdF9zcXMucHk=) | `100.00% <100.00%> (ø)` | |

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

emmaai commented 10 months ago

I’ve tested but I’m not sure if it solved the issue or it didn’t hit the odd. So I wanna test it in a significant large scale in the coming summary products 2015 reprocessing test.

On Thu, 26 Oct 2023 at 11:11 am, Damien Ayers @.***> wrote:

@.**** approved this pull request.

Sure, give it a go.

Why do you need to commit the change to test it out though?

— Reply to this email directly, view it on GitHub https://github.com/opendatacube/odc-stats/pull/101#pullrequestreview-1698539627, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLBWNTRZIVJVUWF52CQXETYBGWUBAVCNFSM6AAAAAA6OYNKL6VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTMOJYGUZTSNRSG4 . You are receiving this because you authored the thread.Message ID: @.***>