awslabs / sagemaker-debugger

Amazon SageMaker Debugger provides functionality to save tensors during training of machine learning jobs and analyze those tensors
Apache License 2.0
158 stars 82 forks source link

New version of the numpy_reader, previous branch closed by mistake #625

Closed mariumof closed 1 year ago

mariumof commented 1 year ago

Description of changes:

On top of the previous version (numpy_reader), added forward accumulation of small files, if asked for.

Style and formatting:

I have run pre-commit install && pre-commit run --all-files to ensure that auto-formatting happens with every commit.

Issue number, if available

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

codecov-commenter commented 1 year ago

Codecov Report

Base: 73.89% // Head: 53.81% // Decreases project coverage by -20.08% :warning:

Coverage data is based on head (aaf533d) compared to base (2caacc2). Patch coverage: 8.29% of modified lines in pull request are covered.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #625 +/- ## =========================================== - Coverage 73.89% 53.81% -20.09% =========================================== Files 127 117 -10 Lines 11268 11068 -200 =========================================== - Hits 8327 5956 -2371 - Misses 2941 5112 +2171 ``` | [Impacted Files](https://codecov.io/gh/awslabs/sagemaker-debugger/pull/625?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs) | Coverage Δ | | |---|---|---| | [smdebug/profiler/\_\_init\_\_.py](https://codecov.io/gh/awslabs/sagemaker-debugger/pull/625/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c21kZWJ1Zy9wcm9maWxlci9fX2luaXRfXy5weQ==) | `100.00% <ø> (ø)` | | | [smdebug/profiler/system\_metrics\_reader.py](https://codecov.io/gh/awslabs/sagemaker-debugger/pull/625/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c21kZWJ1Zy9wcm9maWxlci9zeXN0ZW1fbWV0cmljc19yZWFkZXIucHk=) | `11.44% <8.01%> (-87.26%)` | :arrow_down: | | [smdebug/core/access\_layer/s3handler.py](https://codecov.io/gh/awslabs/sagemaker-debugger/pull/625/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c21kZWJ1Zy9jb3JlL2FjY2Vzc19sYXllci9zM2hhbmRsZXIucHk=) | `67.76% <10.00%> (-1.67%)` | :arrow_down: | | [smdebug/\_version.py](https://codecov.io/gh/awslabs/sagemaker-debugger/pull/625/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c21kZWJ1Zy9fdmVyc2lvbi5weQ==) | `100.00% <100.00%> (ø)` | | | [smdebug/pytorch/\_\_init\_\_.py](https://codecov.io/gh/awslabs/sagemaker-debugger/pull/625/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c21kZWJ1Zy9weXRvcmNoL19faW5pdF9fLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [smdebug/profiler/analysis/utils/merge\_timelines.py](https://codecov.io/gh/awslabs/sagemaker-debugger/pull/625/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c21kZWJ1Zy9wcm9maWxlci9hbmFseXNpcy91dGlscy9tZXJnZV90aW1lbGluZXMucHk=) | `0.00% <0.00%> (-97.52%)` | :arrow_down: | | [...ug/profiler/analysis/utils/pandas\_data\_analysis.py](https://codecov.io/gh/awslabs/sagemaker-debugger/pull/625/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c21kZWJ1Zy9wcm9maWxlci9hbmFseXNpcy91dGlscy9wYW5kYXNfZGF0YV9hbmFseXNpcy5weQ==) | `0.00% <0.00%> (-97.37%)` | :arrow_down: | | [smdebug/profiler/analysis/python\_stats\_reader.py](https://codecov.io/gh/awslabs/sagemaker-debugger/pull/625/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c21kZWJ1Zy9wcm9maWxlci9hbmFseXNpcy9weXRob25fc3RhdHNfcmVhZGVyLnB5) | `0.00% <0.00%> (-94.29%)` | :arrow_down: | | [smdebug/pytorch/collection.py](https://codecov.io/gh/awslabs/sagemaker-debugger/pull/625/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c21kZWJ1Zy9weXRvcmNoL2NvbGxlY3Rpb24ucHk=) | `0.00% <0.00%> (-90.91%)` | :arrow_down: | | [...debug/profiler/analysis/python\_profile\_analysis.py](https://codecov.io/gh/awslabs/sagemaker-debugger/pull/625/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c21kZWJ1Zy9wcm9maWxlci9hbmFseXNpcy9weXRob25fcHJvZmlsZV9hbmFseXNpcy5weQ==) | `0.00% <0.00%> (-90.91%)` | :arrow_down: | | ... and [56 more](https://codecov.io/gh/awslabs/sagemaker-debugger/pull/625/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs) | | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.