Upgrade to V1

[rettigl]

Collection PR for the update to V1. This contains breaking changes to current behavior and config file layouts.

Items to be addressed here:

• [ ] Upgrade to Py3.9-3.12 #368
• [x] Energy calibration from bias values #411
• [x] implement logging #249 #490
• [x] change binning factors from n to 2^n #173 #451
• [x] move num_cores from binning to core #404 #451
• [x] Unify axis naming #138
• [ ] Unify config scheme (pydantic)
• [x] FlashLoader update/rewrite #329
• [ ] Change version to 1.0.0

[coveralls]

Pull Request Test Coverage Report for Build 9635884851

Details

• 182 of 183 (99.45%) changed or added relevant lines in 41 files are covered.
• 7 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-0.1%) to 91.857%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
sed/calibrator/momentum.py	13	14	92.86%
<!--	Total:	182	183	99.45%	-->

Files with Coverage Reduction	New Missed Lines	%
sed/core/metadata.py	1	93.22%
sed/calibrator/momentum.py	6	86.78%
<!--	Total:	7		-->

Totals
Change from base Build 9635820675:	-0.1%
Covered Lines:	6430
Relevant Lines:	7000

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 9635881058

Details

• 182 of 183 (99.45%) changed or added relevant lines in 41 files are covered.
• 7 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-0.1%) to 91.857%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
sed/calibrator/momentum.py	13	14	92.86%
<!--	Total:	182	183	99.45%	-->

Files with Coverage Reduction	New Missed Lines	%
sed/core/metadata.py	1	93.22%
sed/calibrator/momentum.py	6	86.78%
<!--	Total:	7		-->

Totals
Change from base Build 9635820675:	-0.1%
Covered Lines:	6430
Relevant Lines:	7000

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 9635919510

Details

• 182 of 183 (99.45%) changed or added relevant lines in 41 files are covered.
• 7 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-0.1%) to 91.857%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
sed/calibrator/momentum.py	13	14	92.86%
<!--	Total:	182	183	99.45%	-->

Files with Coverage Reduction	New Missed Lines	%
sed/core/metadata.py	1	93.22%
sed/calibrator/momentum.py	6	86.78%
<!--	Total:	7		-->

Totals
Change from base Build 9635820675:	-0.1%
Covered Lines:	6430
Relevant Lines:	7000

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 9636056013

Details

• 182 of 183 (99.45%) changed or added relevant lines in 41 files are covered.
• 7 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-0.1%) to 91.857%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
sed/calibrator/momentum.py	13	14	92.86%
<!--	Total:	182	183	99.45%	-->

Files with Coverage Reduction	New Missed Lines	%
sed/core/metadata.py	1	93.22%
sed/calibrator/momentum.py	6	86.78%
<!--	Total:	7		-->

Totals
Change from base Build 9636055471:	-0.1%
Covered Lines:	6430
Relevant Lines:	7000

---

💛 - Coveralls

[rettigl]

Flash-loader is consistently slower with the new packages. Not sure why.

[coveralls]

Pull Request Test Coverage Report for Build 9636181156

Details

• 182 of 183 (99.45%) changed or added relevant lines in 41 files are covered.
• 7 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-0.1%) to 91.857%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
sed/calibrator/momentum.py	13	14	92.86%
<!--	Total:	182	183	99.45%	-->

Files with Coverage Reduction	New Missed Lines	%
sed/core/metadata.py	1	93.22%
sed/calibrator/momentum.py	6	86.78%
<!--	Total:	7		-->

Totals
Change from base Build 9636055471:	-0.1%
Covered Lines:	6430
Relevant Lines:	7000

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 9636181338

Details

• 182 of 183 (99.45%) changed or added relevant lines in 41 files are covered.
• 7 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-0.1%) to 91.857%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
sed/calibrator/momentum.py	13	14	92.86%
<!--	Total:	182	183	99.45%	-->

Files with Coverage Reduction	New Missed Lines	%
sed/core/metadata.py	1	93.22%
sed/calibrator/momentum.py	6	86.78%
<!--	Total:	7		-->

Totals
Change from base Build 9636055471:	-0.1%
Covered Lines:	6430
Relevant Lines:	7000

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 9636221019

Details

• 182 of 183 (99.45%) changed or added relevant lines in 41 files are covered.
• 10 unchanged lines in 3 files lost coverage.
• Overall coverage decreased (-0.1%) to 91.814%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
sed/calibrator/momentum.py	13	14	92.86%
<!--	Total:	182	183	99.45%	-->

Files with Coverage Reduction	New Missed Lines	%
sed/core/metadata.py	1	93.22%
sed/binning/numba_bin.py	3	87.5%
sed/calibrator/momentum.py	6	86.78%
<!--	Total:	10		-->

Totals
Change from base Build 9636055471:	-0.1%
Covered Lines:	6427
Relevant Lines:	7000

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 9636258361

Details

• 182 of 183 (99.45%) changed or added relevant lines in 41 files are covered.
• 5 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-0.08%) to 91.886%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
sed/calibrator/momentum.py	13	14	92.86%
<!--	Total:	182	183	99.45%	-->

Files with Coverage Reduction	New Missed Lines	%
sed/core/metadata.py	1	93.22%
sed/calibrator/momentum.py	4	87.03%
<!--	Total:	5		-->

Totals
Change from base Build 9636055471:	-0.08%
Covered Lines:	6432
Relevant Lines:	7000

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 9636290716

Details

• 182 of 183 (99.45%) changed or added relevant lines in 41 files are covered.
• 7 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-0.1%) to 91.857%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
sed/calibrator/momentum.py	13	14	92.86%
<!--	Total:	182	183	99.45%	-->

Files with Coverage Reduction	New Missed Lines	%
sed/core/metadata.py	1	93.22%
sed/calibrator/momentum.py	6	86.78%
<!--	Total:	7		-->

Totals
Change from base Build 9636055471:	-0.1%
Covered Lines:	6430
Relevant Lines:	7000

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 9636630541

Details

• 182 of 183 (99.45%) changed or added relevant lines in 41 files are covered.
• 7 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-0.1%) to 91.857%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
sed/calibrator/momentum.py	13	14	92.86%
<!--	Total:	182	183	99.45%	-->

Files with Coverage Reduction	New Missed Lines	%
sed/core/metadata.py	1	93.22%
sed/calibrator/momentum.py	6	86.78%
<!--	Total:	7		-->

Totals
Change from base Build 9636624947:	-0.1%
Covered Lines:	6430
Relevant Lines:	7000

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 9636663837

Details

• 182 of 183 (99.45%) changed or added relevant lines in 41 files are covered.
• 10 unchanged lines in 3 files lost coverage.
• Overall coverage decreased (-0.1%) to 91.814%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
sed/calibrator/momentum.py	13	14	92.86%
<!--	Total:	182	183	99.45%	-->

Files with Coverage Reduction	New Missed Lines	%
sed/core/metadata.py	1	93.22%
sed/binning/numba_bin.py	3	87.5%
sed/calibrator/momentum.py	6	86.78%
<!--	Total:	10		-->

Totals
Change from base Build 9636624947:	-0.1%
Covered Lines:	6427
Relevant Lines:	7000

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 9636664048

Details

• 182 of 183 (99.45%) changed or added relevant lines in 41 files are covered.
• 7 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-0.1%) to 91.857%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
sed/calibrator/momentum.py	13	14	92.86%
<!--	Total:	182	183	99.45%	-->

Files with Coverage Reduction	New Missed Lines	%
sed/core/metadata.py	1	93.22%
sed/calibrator/momentum.py	6	86.78%
<!--	Total:	7		-->

Totals
Change from base Build 9636624947:	-0.1%
Covered Lines:	6430
Relevant Lines:	7000

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 9636700237

Details

• 182 of 183 (99.45%) changed or added relevant lines in 41 files are covered.
• 7 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-0.1%) to 91.857%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
sed/calibrator/momentum.py	13	14	92.86%
<!--	Total:	182	183	99.45%	-->

Files with Coverage Reduction	New Missed Lines	%
sed/core/metadata.py	1	93.22%
sed/calibrator/momentum.py	6	86.78%
<!--	Total:	7		-->

Totals
Change from base Build 9636624947:	-0.1%
Covered Lines:	6430
Relevant Lines:	7000

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 9636700470

Details

• 182 of 183 (99.45%) changed or added relevant lines in 41 files are covered.
• 7 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-0.1%) to 91.857%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
sed/calibrator/momentum.py	13	14	92.86%
<!--	Total:	182	183	99.45%	-->

Files with Coverage Reduction	New Missed Lines	%
sed/core/metadata.py	1	93.22%
sed/calibrator/momentum.py	6	86.78%
<!--	Total:	7		-->

Totals
Change from base Build 9636624947:	-0.1%
Covered Lines:	6430
Relevant Lines:	7000

---

💛 - Coveralls

[rettigl]

Flash-loader is consistently slower with the new packages. Not sure why.

This directly comes from the update of dask from 2023.5.0 to 2023.12.0, tested on python 3.9. Surprisingly, the binning in the Flash example notebook however becomes faster with this new configuration (44.5s vs. 26.9 s). As we need to look into dask anyways still, I would not consider this a problem for now.

[zain-sohail]

Flash-loader is consistently slower with the new packages. Not sure why.

This directly comes from the update of dask from 2023.5.0 to 2023.12.0, tested on python 3.9. Surprisingly, the binning in the Flash example notebook however becomes faster with this new configuration (44.5s vs. 26.9 s). As we need to look into dask anyways still, I would not consider this a problem for now.

Are you using the refactored branch or the one on main?

Considering that the creation of buffer files has is independant of dask, I assume it's another package that causes this issue (maybe pandas?) Dask is only used when loading the buffer files and subsequent processing like sector ID seperation, forward filling, etc. and eventually binning. So it seems to be that the problem is not from dask. At least this is the case in refactored branch. In main, the sector ID seperation takes place with pandas and not dask.

[rettigl]

Dask is only used when loading the buffer files and subsequent processing like sector ID seperation, forward filling, etc. and eventually binning. So it seems to be that the problem is not from dask.

It is from dask, the performance changes if I only change the version of dask (from 2024.12.0 to 2024.5.0). There seem to be some other issues with the map_overlap stuff after loading the dataframe. see #448

[coveralls]

Pull Request Test Coverage Report for Build 9756916343

Details

• 884 of 911 (97.04%) changed or added relevant lines in 49 files are covered.
• 2 unchanged lines in 2 files lost coverage.
• Overall coverage increased (+0.6%) to 92.462%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
sed/calibrator/momentum.py	13	14	92.86%
sed/loader/flash/buffer_handler.py	81	82	98.78%
sed/loader/flash/utils.py	45	46	97.83%
sed/loader/sxp/loader.py	16	17	94.12%
tests/loader/flash/conftest.py	28	29	96.55%
sed/loader/flash/instruments.py	3	6	50.0%
sed/loader/utils.py	32	35	91.43%
sed/loader/flash/dataframe.py	85	90	94.44%
sed/loader/flash/loader.py	71	76	93.42%
tests/loader/test_loaders.py	18	24	75.0%
<!--	Total:	884	911	97.04%	-->

Files with Coverage Reduction	New Missed Lines	%
sed/calibrator/momentum.py	1	87.03%
sed/core/metadata.py	1	93.33%
<!--	Total:	2		-->

Totals
Change from base Build 9756700337:	0.6%
Covered Lines:	6869
Relevant Lines:	7429

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 9820936310

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 899 of 926 (97.08%) changed or added relevant lines in 49 files are covered.
• 2 unchanged lines in 2 files lost coverage.
• Overall coverage increased (+0.6%) to 92.469%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
sed/calibrator/momentum.py	13	14	92.86%
sed/loader/flash/buffer_handler.py	81	82	98.78%
sed/loader/flash/utils.py	45	46	97.83%
sed/loader/sxp/loader.py	16	17	94.12%
tests/loader/flash/conftest.py	28	29	96.55%
sed/loader/flash/instruments.py	3	6	50.0%
sed/loader/utils.py	32	35	91.43%
sed/loader/flash/dataframe.py	85	90	94.44%
sed/loader/flash/loader.py	71	76	93.42%
tests/loader/test_loaders.py	18	24	75.0%
<!--	Total:	899	926	97.08%	-->

Files with Coverage Reduction	New Missed Lines	%
sed/calibrator/momentum.py	1	87.03%
sed/core/metadata.py	1	93.33%
<!--	Total:	2		-->

Totals
Change from base Build 9756700337:	0.6%
Covered Lines:	6876
Relevant Lines:	7436

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 9820936107

Details

• 899 of 926 (97.08%) changed or added relevant lines in 49 files are covered.
• 2 unchanged lines in 2 files lost coverage.
• Overall coverage increased (+0.6%) to 92.469%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
sed/calibrator/momentum.py	13	14	92.86%
sed/loader/flash/buffer_handler.py	81	82	98.78%
sed/loader/flash/utils.py	45	46	97.83%
sed/loader/sxp/loader.py	16	17	94.12%
tests/loader/flash/conftest.py	28	29	96.55%
sed/loader/flash/instruments.py	3	6	50.0%
sed/loader/utils.py	32	35	91.43%
sed/loader/flash/dataframe.py	85	90	94.44%
sed/loader/flash/loader.py	71	76	93.42%
tests/loader/test_loaders.py	18	24	75.0%
<!--	Total:	899	926	97.08%	-->

Files with Coverage Reduction	New Missed Lines	%
sed/calibrator/momentum.py	1	87.03%
sed/core/metadata.py	1	93.33%
<!--	Total:	2		-->

Totals
Change from base Build 9756700337:	0.6%
Covered Lines:	6876
Relevant Lines:	7436

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 10949056106

Details

• 1520 of 1622 (93.71%) changed or added relevant lines in 51 files are covered.
• 2 unchanged lines in 2 files lost coverage.
• Overall coverage increased (+0.6%) to 92.522%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
sed/core/logging.py	37	38	97.37%
sed/loader/flash/buffer_handler.py	116	117	99.15%
sed/loader/flash/utils.py	46	47	97.87%
sed/loader/utils.py	36	37	97.3%
tests/loader/flash/conftest.py	34	35	97.14%
sed/loader/base/loader.py	11	13	84.62%
tests/loader/flash/test_buffer_handler.py	111	113	98.23%
sed/calibrator/delay.py	31	34	91.18%
sed/loader/flash/instruments.py	3	6	50.0%
sed/loader/sxp/loader.py	27	32	84.38%
<!--	Total:	1520	1622	93.71%	-->

Files with Coverage Reduction	New Missed Lines	%
sed/calibrator/energy.py	1	92.6%
sed/core/metadata.py	1	93.33%
<!--	Total:	2		-->

Totals
Change from base Build 10936420583:	0.6%
Covered Lines:	7238
Relevant Lines:	7823

---

💛 - Coveralls