Excise "exclude" column from meta and add a own attribute

[danielhuppmann]

Please confirm that this PR has done the following:

• [x] Tests Added
• [x] Documentation Added
• ~Name of contributors Added to AUTHORS.rst~
• [x] Description in RELEASE_NOTES.md Added

Description of PR

This PR excises the "exclude" column from the "meta" attribute to reduce confusion among users why an exclude column is added from a timeseries-only data file. Also, this currently requires some tinkering in the IIASA Scenario Explorer upload workflow.

The PR:

• creates a new attribute exclude and refactors all validation methods such that the argument exclude_on_fail changes this attribute
• extends the filter such that filter(exclude=<value>) works as expected
• adds "exclude" to the list of illegal column names for (timeseries) data and the meta attribute
• when importing an xlsx file created with pyam < 2.0 (which has an "exclude" column in "meta"), that column is moved to the new exclude attribute (with a log message)
• streamlines a number of tests and updates the documentation

fyi @gidden @byersiiasa @phackstock

closes #755

[codecov[bot]]

Codecov Report

Merging #759 (94dd226) into main (19d08a5) will decrease coverage by 0.1%. The diff coverage is 94.3%.

:exclamation: Current head 94dd226 differs from pull request most recent head 149efc3. Consider uploading reports for the commit 149efc3 to get more accurate results

@@           Coverage Diff           @@
##            main    #759     +/-   ##
=======================================
- Coverage   94.5%   94.4%   -0.1%     
=======================================
  Files         59      59             
  Lines       6016    6080     +64     
=======================================
+ Hits        5686    5744     +58     
- Misses       330     336      +6     

Impacted Files	Coverage Δ
pyam/iiasa.py	87.1% <50.0%> (-0.3%)	:arrow_down:
pyam/logging.py	64.1% <66.6%> (-0.8%)	:arrow_down:
pyam/core.py	95.2% <91.6%> (-0.2%)	:arrow_down:
pyam/utils.py	92.7% <92.3%> (-0.1%)	:arrow_down:
pyam/_debiasing.py	100.0% <100.0%> (ø)
pyam/testing.py	77.7% <100.0%> (+2.7%)	:arrow_up:
tests/test_core.py	100.0% <100.0%> (ø)
tests/test_feature_aggregate.py	98.9% <100.0%> (ø)
tests/test_feature_append_concat.py	100.0% <100.0%> (ø)
tests/test_feature_rename.py	100.0% <100.0%> (ø)
... and 3 more

[danielhuppmann]

Responding to this comment by @byersiiasa

But would there be a method to edit the exclude attribute manually? e.g., perhaps similar to how one might use set_meta('exclude'...)

You can currently do the following in the new API (in addition to the validation methods):

df.exclude = False

to change the entire exclude pd.Series

df.exclude[idx] = True

where idx can be an integer, list of integer, or a pd.MultiIndex.

So you could do the following

index = df.filter(...).index
df.exclude[index] = True

Is that good enough for your use case? Or are there other uses that you encountered?

[byersiiasa]

Yes - that sounds good!

[danielhuppmann]

Merging after discussions in the MESSAGE group meeting...