search

IAMconsortium / pyam

Analysis & visualization of energy & climate scenarios
https://pyam-iamc.readthedocs.io/
Apache License 2.0
221 stars 115 forks source link

Refactor the "exclude" feature #755

Closed danielhuppmann closed 11 months ago

danielhuppmann commented 1 year ago

All validation methods have a flag "exclude_on_fail" feature to easily track which scenarios fail one or several validation steps. This is implemented via an "exclude" column in the meta indicators.

However, this approach causes some hiccups in the integration with the ixmp database infrastructure - the "exclude" meta indicator is always added by pyam and then has to be removed before importing to the database (or can cause confusion by users).

My proposal: drop the "exclude" column from the "meta" dataframe and move it to an own attribute. in addition, extend the filter() method such that df.filter(exclude=True) continues to work as is.

Any thoughts @phackstock @gidden @byersiiasa?

gidden commented 1 year ago

I always end up making my own meta indicator anyway - so I agree with your proposal to remove

On Wed, Jun 28, 2023 at 9:59 AM Daniel Huppmann @.***> wrote:

All validation methods have a flag "exclude_on_fail" feature to easily track which scenarios fail one or several validation steps. This is implemented via an "exclude" column in the meta indicators.

However, this approach causes some hiccups in the integration with the ixmp database infrastructure - the "exclude" meta indicator is always added by pyam and then has to be removed before importing to the database (or can cause confusion by users).

My proposal: drop the "exclude" column from the "meta" dataframe and move it to an own attribute. in addition, extend the filter() method such that df.filter(exclude=True) continues to work as is.

Any thoughts @phackstock https://github.com/phackstock @gidden https://github.com/gidden @byersiiasa https://github.com/byersiiasa?

— Reply to this email directly, view it on GitHub https://github.com/IAMconsortium/pyam/issues/755, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKUAEN5JNHUDMRZSWDW2I3XNPP7LANCNFSM6AAAAAAZWUVVRI . You are receiving this because you were mentioned.Message ID: @.***>

byersiiasa commented 1 year ago

Sure - sounds fine to me. But would there be a method to edit the exclude attribute manually? e.g., perhaps similar to how one might use set_meta('exclude'...)