IAMconsortium / pyam

Analysis & visualization of energy & climate scenarios
https://pyam-iamc.readthedocs.io/
Apache License 2.0
226 stars 118 forks source link

Consider replacing regex for matching * with fnmatch #542

Closed phackstock closed 5 months ago

phackstock commented 3 years ago

Fnmatch is a python standard library tool which provides unix style wildcard pattern matching. Not as powerful as regex of course but for just matching with * it should be sufficient.

Example for finding all .csv files in a folder:

import fnmatch
import os

for file in os.listdir('.'):
    if fnmatch.fnmatch(file, '*.csv'):
        print(file)

This could replace using regex per default for the pyam filter function.

danielhuppmann commented 3 years ago

Thanks for the suggestion @phackstock - this could be a nice way to get rid of the utils function _escape_regexp() within pattern_match() (for the non-regexp use case) , which seems to be doing pretty much the same thing...

https://github.com/IAMconsortium/pyam/blob/a4c6c71a20666ac70a1812f77b61d80e18cfd96e/pyam/utils.py#L492

danielhuppmann commented 2 years ago

This is used as a trial for filtering the index-query to the IIASA Database API, see #697

danielhuppmann commented 2 years ago

I reverted the previous PR in #699, because fnmatch does not natively support filtering lists, using the pyam function pattern_match() directly was an easier fix. fnmatch might still be a useful option within pattern_match().

danielhuppmann commented 5 months ago

Looked at this again as a possible improvement to #849, but this package also has issues with square brackets and would require some customization... See https://docs.python.org/3/library/fnmatch.html