CADWRDeltaModeling / dms_datastore

Data download and management tools for continuous data for Pandas. See documentation https://cadwrdeltamodeling.github.io/dms_datastore/
https://cadwrdeltamodeling.github.io/dms_datastore/
MIT License
1 stars 0 forks source link

Write a dms_datastore univariate format reader and writer that supports user_flag field #25

Closed dwr-psandhu closed 6 months ago

dwr-psandhu commented 6 months ago
          Adding an option to retrieve user_flags is more fraught. I'm not sure I want this to be an option for the provider flags, and the design that might do so maybe with a unified flag sounds like the convoluted logic I'd like to avoid. 

I wonder if it would be more pragmatic to write a different reader? I have a writer? Reading our format is incredibly easy ... practically nothing except maybe a dtype = { "user_flag": str}.

_Originally posted by @water-e in https://github.com/CADWRDeltaModeling/dms_datastore/issues/20#issuecomment-1847700572_

water-e commented 6 months ago

In read_ts:

def read_flagged(fpath_pattern,start=None,end=None,apply_flags=True,return_flags=False,return_meta=False):
    """ This is an independent reader routine for univariate data in dms_datastore format 
    Parameters
    ----------

    fpath_pattern : str
        File path to read, wildcards included

    start : str or Timestamp
        Start of read, saves time if years can be avoided

    end : str or Timestamp
        End of read, saves time if years can be avoided

    apply_flags: bool
        Apply user_flag as a mask, converting data to nan

    return_flags: bool
        If True, the user_flag column is returned

    return_meta: bool
        If True, the yaml header is parsed and returned, such that the return value is a tuple (meta,ts)

    Returns
    -------
        DataFrame if return_meta is false, otherwise tuple(yaml structure, DataFrame)

    """