For the HAPI server when output=csv is used, we would like to use the precision part when printing floating point values when the variable has a FORMAT or FORM_PTR attribute.
Two places that seem to handle the FORMAT value are
The following code produces the table that follows it. The first column of the table is the given FORMAT and the third column is the parsed format. If the parsed value is '', the FORMAT value did not have precision information or was invalid. The plan is to include a x_FORMAT attribute in the metadata for a HAPI parameter if a valid one exists. If none exists for a parameter of type=double,.7g will be used if the parameter was stored in the CDF as a 32-bit float and .16g will be used for 32-bit floats (as done in CDAWlib). Note that Nand uses .3e for both, which does not seem to be a good choice.
Values of ? and ⎵ in the first column were not actually in CDF Master metadata. For the SQL table, ? was inserted for all ISTP attributes when a variable had no VariableAttributes (e.g., https://hapi-server.org/meta/cdaweb/variable/#FORMAT=?) and ⎵ was inserted when FORMAT=' ' (a single space).
Please review the output of the code at the bottom of this page.
If you want to determine where a given format specifier occurs, use, for example
import re
def f2c_specifier(f_template, cdf_type):
"""Extract the precision part of a Fortran format string found as a FORMAT
value.
"""
# See also
# https://github.com/rweigel/CDAWlib/blob/952a28b08658413081e75714bd3b9bd3ba9167b9/cdfjson__define.pro#L132
# https://git.smce.nasa.gov/spdf/hapi-nand/-/blob/main/src/java/org/hapistream/hapi/server/cdaweb/CdawebUtil.java?ref_type=heads#L23
if isinstance(f_template, list):
fmts = []
for f in f_template:
fmt = f2c_specifier(f, cdf_type)
if fmt == "":
# If any not valid, don't use any
return ""
else:
fmts.append(fmt)
return fmts
defaults = {
'CDF_FLOAT': '.7g',
'CDF_DOUBLE': '.17g',
'CDF_REAL4': '.7g',
'CDF_REAL8': '.17g'
}
f_template = f_template.lower().strip()
if f_template == "":
return ""
# No precision.
if "." not in f_template:
return ""
# Drop any string or integer related format string.
fmt = re.sub(r".*[a|s|c|z|b|i].*", "", f_template)
if fmt == "":
return fmt
# e.g., E11.4 => .4e, F8.1 => .1f
fmt = re.sub(r".*([f|e|g])([0-9].*)\.([0-9].*)", r".\3\1", f_template)
# Test the format string
try:
templ = "{:" + fmt + "}"
templ.format(1.0)
except:
#print("Failed: " + fmt)
return ""
return fmt
if __name__ == "__main__":
import cdawmeta
kwargs = {
"id": None,
"data_dir": "../data",
"embed_data": True,
"update": False,
"max_workers": 1,
"diffs": False,
"restructure_master": True,
"no_spase": True,
"no_orig_data": True
}
meta = cdawmeta.metadata(**kwargs)
formats = []
for id in meta.keys():
if "master" not in meta[id]:
continue
master = meta[id]["master"]['data']
if master is None or "CDFVariables" not in master:
continue
variables = master['CDFVariables']
for variable_name, variable in variables.items():
if "VarAttributes" not in variable:
continue
if "VarDescription" not in variable:
continue
if 'DataType' in variable['VarDescription']:
cdf_type = variable['VarDescription']['DataType']
else:
continue
if 'FORMAT' in variable['VarAttributes']:
format = variable['VarAttributes']['FORMAT']
formats.append(f"{format}; {cdf_type}; {f2c_specifier(format, cdf_type)}")
if 'FORM_PTR' in variable['VarAttributes']:
FORM_PTR = variable['VarAttributes']['FORM_PTR']
if FORM_PTR in variables:
variable = variables[FORM_PTR]
if 'VarData' in variable:
VarData = variable['VarData']
if VarData[0][0:2] == '10':
# https://github.com/rweigel/cdawmeta/issues/11
print(id, variable_name, VarData)
format = ", ".join(variable['VarData'])
formats.append(f"{format}; {cdf_type}; {','.join(f2c_specifier(VarData, cdf_type))}")
uniques = list(set(formats))
for unique in uniques:
unique = unique.split("; ")
if len(unique[0].split(",")) > 1:
if len(unique[2].split(",")) == 0:
print(f"[{unique[0]}] {unique[1]}' => ''")
else:
print(f"[{unique[0]}] {unique[1]}' => [{unique[2]}]")
else:
print(f"'{unique[0]}' {unique[1]} => '{unique[2]}'")
For the HAPI server when
output=csv
is used, we would like to use the precision part when printing floating point values when the variable has aFORMAT
orFORM_PTR
attribute.Two places that seem to handle the
FORMAT
value areThe following code produces the table that follows it. The first column of the table is the given
FORMAT
and the third column is the parsed format. If the parsed value is''
, theFORMAT
value did not have precision information or was invalid. The plan is to include ax_FORMAT
attribute in the metadata for a HAPI parameter if a valid one exists. If none exists for a parameter oftype=double
,.7g
will be used if the parameter was stored in the CDF as a 32-bit float and.16g
will be used for 32-bit floats (as done in CDAWlib). Note that Nand uses.3e
for both, which does not seem to be a good choice.Values of
?
and⎵
in the first column were not actually in CDF Master metadata. For the SQL table,?
was inserted for all ISTP attributes when a variable had noVariableAttributes
(e.g., https://hapi-server.org/meta/cdaweb/variable/#FORMAT=?) and⎵
was inserted whenFORMAT=' '
(a single space).Please review the output of the code at the bottom of this page.
If you want to determine where a given format specifier occurs, use, for example