cfenell
commented
7 years ago Update on Level 3 data in Madrigal 3.0 HDF format.
The files contain a group Metadata, with 4 relevant tables (see following screenshot).
In principle all of these metadata can be extracted. Details of the 4 tables:
_record_layout: not sure whether this always exists. Lists parameters that exist in the two other parameter tables(?)
Experiment Notes: A text string. Usually empty but if existing it should be extracted and inserted into the Notes field.
Experiment Parameters : The following fields should be in the B2SHARE metadata schema. The values should be extracted and inserted into the metadata fields.
instrument code(s): list of numbers of instruments kindat code(s): list of data codes start time : Time UT end time : Time UT status description : Text (could/should check == "Final" ??) instrument latitude: degrees N instrument longitude : degrees E instrument altitude: km
If possible insert the following into the contact info in the standard metadata fields: instrument PI instrument PI email
Data Parameters: Description of the data in Data/Table Layout
These could be omitted for now or converted into a list of keywords, Occuring in EISCAT data are typically UT1_UNIX, UT2_UNIX (Start and stop of integration in UNIX time) or YEAR, MONTH, DAY, HOUR, MIN, SEC KINDAT, AZM, ELM, RANGE, GDALT, GDLAT, GLON GFIT, CHISQ (Fit quality parameters) POPL, NEL, PO+, TI, TR, COL, VO or VOBI (Fitted or derived parameters) errors of the above: D\<parameter>
For completeness I attach the definition of all available Madrigal parameters, maybe it is useful.
emanueldima
janiheikkinen
Modify metadata fields (add and remove), and add extracted metadata from uploaded files. There should be default values for all metadata fields.
EISCAT records in B2SHARE
-Result files (HDF5 files) contain
Data/Table Layout EISCAT Level 3 Data (analysis results as function of time and positions).
Metadata/Data Parameters Description of the parameters in the data table above
Metadata/Experiment Notes Text (usually empty)
Metadata/Experiment Parameters Metadata to be extracted and pre-inserted into B2SHARE metadata fields when uploading a new file to deposit.
Graphical visualisations (png or eps files) can be added to the EISCAT records. They are metadata to the above files.
-Format of Metadata/Experiment Parameters Asterisk (*) marks compulsory fields. They should either have default values or be extracted from uploaded HDF5 file.
--Metadata fields, Generic Title: default to name of file except for extensions Licence: default to EISCAT rules of the road (add to list) Keywords: default "eiscat" Contact email: default ingemar.haggstrom@eiscat.se Discipline: default to 3.3 Earth sciences. Would need new subcategories: Atmospheric physics, Ionospheric physics, Solar-terrestrial physics. Resource type: default to Time series
--Existing metadata fields, EISCAT Extract from HDF5:
Instrument name (text) Metadata/Experiment Parameters/instrument Kind of data (NB rename) Metadata/Experiment Parameters/kindat Start time (Date UTC) Metadata/Experiment Parameters/start time End time (Date UTC) Metadata/Experiment Parameters/end time Status description (preliminary or final) Metadata/Experiment Parameters/status description Instrument latitude (Degrees north) Metadata/Experiment Parameters/instrument latitude Instrument longitude (Degrees east) Metadata/Experiment Parameters/instrument longitude Instrument altitude (Kilometres) Metadata/Experiment Parameters/instrument altitude
--Remove from EISCAT metadata fields Kind of data file (text description of kindat)
-- Add metadata fields, EISCAT, if possible to extract min and max of data, to enable some position search.
Max height kindat 1 Max of numerical values (excluding NaNs) Data/Table Layout/gdalt where Data/Table Layout/kindat == <kindat Show quoted text Min height kindat 1 Min of numerical values (excluding NaNs) Data/Table Layout/gdalt where Data/Table Layout/kindat == <kindat Show quoted text Max height kindat 2 Max of numerical values (excluding NaNs) Data/Table Layout/gdalt where Data/Table Layout/kindat != <kindat Show quoted text Min height kindat 2 Min of numerical values (excluding NaNs) Data/Table Layout/gdalt where Data/Table Layout/kindat != <kindat Show quoted text Max azimuth Max of numerical values (excluding NaNs) Data/Table Layout/azm Min azimuth Min of numerical values (excluding NaNs) Data/Table Layout/azm Max elevation Max of numerical values (excluding NaNs) Data/Table Layout/elm Min elevation Min of numerical values (excluding NaNs) Data/Table Layout/elm