We are working with Eli to develop a pipeline for uploading his XAS beam line data into aimmdb. Particularly, we want to accomplish the following with this issue:
Summary
Our endpoint (for now) will be a .dat file which contains comments starting with # (some of which are critical pieces of metadata), and otherwise columnated data which are space-delimited.
Using the existing xas schema, each channel (column other than the energy, basically) will be read into aimmdb, where the energy column is self-explanatory, and the mu column will be any of the many channels. The channel which is chosen will be indicated in the metadata as measurement_type. Eli's code below provides a good starting point. It's somewhat pseudocode and some work needs to be done.
import numpy as np
import pandas as pd
MEASUREMENT_INSTRUCTIONS = {
"transmission": {
"name": "transmission",
"numerator": "it",
"denominator": "i0",
"log": True,
"invert": True,
"col_name": "mu_trans",
},
"fluorescence": {
"name": "fluorescence",
"numerator": "iff",
"denominator": "i0",
"log": False,
"invert": False,
"col_name": "mu_fluo",
},
}
def extract_mu(path, measurement_kind):
df = pd.read_csv(path)
measurement_description = MEASUREMENT_INSTRUCTIONS[measurement_kind]
energy = df["energy"]
mu = (
df[measurement_description["numerator"]]
/ df[measurement_description["denominator"]]
)
if measurement_description["log"]:
mu = np.log10(mu)
if measurement_description["invert"]:
mu = -mu
# Also read the metadata from the file, include all commented lines, but
# we need to pick out the particularly important databroker unique id
metadata = ...
# process data frame...
return df, metadata
Specific steps
Create a module aimmdb.ingest
Create a particular file aimmdb/ingest/eli.py (we'll rename this to the name of Eli's beam line later)
Create a single function (ingest) which takes a single path as an argument and returns a pd.DataFrame (the data) and dict (metadata).
Don't forget that the pd.DataFrame columns must be energy and mu. The actual column we use for mu will change depending on the channel we're looking at.
In Eli's examples, we only have "transmission" and "fluorescence". Eli has provided instructions (code above) on how to process these particular types of data and how they should be represented in aimmdb
We MUST document every type of processing we do (see above code) before it gets uploaded into aimmdb. I recommend a README file in aimmdb.ingest for now, until we move to a more standard documentation solution.
Building the ingestion pipeline
We are working with Eli to develop a pipeline for uploading his XAS beam line data into
aimmdb. Particularly, we want to accomplish the following with this issue:Summary
.datfile which contains comments starting with#(some of which are critical pieces of metadata), and otherwise columnated data which are space-delimited.xasschema, each channel (column other than the energy, basically) will be read intoaimmdb, where the energy column is self-explanatory, and themucolumn will be any of the many channels. The channel which is chosen will be indicated in the metadata asmeasurement_type. Eli's code below provides a good starting point. It's somewhat pseudocode and some work needs to be done.Specific steps
aimmdb.ingestaimmdb/ingest/eli.py(we'll rename this to the name of Eli's beam line later)ingest) which takes a single path as an argument and returns apd.DataFrame(the data) anddict(metadata).pd.DataFramecolumns must beenergyandmu. The actual column we use formuwill change depending on the channel we're looking at.aimmdbaimmdb. I recommend aREADMEfile inaimmdb.ingestfor now, until we move to a more standard documentation solution.