Plug the EPIC scraper to a parser

[aalbino2]

This issue collects all the points to be implemented to parse the process data.

The new features are pushed directly in the main branch

[aalbino2]

Questions:

• [x] Messages.txt: which ID should be logged when a Growth Chamber is loaded.
  • Currently there is the growth run ID. 16/06/2023 16:13:06.903,Location Tab,M8abcd moved from MC_manip to GC,-16777216. 'M8abcd' will be instead: growth_id@holder_id e.g., growth_id="M84123", holder_id="H4_10".
• [x] "MBE sources" sheet in xlsx file: can we delete the EPIC loop column as we now have columns for the logged parameters directly?
  • EpicLoopHL column can be deleted but the bare EPIC loop is still useful for the flux calibration "Fitting.txt" file, now added in the raw files folder.
• [x] insert units around as in source_object.vapor_source.temperature.value
• [x] "MBE gas mixing" how did we decide to organize this sheet? one MFC per line? how do we link more lines to the same plasma source?
  • each line is one MFC measure. We link more lines to the same growth process through the datetime.
• [x] check that missing columns do not stop processing. Check related issue: https://github.com/PDI-Berlin/pdi-nomad-plugin/issues/23
• [x] upload xlsx in github
• [x] Units of the raw files ??
  • new columns have been added to the xlsx file
• [x] which excel sheets include in the settings xlsx that will be uploaded together to the raw txt files?
  • all the excel sheets will be kept in each growth as it is not a heavy file. Checks in NOMAD will be made to see if an instrument with same ID already exist and if the archive is the very same or something changed.
• [x] let's split the xlsx file. which sheets are process-specific, which one are more general?
  • we don't split it anymore.
• [x] Substrate: SUB.MV SUB.WOP SUB.PID.I SUB.PID.V are now included in growth->sample_parameters. Is this the right place?
  • it's okay. But do not parse for now SUB.PID.I SUB.PID.V
• [x] Sub.PID.I and Sub.PID.V were not found in example folders
• [x] Pyro.PID.MV is inside in-situ reflectance Pyrometry
  • it's okay. Plot the pyrometer_temperature into the growth_step.substrate_tempearture plot
• [x] where should we show in-situ reflectance pyrometry and laser emission (LR) section?
  • make a dedicated issue to be tackled when the process is divided into steps. Here is the issue: https://github.com/PDI-Berlin/pdi-nomad-plugin/issues/22
• [x] XML file for in-situ: will it be included in the same folder (PDIRS folder)? can we include it in the xlsx setting file?
  • XML is not used, we have an excel sheet called LR config, create Laser Reflectance class.
• [x] Plasma source: fill in dissipated power as the difference between forward and reflected power
  • leave out dissipated
• [x] In situ Emission or Pyrometry (change the current in situ reflectance name)
• [x] Pyrometry settings (new excel sheet)
• [x] CLC (cold lip cells) which parameters do you record for this source?
  • same as SFC. generate a new class and add source_type.
• [x] substrate holder is now in the "samples" section in the growth process archive
  • fix empty sample holder reference in filled sample holder.
  • insert_reduction should not always be in the position, only optionally
• [x] fill the substarte with "grown" status attribute
• [x] check the source and port linking network
• [x] change source_name in xlsx for plasma-type source (material cell is a dummy one)
• [x] Integrate the use of HDF5Dataset
• [x] test if units are figured out in HDF5 quantities
• [x] inside the growth step, generate in situ characterization polimorph. Pyrometry, Laser Reflectometry, line-of-sight Quadrupole Mass Spectometry, RHEED (reflection high-Energy electron diffraction)

[aalbino2]

Questions:

• [ ] fill impinging flux in Source. Check the formula:
  • 'EPIC Fitting Config File Fit=AEF ' activation energy fit: BEP=A*exp(T0/T_cell[K]) BEP typically 1e-7mbar; FLUX typically 0.01--1 nm^-2s^-1 Order=1 ' ignored for aef BEPtoFlux=4926000 ' mbar into nm^-2s^-1: FLUX=BEPtoFlux x BEP Coeff=4.614E00,-2.313E04 ' A, T0 (from the Fitting.txt file) Edit: it was filled but the value looks strange (~E-11)
• [ ] "MBE sources" : do we need an ID for each source?
  • No ID for sources. The gas mixing of PLASMA source will be taken given that the growth time interval falls in the timestamp of the gas mixing lines. Problem: there are many gas mixings that match the condition mixing_date < growth_start, hence an addictional way to parse those is needed!
• [ ] "MBE sources" : do PLASMA sources need a Fitting.txt file ? FIll accordingly the cell
• [ ] "MBE gas mixing". grower: where to put it? and port number: pleonastic as it is in mbe source already?
• [ ] Shutters.txt should follow the EPIC loop naming to modulate the impinging flux
  • only impinging flux should be modulated?
  • which is the first line I will find in Shutters.txt? Does it align exactly with the growth start (e.g. "to GC")?
• [ ] I see the Ga_Sh is always shutted off. Although we have some Ga.MV file. Can we find a meaningful dataset, now that we start having a complex series of implications in the parser?
• [ ] problem: how to identify the correct LR from the "LR settings" (same issue as gas mixing)
• [ ] Parse LR.MV.txt file into LR (wait for https://github.com/PDI-Berlin/EPIC_scraper/issues/6)
• [ ] check source_geometry and source length