Major refactor: Python 3.8+ and use pvWebMonitor

[prjemian]

This entire process is suffering from diversion from the original design (at 33id or 32id). Since the code is Py2.7 and Py3.8 is current release, we should revise the livedata process completely to rely on the design of the pvwebmonitor package for the handling of the public web site. The new livedata would be responsible only for maintain the local www directory (which is copied to the public site).

Originally posted by @prjemian in https://github.com/APS-USAXS/livedata/issues/47#issuecomment-698526695

---

This will be livedata version 3.

[added]

These subtasks are the responsibility of the livedata support:

• current PV values
• plot of most recent scans
  • needs data reduction of these scans
• current batch file or list
• scan log
• plots of scans and tunes (so-called "SPEC plots")
  • needs data reduction of these scans
• copy local web site to XSD server for https://usaxslive.xray.aps.anl.gov/

[prjemian]

Issue is old enough that current release of Python is now at 3.11. Will use whatever version bluesky has now. In a branch.

[prjemian]

Hit a problem with import spec2nexus using version 2021.2.4 (current):

InvalidVersion: Invalid version: '08-ops'

[prjemian]

Problem (with import spec2nexus) is local. Work with a new environment.

[prjemian]

First problem with calc.py:

  File "/home/beams11/USAXS/Documents/jemian/livedata/calc.py", line 268, in reduce_uascan
    wavelength = entry["instrument/monochromator/wavelength"].value
AttributeError: 'Dataset' object has no attribute 'value'

[prjemian]

First problem with calc.py:

  File "/home/beams11/USAXS/Documents/jemian/livedata/calc.py", line 268, in reduce_uascan
    wavelength = entry["instrument/monochromator/wavelength"].value
AttributeError: 'Dataset' object has no attribute 'value'

see https://stackoverflow.com/questions/67409919/attributeerror-dataset-object-has-no-attribute-value

[prjemian]

better: https://stackoverflow.com/questions/70181801/problems-with-h5py-dataset-object-has-no-attribute-value

[prjemian]

New code creates this structure in HDF5 file:

  uascan:NXentry
    @NX_class = "NXentry"
    @signal = "data"
    title:NX_CHAR = b'/share1/USAXS_data/2022-11/11_03_24keVTest/11_03_24keVTest.dat'
    data:NXdata
      @NX_class = "NXdata"
      @axes = "Q"
      @signal = "R"
      Q:NX_FLOAT64[268] = [-0.0001923405239672021, -0.00018620826937117252, -0.0001797222308563532, '...', 0.22467713002572465]
      R:NX_FLOAT64[268] = [0.015278255414231697, 0.004779993134122658, 0.5015744822090147, '...', 5.82520351000675e-06]
      ar:NX_FLOAT64[268] = [15.604198372322898, 15.604169484236456, 15.604138929529643, '...', 14.544858631760151]
      ar_0:NX_FLOAT64 = 15.603292286380842
      ar_r_peak:NX_FLOAT64 = 15.603416171828467
      r:NX_FLOAT64[268] = [2.6666667e-06, 8.393333343999999e-07, 8.81893333344e-05, '...', 6.0582000000000004e-09]
      r0:NX_FLOAT64[268] = [0.00017454, 0.000175593, 0.000175825, '...', 0.001039998]
      r_peak:NX_FLOAT64 = 1867.2788201720327

[prjemian]

Expect this to be identical with structure created by old code.

[prjemian]

Hit a problem with import spec2nexus using version 2021.2.4 (current):

InvalidVersion: Invalid version: '08-ops'

@jilavsky : This might be (become) a problem in the environment we are using for bluesky.

[prjemian]

In commit 1df421b, the code to reduce fly scans was restored from the archive. All these files were necessary to make this one call: https://github.com/APS-USAXS/livedata/blob/c529281045618dd5b8efab7bd1e6a07e0845a932/tests/test_calc.py#L43

That's too much!

[prjemian]

All tests pass at this time (reduces data from uascan and fly scan).

TODO (next):

• [ ] Refactor the fly scan reduce code to minimize the number of additional modules it imports.
• [ ] Add tests for tune scans.
• [ ] Add tests for SAXS scans.
• [ ] Add tests for WAXS scans.

[prjemian]

For the current PV values step above, pvWebMonitor v2021.0.0+ should be used. Perhaps also for the copy local web site to XSD server step, too?

So, build a new application based on pvWebMonitor but including the other components. Reorganize this project to be installable (with a setup.py). Organize the source by the subtasks above.

[prjemian]

The needs data reduction of these scans subtask is a common task, needs a subdirectory containing modules for each type of data reduction.

This should make it easier to maintain the code base.