Interoperability of light_curve_generation and light_curve_analysis

tylerbarna commented 1 year ago

This is a mix of a bug report and a feature request

Feature Summary currently, using an injection from nmma_create_injection to create a lightcurve with light_curve_generation creates a lightcurve with the following header:

# t[days] ztfg ztfr ztfi
0.10000 19.357 19.597 19.839

If you attempt to run light_curve_analysis on the file, it errors out because the lightcurve file is not in the format nmma expects

To Reproduce (note: these were run in a directory in the same root as the nmma repo, but not actually in the nmma repo folder)

nmma_create_injection --prior-file ../nmma/priors/Me2017.prior --eos-file ../nmma/example_files/eos/ALF2.dat --binary-type BNS --n-injection 100 --original-parameters --extension json

light_curve_generation --injection injection.json --model Me2017 --outdir outdir --label lc --tmin 0.1 --tmax 20 --dt 0.5  --injection ./injection.json --generation-seed 42 --filters ztfg,ztfr,ztfi --plot

light_curve_analysis --model Me2017 --outdir outdir --label injected_lc_analysis --prior ../nmma/priors/Me2017.prior --tmin 0.1 --tmax 20 --dt 0.5 --error-budget 1 --nlive 512 --Ebv-max 0 --data outdir/lc_0.dat  --generation-seed 42 --filters ztfg,ztfr,ztfi --plot --remove-nondetections

Returns the following error

Creating light curve model for inference
/home/tbarna/anaconda3/envs/nmma_dev/lib/python3.9/site-packages/astropy/time/formats.py:1580: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  val1_uint32 = val1.view((np.uint32, val1.dtype.itemsize // 4))
Traceback (most recent call last):
  File "/home/tbarna/anaconda3/envs/nmma_dev/lib/python3.9/site-packages/astropy/time/formats.py", line 1527, in set_jds
    jd1, jd2 = self.get_jds_fast(val1, val2)
  File "/home/tbarna/anaconda3/envs/nmma_dev/lib/python3.9/site-packages/astropy/time/formats.py", line 1592, in get_jds_fast
    time_struct = self._fast_parser(chars)
ValueError: fast_parser: Input operand 0 does not have enough dimensions (has 0, gufunc core with signature (n)->() requires 1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tbarna/anaconda3/envs/nmma_dev/lib/python3.9/site-packages/astropy/time/core.py", line 597, in _get_time_fmt
    return cls(val, val2, scale, precision, in_subfmt, out_subfmt)
  File "/home/tbarna/anaconda3/envs/nmma_dev/lib/python3.9/site-packages/astropy/time/formats.py", line 160, in __init__
    self.set_jds(val1, val2)
  File "/home/tbarna/anaconda3/envs/nmma_dev/lib/python3.9/site-packages/astropy/time/formats.py", line 1533, in set_jds
    jd1, jd2 = self.get_jds_python(val1, val2)
  File "/home/tbarna/anaconda3/envs/nmma_dev/lib/python3.9/site-packages/astropy/time/formats.py", line 1562, in get_jds_python
    ) = self.parse_string(val, subfmts)
  File "/home/tbarna/anaconda3/envs/nmma_dev/lib/python3.9/site-packages/astropy/time/formats.py", line 1721, in parse_string
    return super().parse_string(timestr, subfmts)
  File "/home/tbarna/anaconda3/envs/nmma_dev/lib/python3.9/site-packages/astropy/time/formats.py", line 1511, in parse_string
    raise ValueError(f"Time {timestr} does not match {self.name} format")
ValueError: Time # does not match isot format

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/tbarna/anaconda3/envs/nmma_dev/bin/light_curve_analysis", line 8, in <module>
    sys.exit(main())
  File "/home/tbarna/anaconda3/envs/nmma_dev/lib/python3.9/site-packages/nmma/em/analysis.py", line 526, in main
    data = loadEvent(args.data)
  File "/home/tbarna/anaconda3/envs/nmma_dev/lib/python3.9/site-packages/nmma/em/io.py", line 22, in loadEvent
    mjd = Time(lineSplit[0], format="isot").mjd
  File "/home/tbarna/anaconda3/envs/nmma_dev/lib/python3.9/site-packages/astropy/time/core.py", line 1850, in __init__
    self._init_from_vals(
  File "/home/tbarna/anaconda3/envs/nmma_dev/lib/python3.9/site-packages/astropy/time/core.py", line 535, in _init_from_vals
    self._time = self._get_time_fmt(
  File "/home/tbarna/anaconda3/envs/nmma_dev/lib/python3.9/site-packages/astropy/time/core.py", line 605, in _get_time_fmt
    raise ValueError(
ValueError: Input values did not match the format class isot:
ValueError: Time # does not match isot format

Usage / behavior It should be possible to create an injection, generate a lightcurve from the injection, and analyse the generated lightcurve with minimal effort within nmma

This would entail making light_curve_analysis handle file structures different from the expected isot format as it currently exists

Alternative Solutions we can also modify light_curve_generation to match the format expected by light_curve_analysis, but I think this is less preferable as a long-term solution

We can also implement a function that converts from the original format to our standard format; people could submit PRs to add new formats to the converter function

Implementation details Alter light_curve_analysis and/or light_curve_generation

Additional context We also should better document the output formats of the various files generated by nmma

tylerbarna commented 1 year ago

I also noticed that previous versions would output lightcurves in a format like

{
[filter1]:[[t0,m0,m0_unc],
           [t1,m1,m1_unc],...], 
[filter2]:...
}

rather than the current format of

# t[days] [filter1] [filter2] ...
t0        m0_filt1  m0_filt2 ...
...

tylerbarna commented 1 year ago

I think the disconnect when loading in a lightcurve file is in load_event() as it's called in analysis.py

tylerbarna commented 1 year ago

Resolved by https://github.com/nuclear-multimessenger-astronomy/nmma/commit/f61fcd25a698f8d302d0c32c98dffd71152bfaa2

nuclear-multimessenger-astronomy / nmma

Interoperability of light_curve_generation and light_curve_analysis #187