Implementation of the KDE/photospline PSF (9yr/11yr NT samples) as spatial PDF for flarestack

[msackerm]

Features

• defines NorthernTracksKDE(SignalSpatialPDF) class (name: "northern_tracks_kde")
  • class can be used with 3D (psi,sigma,logE) and 4D (psi, sigma,logE,gamma) spline fits of the PSF.
  • can be used with all likelihoods, but psf does not change with gamma except in dedicated new likelihood classes
• defines two new Likelihood classes
  • StandardKDEEnabledLLH(StandardLLH) (name: "standard_kde_enabled"), and
  • StdMatrixKDEEnabledLLH(StandardOverlappingLLH) (name:"std_matrix_kde_enabled")
  • new llh classes support variations of the PSF with spectral index gamma

Tests

• E^-2 PS sensitivity and discovery potential were calculated with 9yr and 11yr NT samples and agree with published values
• n_s and gamma bias have been checked and improvements found. They are consistent with tests done here.
• See also figures below.

Other changes

• submitter.py was modified to address problems with DESY ht-condor.
  • Submission command was adjusted to first switch to local directory of the job.submit file, since condor_submit would not accept a path prepending the file name.
  • log dir creation was modified to create a new subdirectory for each cluster of jobs submitted (after crashing the DESY farm with 32000 log files in a directory).
• nt_v005_p00.py was modified to use NTSeasonNewStyle (solves memory consumption issues)
• photospline (conda-forge) was added as a dependency for flarestack

[mlincett]

This is a very welcome addition.

I see the photospline dependency is handled entirely through conda, and poetry should work on top of that.

It could be nice if we could make photospline (and hence the KDE features) optional, but I am not sure whether there is an elegant way of achieving this (in the past I used to do try import / except ImportError and set a boolean switch).

I use conda for everything myself, but I am afraid it is somewhat an "expensive" requirement.

[robertdstein]

This is very cool! Nice to see it finally being added, I always had it in mind when writing all the spatial LLH classes.

[JannisNe]

This is a very welcome addition.

I see the photospline dependency is handled entirely through conda, and poetry should work on top of that.

It could be nice if we could make photospline (and hence the KDE features) optional, but I am not sure whether there is an elegant way of achieving this (in the past I used to do try import / except ImportError and set a boolean switch).

I use conda for everything myself, but I am afraid it is somewhat an "expensive" requirement.

@mlincett do you mean make it an optional dependency through poetry? That's not possible because photospline needs some pre-compiled things so it's only available through conda. I guess the idea is to set up a conda environment with conda create -f conda_env.yml and then do the poetry install inside that environment. I will add some of thet to the README.

[mlincett]

@mlincett do you mean make it an optional dependency through poetry? That's not possible because photospline needs some pre-compiled things so it's only available through conda.

I am aware of that. What I did in other circumstances is, in case of a missing "external" dependency, to catch the "ImportError" and add a gracious failure mode for when the downstream code tries to use the classes that require the dependency.

Yet this is not very elegant and can be a pain to maintain (especially in an heavily structured code like flarestack). If no one else is concerned with the conda / photospline dependency, I am fine with taking this step as well.

[JannisNe]

Seems like there is some issue with urllib3 v2.0.7 and python 3.10. I pinned v2.0.6 for now. @mlincett good to merge?