Laundry list of decisions needed for WCP clustering / pattern recognition support

[brettviren]

This issue is kind of more a "project". It is meant to capture various decisions we need to make for a number of things that WCT needs to accommodate the clustering and pattern recognition algorithms from WCP.

Check list of decisions:

Here are questions needing decisions. Let's check them off and add their answers. After each check list item are some requirements, comments and secondary questions. We will add lines giving the decisions as we make them.

• [x] What Point Cloud (PC) IData representation to use?

  • Must handle attributes of combined 3D and 3x 2D PCs.
  • Consider to use ITensor first. A PC is ultimately a 2D array eg with each column representing a point and each row representing some numeric data type.
  • A: ITensorSet will be the DFP node input/output. This allows for both floating point and integer types and is compatible with providing a nanoflann adaptor.

• [x] What PC operations library to use?

  • WCP uses https://github.com/jlblancoc/nanoflann which will used unless some show-stopper is found.
  • A: nanoflann, see comment below.

• [x] How do we represent transient PC data for algorithmic use

  • Must be mutable.
  • Must be exposed at least to iface (see why below).
  • Can nanoflann data types work?
  • A: the data from ITensor can be used via boost::multiarray or Eigen array via sharing memory which will be fast. A single copy at the input of the DFP node is needed to make a mutable copy.
  • See PointCloud::Dataset and its constituent PointCloud::Array in PR #187.

• [x] How do we convert between IData and the transient PC data structure?

  • This should be relatively time efficient.
  • Ideally we can directly share memory. Eg, ultimately one can get a C-array pointer from the boost::multiarray held by ITensor.
  • A: a nanoflann "adaptor" is required, but it looks trivial.
  • Okay, it was maybe not trivial but easy enough. See KDTree in PR #187

• [x] How do we represent collections of PC points?

  • A PC associates data to each point and an "ID" can be used to identify a collection.
  • Explicit container-based collections are also required.
  • May simply be std::vector<size_t> holding indices into PC array.
  • A: indeed, a vector of indices. See KDTree in PR #187.

• [ ] PC point sampling following "2D projected measure" algorithm.

  ~This produces "2D" points that lack Cartesian coordinate attributes and have quasi-coordinates of wire-in-plane and time tick. They also carry channel and charge attributes.~

  This is not projected but rather is formed from 2D channel-vs-tick "pixels".

• [x] PC point sampling following "3D stepped" algorithm:

  1. find N_p wires in blob plane view p.
  2. For view with min and max N_p find step size S_p = max(3,N_p/12)
  3. Accept all crossing points for min/max view which are either:
     • on the stepped grid, or
     • on the boundary of the blob, and in either case are,
     • in the third view.

• [ ] Determine if two points are inside blob-neighbors.

  • This requires going from point to blob for each and comparing at blob level.
  • This should be done by keeping a blob array in context with the PC and using a blob array index that is associated with each point.

• [ ] Test memory usage for realistic events and kitchen sink of point attributes.

• [ ] Filtered point cloud

  • A large API with functions taking reference-to-PC is expected.
  • Many functions must operate on a subset of a larger PC held by the user.
  • To avoid copying and to keep connection with the larger PC a "filtered point cloud" similar to boost::filtered_graph that provides "dynamic" selection is wanted.
  • Such functions may produce a modified PC with additional or only novel attributes, different numbers of points, etc. See next about PC union operation.

• [ ] PC union operation.

  • The cluster fitting stage will add novel points to a PC which may also have novel attributes and the two should become one
  • In general, each PC may have an intersection in the space of points and attributes and the union must fill in their differences with zeros.

• [ ] PCA on PC

  • PCA is simple enough to just implement.
  • An initial test with Eigen3 is in https://github.com/WireCell/wire-cell-toolkit/blob/apply-pointcloud/util/test/doctest_eigen_pca.cxx#L24

• [ ] Hough transform

  • Hough may be simple enough to just implement.
  • An initial test with boost::histogram is in https://github.com/WireCell/wire-cell-toolkit/blob/apply-pointcloud/util/test/doctest_pointcloud_hough.cxx

• [ ]

Related

• 189

• 190

[brettviren]

nanoflann should work as a PC library.

It's a single header and the devs recommend simply copying that into a project which is something we do already for others.

The documentation is a little sparse but with the examples and ref docs its all good enough. We must supply a DatasetAdaptor to return number of points and an individual point given its index. This should let us easily share memory with a monolithic array that can even span all of the 3D and 3x 2D point clouds and which can include additional values beyond point info. This should work fine using ITensor (or perhaps better, ITensorSet) for the IData. There is also provided an adaptor to an Eigen matrix.

I think adopting nanoflann for operations and ITensor for storage resolves the first 4 items and strongly influences the 6th. I'll start a branch to begin some integration work with nanoflann.hpp placed in util include area and providing an "adaptor" that can work with boost::multiarray and thus with ITensor.