Fully support GIDCollections in SLI and Python

[heplesser]

We decided in the Open NEST VC 8 Aug to removed subnets which represent structure in the simulation kernel and move all structure representation to the script layer in form of GIDCollections (see als #417). We further decided to

• create release 2.12 as a release that will be backward compatible (so old scripts still run) but issue deprecation warnings on functions and argument types that will be removed with NEST 3.0
• have subnets fully removed from the NEST 3.0 release.

To ensure a smooth transition, it is essential that release 2.12 provides the full user interface of NEST 3.0, so that users can convert their scripts using the 2.12 release, before we release NEST 3.0. We should thus perform as much work as possible on master, before creating the 2.12 release branch (where we will add deprecation warnings) and the no-subnet-branch preparing 3.0.

In particular, we need to ensure that

• all connection functions support GIDCollections
• all get/set functions support GIDCollections
• GetConnections supports GIDCollections
• creation functions return GIDCollections
• topology supports GIDCollections (create separate issue)
• related issues are resolved (#266, #355, #369)
• all tests and examples use GIDCollections instead of subnets (create separate issue) before branching.

For Create and other related functions, we need to introduce NewCreate and similar which return GIDCollections, so that they can coexist with the old Create in 2.12. When converting scripts to 3.0 later, one will then just need to replace NewCreate with Create.

[jougs]

Thanks for writing this up. However, I disagree about the creation of NewCreate, NewGetStatus, etc.

In my opinion, no one will use them anyway until things break. Adding them just creates more work for us now and again when the transition occurs. Let's rather issue a good deprecation warning in the functions for 2.12.

Moreover, this is only relevant for SLI users, which are the smaller part of the community nowadays and we can provide support for the transition on the mailing list if required.

In PyNEST, most code will probably work seamlessly even with the change, e.g. consider

inhibitory = nest.Create("iaf_neuron", 10000)
nest.SetStatus(inhibitory, "V_m", -45.0)

[heplesser]

@jougs I think your argument is sound.

[heplesser]

Design of GIDCollections

GIDCollections were introduced as a compact representation of network nodes. They currently exist in two forms, either representing a contiguous range of GIDs by storing the first and last neuron, or by storing an explicit array of arbitrary GIDs. GIDCollections are currently only created by explicit conversion operations on the SLI level, and are accepted by the Connect function as argument. GIDCollections support iteration, but not indexing or slicing.

"Current" here refers to the public master branch at 1fd378f8d95d64ed0af32fadd373a9500a68e719 .

Principles

GIDCollections are to represent the nodes of a network on the script level (i.e., outside the kernel). They shall be as compact as possible and support efficient operations, especially connection generation and status changes or queries.

The script shall control the lifetime of the GIDCollection, i.e., a GIDCollection should be deleted when the corresponding object is deleted at the script level, either explicitly or by going out of scope.

Consideration will be given first and foremost to the Python level, since it is the prime user interface.

Terminology

• A GIDCollection is contiguous if it represents a contiguous range of GIDs
• A GIDCollection is homogeneous if all GIDs in the collection refer to nodes of the same type (same model ID)
• A GIDCollection is primitive if it is contiguous and homogeneous
• A GIDCollection is composite if it is not primitive

Assumptions

The following assumptions apply to non-explicit GIDCollections:

1. GIDCollection objects are created rarely but looked up frequently
2. GIDCollection objects are immutable
3. GIDCollection objects will first and foremost be created by Create calls; each such call will return a primitive collection
4. A GIDCollection holds information about the model ID of each GID in the collection
5. A GIDCollection may hold additional metadata (see section on Topology below for discussion on Metadata)
6. A GID occurs in a GIDCollection at most once
7. GIDs are ordered in ascending order in a GIDCollection
8. If a composite GIDCollection contains at least one primitive GIDCollection with metadata, then all primitive GIDCollections in the GIDCollection must hold the same metadata. Same-ness means in this case that the metadata pointers are identical.
9. Primitive GIDCollections with metadata are never coalesced into a single primitive GIDCollection, even if their ranges are adjacent and they have the same model type.

It appears plausible to assume that GIDCollections will represent over 100, often over 1000 nodes, in extreme cases (4g benchmark), billions of GIDs.

When creating data-driven connections using one-to-one connect, one may require source and target lists that contain the same GID multiple times and in which order needs to be preserved, since GIDs are matched by position between source and target lists, and may also be matched to connection parameter lists. To keep the interface of Connect clean, one may represent also such collections as GIDCollections. After a lengthy discussion, @terhorstd, @heplesser, @stinebuu and @hakonsbm concluded that such collections would behave so differently from ordinary GIDCollections that it would not be sensible to include them in the same class. They are therefore no further discussed here.

Operations on GIDCollections

GIDCollections shall support the following operations:

1. Test of membership
2. Test whether one GIDCollection is equal to another (contains the same GIDs)
3. Concatenation of two non-overlapping GIDCollections
4. Iteration
5. Indexing
6. Slicing
7. Conversion to and from lists

Examples

SLI

/Enrns /iaf_psc_alpha 800 Create def      % (1, 800)
/Inrns /iaf_psc_alpha 200 Create def        % (801, 1000)

/nrns Enrns Inrns join def    % addition    % (1, 1000)

Enrns nrns << /rule /fixed_indegree /degree 100 >> Connect

10 Enrns MemberQ =       % membership

/gc [ 10 20 30 ] cvgidcollection def    % conversion from list
/Ilist Inrns cva def                % conversion to array

Enrns Inrns eq =     % equality

Enrns { ShowStatus } forall     % iteration

Inrns 10 get =      % indexing
Enrns 20 Take =    % slicing

PyNEST

Enrns = nest.Create('iaf_psc_alpha', 800)
Inrns = nest.Create('iaf_psc_alpha', 200)
nrns = Enrns + Inrns

nest.Connect(Enrns, nrns, {'rule': 'fixed_indegree', 'degree': 100})

10 in Enrns   % membership

gc = nest.GIDCollection([10, 20 ,30])
Ilist = list(gc)

for gid in Enrns:
    print( nest.GetStatus( gid ) )

print( Inrns[10] )
print( Enrns[:20] )

Topology and Metadata

This first use of metadata for GIDCollections is the subnet-free reimplementation of Topology (#481). Layers containing only a single neuron model can be represented by a primitive GIDCollection, while layers with composite elements will require a composite GC with one primitive GC per component. All these primitive GCs are part of the same layer and thus share the same geometry. They therefore have all the same metadata. This motivates the requirement that all primitive GCs in a composite GC must have the same metadata.

GCs with metadata are never coalesced, even if they have adjacent ranges and identical model ids, since users may have specified composite elements with identical models.

For minor changes to the user interface, see #481.

Remarks

• Tests can be implemented efficiently by exploiting sortedness
• Concatenation permitted if uniqueness i preserved and will preserve sortedness
• Iteration, indexing and slicing will be based on sortedness and are thus deterministic and independent of how the GIDCollection was constructed
• The operations are required to permit existing Python scripts employing operations on lists of GIDs to continue to work (neurons = Eneurons + Ineurons, Connect(E_neurons[:50], spike_det))

Implementation

Python interface

• Currently GIDCollections cannot be converted to or from the Python level. This must become possible, including all operations listed above.
• Even if most GIDCollections are lightweight, unnecessary conversion between Python, SLI and C++ levels should be avoided. Therefore, the representation at the Python level should just be a pointer to the underlying C++ representation.

Composite GIDCollections

Non-primitive GIDCollections are represented as lists of pointers to the GIDCollections they are constructed from. Any immediately adjacent GIDCollections of the same node type are combined into a primitive GIDCollection.

Model ID information

• Each primitive GIDCollection stores the model ID of the GIDs it represents
• The main purpose of storing model ID information in GIDCollections is to reduce GID-based lookups for model type, existence of thread-siblings, etc in connection, setting and getting routines

Metadata implementation

To provide necessary flexibility, a GIDCollectionMetadata abstract base class will be provided. This class has only a minimal interface, all details are added by derived classes special to the use of the metadata, e.g., in topology:

class GIDCollectionMetadata 
{
  public:
     GIDCollectionMetadata() {}
     virtual ~GIDCollectionMetadata() = 0;
};

Since all elements of a composite GC must contain the same GC pointer, the composite GC can also return a metadata pointer.

For an example of metadata, see #481.

[terhorstd]

first version of GIDCollection UML:

[heplesser]

@terhorstd I struggle to get Dia to run on OSX, so could you add the following additions to the UML diagram:

• GIDCollectionMetadata class, see above
• GIDCollectionMetadata const * get_metadata() const method
  • zero pointer indicates no metadata
  • GIDCollectionComposite returns parts[0].get_metadata()
  • operator+() raises an exception unless metadata_ == other.metadata_ (either neither have metadata, i.e., zero pointers, or both point to the same metadata object)

I have created branch fix-455-gidcollections for work on this issue.

[heplesser]

Summary of Design Discussion 8 Nov 2016

@terhorstd, @stinebuu, @hakonsbm and @heplesser discussed design via VC on 8 Nov. Here a summary of conclusions and open questions. This post is based on notes by @terhorstd .

How to include "explicit collections" (general unsorted non-unique arrays) in GIDCollection interface?

• not the same type of objects and used for entirely different purposes
• difficult to construct common interface
• still possible to subclass separate types later if required at all.

Decision

Do not include "explicit collections" (general unsorted non-unique arrays) in GIDCollection interface

Can a GIDCollectionComposite contain GIDCollectionComposites?

• inside composites could not be sorted by gid
• substructure would not be accessible without special methods/iterators
• would prevent joining of primitives

Decision

GIDCollectionComposite is always a flat container for GIDCollectionPrimitives.

How to join metadata while joining two primitives?

• would require same operations for metadata that are defined for GIDCollections

Decision

Joining of primitives allowed only for same metadata (pointer equality)

A GIDcollection has only ONE metadata?

• underlying question: should GIDcollections be uniform containers without substructure when seen from the outside?
• would be very easy to implement / re-implement under the hood.
• easy and clear API
• having internal structure would require additional iterators/methods to access stuctural information.

Option

If (different metadata OR different model_id): don't merge primitives but keep separate in composite

• would require accessor to different metadata pointers
• extending the iterator returned struct is not efficient solution
• => keeping different metadata in a single GIDCollection would bloat the class a lot.

Decision

• GIDCollections are HOMOGENOUS CONTAINERS.
• structure between different populations has to be taken care of by the user.
• if structure handling becomes problematic a new container class for the specific case can be added (external of GIDCollections)

[stinebuu]

We will introduce fingerprints in order to properly handle ResetKernel(). After using ResetKernel() all metadata etc. will be gone, but we will still have the GIDs and/or other data the user might have extracted. It is important that we are not able to use these GIDs in functions such as Connect, because the user might have created a new set of metadata that does not correspond to the old GIDCollection, and errors will occur.

To solve this problem we introduce fingerprints. This will make it possible to check whether the GIDCollection is valid. We will introduce a fingerprint in the Kernel, and in the GIDCollection class. As a fingerprint, we will use a timestamp. Every time ResetKernel() is called, the fingerprint in the Kernel will be updated. When a GIDCollection is created, we can retrieve the fingerprint from the Kernel, and store it in the GIDCollection class. It will then be possible to test against the Kernel fingerprint and see whether the GIDCollection is valid when we use Connect or other functions that rely on correctly created GIDCollections. If we have an invalid GIDCollection, an IncorrectGIDCollection will be raised.

[hakonsbm]

To update on current progress on GIDCollections:

Everything works now, except for

• Topology module, which is in the progress of having subnets replaced with GIDCollections.
• Sending a Numpy array with GIDs and a GIDCollection to Connect, as described in #369. But this should be easy to implement.

[jougs]

@hakonsbm: Can you please copy and update the information from this issue to a new document under extras/userdoc/md/in our development branch. I'll close this issue once you did that. Many thanks!

[hakonsbm]

@jougs I have created a new document in extras/userdoc/md in our development branch with information from this issue, as you requested.

[jougs]

Many thanks. As the information is now in a safe place, I close this.