Support dynamic updating of distributed sparse matrices

[buddha314]

For some it may be easiest to think of this as a graph problem.

A large, distributed, sparse matrix with sparse subdomain SD keeps the state an application or process. In the graph analogy, if A[17,71] = 1.3 then vertices 17 and 71 have relationship strength 1.3. Should this state change, like 71 says something mean to 17, their edge strength may decrease to -3.14. Or new edges may be created between 71 and 97 that were not in the original sparse subdomain.

If the edge strengths are created by a function called canWeBeFriends(A, i, j), what is the proper way to send in A and update it with a new entry? Maybe...

• @bradcray suggest also sending in the subdomain as an argument. I haven't tried that, but it seems kinda weird.
• Provide a method that extracts SD from A into a new subdomain
• Provide deep copy of A into B, alter B's subdomain and return B, destroying A (like the teleporting phone booth)
• Other ideas?

[buddha314]

In a side conversation with @ben-albrecht and @bradcray it was noted that domains can be shared. So if A and B are sharing a domain and I mess with A's domain, B is now confused, sad and a danger to himself. Perhaps make some domains "private" or "public". I don't know if this is a good idea, just suggesting it.

If the only path forward is to keep the domain in scope, then this needs to be a focus in all examples, since this is not a common pattern.

[buddha314]

One more question, if keeping the domain in scope is the only option, how does one persist the matrix to disk then re-load it?

[ben-albrecht]

Sparse file formats typically track the indices in some form or another, so the domain would be built from those indices upon being read in.

[buddha314]

Yeah, but how do I get to it to manipulate it? E.g.

• Build domain, matrix AD, A
• populate A[SD]
• Save A to disk
• Open A from disk, add an entry to SD?

[ben-albrecht]

A simple version of a sparse array reader might loop over the elements, populating both SD and A together, e.g.

var SD: sparse subdomain(D);
var A: [SD] real;

...

for line in f.lines() {
  const idx = readIndex(line);
  SD += idx;  
  A[idx] = readValue(line);
}

where a simple sparse format might look like this:

(1, 4) 15.23
(2, 5) 27.90
(4, 2) 9.13

[ian-bertolacci]

7730 would probably be solved by support for this.