Serverside culling e.g water

[Ragzouken]

It's easy enough for the client to not render water molecules at will, but for large simulations the inclusion of water in frame data is wasteful and more critically might blow over the size limits of our messages. Do we want to support some means serverside of culling particles--either by residue or particle index or something else?

On the NoH20 branch this is done simplistically by filtering out atom and bonds above a certain index (where water is partitioned to the back half of the simulation), which is provided as yet another server argument. This relies on the simulation input being ordered in a specific way and a manually provided cutoff number, which seems clunky.

Once option could be to provide a flag or utility function to do this preprocessing and cutoff automatically. Another could be to filter by residue name.

Another axis to this is whether the filtering is done globally for all subscribers, or is something that can be requested at the time of frame subscription (similar to how frame interval is).

[Ragzouken]

Also I realised any approach that reorders atoms in a non-global way could be troublesome because the atom indices will not be universal--at least one side is going to need to keep track of atom index in simulation to atom index in frame (does this already exist?) to so that the indexes used for interactions map to the right atoms.

[jbarnoud]

This comment ended up a live dump of my reasoning. TL;DR: it is doable with server-side bookkeeping.

---

reorders atoms in a non-global way could be troublesome

I was going to comment on that. While it is most common to have all the water at the end of the simulation, it can happen that a few water molecules are sprinkled here and there when they are part of the crystal structure. Also, allowing a filter on arbitrary residue names means it can be something else than water that gets filtered out; then there is no guarantee anymore.

It is not impossible to do that reordering, though! We need to account for 2 things, though:

• performance: we do not want to do the reordering for every frame because it is going to be slow
• synchronisation: the state between the client and the server can be out of sync if the topology changes server-side

For the synchronisation issue, we can:

• only activate the feature when we are sure the topology won't change. In many cases, a server will only run a single simulation and its topology won't change. In this case, we could ignore synchronisation issues: the map will remain the same all along the server's lifetime.
• do server side book keeping
• provide the map with the frame: the server could send the map in the FrameData so the frame is self-consistent. We still suffer from the synchronisation issues between frames and state that we have without the feature.

Limiting the scope of the feature is the most robust solution. In that scenario, there is no synchronisation issue whatsoever, not even between the frame stream and the state one. We could have all the mapping done server-side and, therefore, not have to modify the client. Here, the topology is altered before being sent, which is done only once so has a limited cost, and the server maintains a map that links the atom indices known by the client to the indices known by the simulation engine. This map is generated only once. It is read when computing interactions to convert indices from the shared state into indices for the MD engine. The server also needs to create a mask that is used when writing the FrameData to send.

The same approach could be used without limiting the scope of the feature. However, the server then needs to be careful about when to update the map and the mask.

We could include the map into the FrameData and let the client translate what it sends back to the server. However, the server still needs to keep track of the mask to generate the FrameData.

I can see a 2 step fix for the issue:

1. do server-side bookkeeping and limit the scope of the feature to non-changing topologies
2. extend the feature to allow for changing topologies