Exposes client PID, logical connection name/uuid and tags in PlatformConnectedClient

[chrisdennis]

As discussed in various meetings we should add a tag to the connection identifier to allow the user of the voltron api to 'tag' a voltron connection with a human-readable name. This allows for easier parsing of the connection-ids anywhere that they are referenced, be that the server logs, the client logs, or various locations in the M&M stack where this information is used or extended.

[mathieucarbou]

CC @anthonydahanne @ankitamaurya :

Talked to @jd0-sag. He will bring up this feature to the server team. We need it for M&M Slice 2.

Basically, a client will be able to do something like this:

        Properties properties = new Properties();
        properties.setProperty("tag", "123@127.0.0.1:jetty:app:3");
// or, by using the ClientIdentifier from terracotta-management:
        properties.setProperty("tag", ClientIdentifier.create("jetty:app:3").toString());
        this.connection = connectionService.connect(uri, properties);

And the tag will be exposed into the voltron monitoring tree in the PlatformConnectedClient object.

[mathieucarbou]

@jd0-sag , @myronkscott @ramsai1729 : would it be possible to directly expose in PlatformConnectedClient the whole Properties object used to create the connection ?

The objects could also be "extended" to contain connection.uri, which would be the String representation of the URI used to connect.

This way it would be possible for us, by recovering this properties object from the PlatformConnectedClient class, to get:

• connection.timeout : optional
• connection.uri : always there // uri used to connect
• connection.tag : string used to tag this connection
• any other user-defined properties

What do you think about this ?

I see several benefits for this, including:

• ability to show in TMC how the connections have been created
• easier shield support / troubleshooting
• easy to extend and provide some other user-defined properties for eventually other services or monitoring use cases

(CC @anthonydahanne)

[jd0-sag]

I would generally want to avoid something so open-ended since these properties are really more about interacting with the underlying ConnectionService implementation in on-general "hint"-like ways and pushing any of that data through to the server is arguably the wrong approach.

For a defined key, it seemed like a clearer approach than adding to the API in ways which would otherwise seem somewhat redundant (next to Properties). Even that is somewhat questionable, though.

[mathieucarbou]

OK! So if we forget about any other user-defined properties, could be still have all the other 3 properties exposed ?

[jd0-sag]

My thinking is either 1 (as it gives something we could generally output as a human-readable identifier, in many situations) or 0 (and connect this other arbitrary data/behavior through a different path).

You are able to associate the PlatformConnectedClient identifier with the other cases where you need to know this information, on your side, right?

[mathieucarbou]

Right, but I was just asking for connection-specific properties:

• connection.timeout : optional
• connection.uri : always there // uri used to connect
• connection.tag

If we modify PlatformConnectedClient to expose the tag set by the user, I just say we could also expose the properties used to create the connection. Things that we do not have access locally from the management library and could be exposed by the platform as easily as the tag.

I think this will help support, troubleshooting and also be interesting to have that from a M&M point of view.

[jd0-sag]

The problem is that this is producing more hidden API: instead of being able to ask for specific data, we are looking up "magic keys" in a Properties object. If we wanted specific pieces of data, we should just add them to PlatformConnectedClient, directly.

Even if we were to pass the entire map through (which may be something to consider, for general opaque monitoring, not decision-making - still wary of this, though) we would still need these pieces of data to be top-level (since "connection.uri" is a valid key in the map - which one do you see?).

[mathieucarbou]

Ok. So we know that PlatformConnectedClient will expose a sort of tag (== ClientIdentifier for us)

Now, here is my question.

How could we access, from the Tms Entity, the connection properties (_timeout_, _uri_) 
that the client set when connecting to the server ? => to know HOW it connected.

Using some management module on client-side to do that is not a solution because not all client will have a management module and the management module is just about exposing the client-side objects. So it has nothing to do with the connection.

[jd0-sag]

There isn't currently a way to do that since this is a new requirement.

What information do we currently want to expose to M&M regarding these client-side network layer options? Do we actually want this "tag" capability or is this underlying map the data that M&M actually wanted?

If the answer is "both", then we can't use this map to implement the tag and we will have to change the API to make it a first-class piece of information. Arguably, if the server-side wanted to know about this for decorating logs, etc, then it should be part of the API, directly, and not slid in through the properties which are really more about the implementation details of the ConnectionService, treated as opaque by the rest of the platform.

[mathieucarbou]

I'll ask some people to be sure. Because what we are sure for now is that we need a way to pass a string (client identifier) when connecting. What I am not sure yet is what we'll need to know about how the client connected. So let's answer that and then I'll come back to you :-)

[mathieucarbou]

FYI:

ClientIdentifier class is currently in terracotta-management: source | test

We use this class / representation for client tagging in M&M.

<PID>@<CLIENT-ADDRESS>:<CLIENT-APP-NAME>

CC @cljohnso

[mathieucarbou]

@cljohnso @jd0-sag @chrisdennis : about ClientIdentifier integration, here is a proposal:

When a cluster client will create a connection (connectionService.connect(uri, props)), he will need to pass a string tag representation of a ClientIdentifier. This class constructs a client identifier such as <PID>@<CLIENT-ADDRESS>:<CLIENT-APP-NAME> and has several constructors:

• create() => create(generated-short-uuid)
• create(app-name) => create(discovered-pid, discovered-lan-address, app-name)
• create(pid, address, app-name)
• valueOf(string)

The idea would be to allow users to put in the ehcache.xml file an optional attribute with their application name. If the attribute is there, then you can call ClientIdentifier.create(app-name) and if it is not there, ClientIdentifier.create(). The ClientIdentifier class discovers the PID and lan address.

Programmatically, here is how it could looks like:

    final CacheManagerBuilder<PersistentCacheManager> clusteredCacheManagerBuilder =
        CacheManagerBuilder.newCacheManagerBuilder()
            .with(ClusteringServiceConfigurationBuilder.cluster(URI.create("http://example.com:9540/my-cache-manager?auto-create"))
                .clientIdentifier("jetty/EhcachePounder")
                .defaultServerResource("primary-server-resource")
                .resourcePool("resource-pool-a", 128, MemoryUnit.GB)
                .resourcePool("resource-pool-b", 128, MemoryUnit.GB, "secondary-server-resource"))
            .withCache("clustered-cache", CacheConfigurationBuilder.newCacheConfigurationBuilder(Long.class, String.class,
                ResourcePoolsBuilder.newResourcePoolsBuilder()
                    .heap(10, EntryUnit.ENTRIES)
                    .with(ClusteredResourcePoolBuilder.fixed("resource-pool-a", 32, MemoryUnit.GB))));

Then the clientIdentifier() method will call ClientIdentifier.create(name). If clientIdentifier() is not called, then an identifier can be generated for this app with ClientIdentifier.create()

Note: I am wondering if we can re-use some part of the cluster URI to make the client identifier ? Do we have ONE server entity per client application, OR we can have 1 client web application that for exemple connects to several cluster URIs ? I guess so, so .clientIdentifier("jetty/EhcachePounder") would be the same for different cluster URI.

This proposal would also any user-made entities to use the ClientIdentifier. As long as a user uses the ClientIdentifier class to generate a unique tag that he can re-use across different connection creation.

I also propose we move this class at a common place to avoid duplication. It could be in terracotta-apis. Or copied in each project that needs it, but it's not really good I think... You might see it as a class that can be used client-side to help generating a logical cluster client identifier, that should be in this format: <PID>@<CLIENT-ADDRESS>:<CLIENT-APP-NAME>.

Let me know what you think about! We can do a hangout to discuss it.

[mathieucarbou]

Meeting conclusion:

Ehcache:

.tag("jetty")
.tag("EhcachePounder")

Server exposes:

- clientAddress => already there
- pid => PlatformConnectedClient.pid

Also, server will need to provide a way, from a server entity, when a client connects, to recover the right clientIdentifier used to recover the PlatformConnectedClient object from the server monitoring tree, so that we can match the entity call to the connection information exposed.

At connection time: "pre" connection step to publish client metadata (will be a new module in terracotta-management project)

root
  | --platform
         |-- connections
                |- <generated-voltron-conn-id> = PlatformConnectedClient (ip, port, PID)
                              |- metadata = ManagementMetadata (product name, tags)

From the metada-connection-entity

producer.addNode({platform, connections, <id>, metadata}, MetadataObject)

[mathieucarbou]

UPDATE following monday's meeting and the 2 meetings on tuesday.

Goals:

• M&M needs to be able to consolidate all the physical connection information exposed in Voltron tree to regroup connections based on connection uuids (to regroup cluster connections together == logical connection)
• M&M also needs to create the client identifier from PID, client addr, connection UUID and regroup all the logical connections having the same tags / client identifier under the same Client object.
• M&M needs to have a way to easily display a client (== client identifier) and filter them (== tags)
• M&M needs to be able to identify a connected client to a cluster thanks to a client identifier, and then lookup the server entities this client is connected to.

Terminology:

• Connection Uuid: string passed when creating a TC connection
• Tags: set of strings passed when creating a TC connection
• Physical connection == PlatformConnectedClient : client port, client addr, server port, server addr, connection uuid, tags, client PID
• Logical connection == connection to a cluster : grouping of several physical connections having the same connection uuid (multi-stripe)
• ClientIdentifier = <PID>@<ADDR>:<LOGICAL-CONNECTION-UUID> : identifier that will be used in M&M to identify a client in the topology
• Client == ClientIdentifier + Tags + etc.. : client object representing a cluster client in M&M. Identified by a ClientIdentifier

Example:

The same application server (one node) serves 2 web applications, both connecting to the same clustered cache manager on a cluster (2 stripes). User adds the tags {webapp-1, node-1} for the first webapp and {webapp-2, node-1} for the second webapp.

The app server PID is 1234 and is at 10.0.0.100

We would have a topology like this:

cluster
   |-- CLIENT ( ClientIdentifier = 1234@10.0.0.100:CONN-1a4b86 ; 
   |      |     tags = { webapp-1, node-1, ehcache, my-cache-manager-name } )
   |      |-- Physical Connection to stripe-1 ( client ip = 10.0.0.100, 
   |      |     == PlatformConnectedClient      client port = 5678, 
   |      |                                     server ip = 10.0.0.200, 
   |      |                                     server port = 9510, 
   |      |                                     client pid = 1234, 
   |      |                                     tags = { webapp-1, node-1, ehcache, my-cache-manager-name }, 
   |      |                                     connection uuid = 1a4b86)
   |      |-- Physical Connection to stripe-2 ( client ip = 10.0.0.100, 
   |      |     == PlatformConnectedClient      client port = 5679, 
   |      |                                     server ip = 10.0.0.210, 
   |      |                                     server port = 9510, 
   |      |                                     client pid = 1234,
   |      |                                     tags = { webapp-1, node-1, ehcache, my-cache-manager-name }, 
   |      |                                     connection uuid = 1a4b86)
   |-- CLIENT ( ClientIdentifier = 1234@10.0.0.100:CONN-d839ef ; 
          |     tags = { webapp-2, node-1, ehcache, my-cache-manager-name } )
          |-- Physical Connection to stripe-1 ( client ip = 10.0.0.100, 
          |     == PlatformConnectedClient      client port = 5680, 
          |                                     server ip = 10.0.0.200, 
          |                                     server port = 9510, 
          |                                     client pid = 1234, 
          |                                     tags = { webapp-2, node-1, ehcache, my-cache-manager-name }, 
          |                                     connection uuid = d839ef)
          |-- Physical Connection to stripe-2 ( client ip = 10.0.0.100, 
          |     == PlatformConnectedClient      client port = 5681, 
          |                                     server ip = 10.0.0.210, 
          |                                     server port = 9510, 
          |                                     client pid = 1234,
          |                                     tags = { webapp-2, node-1, ehcache, my-cache-manager-name }, 
          |                                     connection uuid = d839ef)

Ehcache opens a connection to voltron with a uuid to identify the physical connection (?) and the tags {ehache, my-cache-manager-name}

Todo for server team:

1. Support a string for the name/uuid + a list of tags when creating (fetching ?) a connection
2. Expose (in PlatformConnectedClient ?) the client PID, tags and name/uuid

cc @ljacomet @cljohnso @jd0-sag @anthonydahanne @chrisdennis

[anthonydahanne]

@mathieucarbou would all this information (topology above) be published in the monitoring tree ? On a user point of view (read tmc), I guess only the logical information makes sense to be published (but I understand there are other needs than just M&M here)

[mathieucarbou]

@anthonydahanne : the tms backend topology needs to have all this information, whether it is shown or not in the TMC. Yes it needs to be in the monitoring tree. Physical connections are already there in the back end topology also. This is what allows us to link connected clients to the server entity they use. Event if it is not represented in the UI, it is there in the back end.

[mathieucarbou]

@jd0-sag : issue automatically closed because of the commit message. You we re-open it, as you told me yesterday that the logical connection uid could be added there ?

[jd0-sag]

This issue has been re-purposed too many times to encapsulate too many divergent ideas. I have opened #72 to handle the specific details of logical connection name to keep the intent clear.

This means that the existing item describes the platform-intrinsic data while #72 will focus on the API-level, extrinsic, data we want to pass through from the client to monitoring.