Question: Best way to view data in the DPS store (Redis)?

[phcaptjim]

We are using Redis in our streams application to do lookups and this works quite well. Where we struggle is knowing exactly what is in the data store at any given time. I do have print statements built into the SPL logic and if I review the console logs I can see the information that way but I am hoping for a simpler solution and wanted to see what this group has done. Any ideas would be greatly appreciated! Thanks!

[brandtol]

You could write a small standalone application that uses the dpsBeginIteration(), dpsGetNext() and dpsEndIteration() methods to loop over the key/value pairs in a given store and print them out. This might get complicated if other applications modify the store during the dump.

[brandtol]

You can use the "keys" and hkeys" commands on the redis-cli, to list all existing keys.

If you use the dpsPutTTL(), dpsGetTTL() APIs, there are two new parameters that let you specify if the keys and values shall be encoded. In that case the keys are shown in cleartext. The values can also be written in cleartext, which probably only makes sense for simple rstring values.

On the redis-cli output might look like this (testkey1 was written by the app)

127.0.0.1:6379> keys *
1) "testkey1"
2) "dps_and_dl_guid"
127.0.0.1:6379> get testkey1
"testval1"

the SPL application to use these APIs looks like this

use com.ibm.streamsx.store.distributed::*;
composite Main
{
    graph

    () as Test = Custom()
    {
        logic   
        onProcess : 
        {
            mutable uint64 errorCode = 0ul; 
            mutable rstring returnVal = "";

            // write unencoded key/value
            dpsPutTTL("testkey1", "testval1", 100u, errorCode, false, false);
            println("errorCode:" + (rstring)errorCode);

            // read back
            dpsGetTTL("testkey1", returnVal, errorCode, false, false);
            println("errorCode:" + (rstring)errorCode);
            println("testkey1 = " + returnVal);
        }
    }
}

[phcaptjim]

Thanks for responding @brandtol

Are these print statements only viewable in the console log? I am trying to get a real time 'dashboard' if you will of our data in the store.

[brandtol]

Yes, they are only viewable in the log/console. But the SPL code in my second comment was only meant to show how to use the APIs that allow storing unencoded keys/values in redis.

The basic idea of comment 2 is to use the redis-cli to list all keys currently in the db. That solution would not require a Streams application at all, instead you could probably wrap it in a shell script or so. Problem is, that the keys are base64 encoded, unless you use the dpsPutTTL() with second last parameter set to false. But you could also deal with the base64 decoding from a shell/perl script using proper tools.

The idea of my first comment is to use a streams application to iterate over all keys and dump them somewhere. With this solution you are free to do anything with the data returned. You can write it to a file, or send it to a websocket to feed some java script dashboard (using the inet toolkit).

Please let us know a little bit more about your environment/requirements 1) do you want to create the dashboard independently from your streams app ? That could be done by periodically fetching all keys from redis using the command line tool and display results by whatever technology you want to use 2) do you use DPS stores, or are you using the dpsXXXTTL() methods, which write into the database global namespace. 3) are you ok with displaying only the keys in your dashboard or do you need to displays the values also ? 4) do you use a simple datatype as value (string,integer) or do you have a complex structure ?

[phcaptjim]

We are using DPS stores. Right now we are just using one in our streams app but we are about to start using two stores. Our current store has information we would like to see in a dashboard and see the changes as they occur in real time (assuming it's not a monumental effort). This is not a must-have requirement just something we would like to do if the lift is not too large. As for what we display, we would like to see the values and not the hex keys. Also, we are using a simple datatype value. We have been playing around with a Redis gui called Redis Dekstop Manager which is pretty decent (connects via SSH).

[brandtol]

Ok, when using stores, you cannot use the unencoded key/values I mentioned in comment 3. I had a quick look at the Redis desktop manager and it seems to have a feature to plugin your own value formatter scripts/programs. http://docs.redisdesktop.com/en/latest/features/#native-value-formatters

I think think that may work for you. But will probably require some effort.