As an administrator, I want to be able to see a list of running, queued, and "remembered" jobs via a tool that monitors the persister

[epag]

---

Author Name: Chris (Chris) Original Redmine Issue: 88805, https://vlab.noaa.gov/redmine/issues/88805 Original Date: 2021-03-02 Original Assignee: Hank

---

The original Description is below. Instead of dumping a list of the running jobs through an endpoint, we are proposing to properly allow an administrator to monitor the persister/Redis. And instead of targeting users with this capability via the WRES GUI, we decided that displaying the list isn't really what is needed. What is needed is enough workers to ensure no one waits and some way to track progress of the evaluation. A ticket for tracking progress already exists, iirc. This ticket will be for the monitoring tool.

Original description below. Thanks,

Hank

=======================================

Periodically, I will need to query the COWRES to get a collection of information on runnings and queued jobs. I can turn around and use this data to present to a user what is going on with their work. Ideally, I'd be able to hit a view via nwcal-wres-prod.[host]/jobs and get:

{
    "jobs": [
        {
            "id": 64464646464888,
            "name": "Example 1",
            "status": "in_progress"
        },
        {
            "id": 984968684684,
            "name": null,
            "status": "in_progress"
        },
        {
            "id": 33348954684,
            "name": "Example 4",
            "status": "in_progress"
        },
        {
            "id": 8468452684465,
            "name": "Example 5",
            "status": "queued"
        }
    ]
}
</code>

The name field is a nice to have, but it is available within the project configurations.

---

Redmine related issue(s): 112668

---

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2023-02-07T14:50:26Z

---

I think we may be talking past each other in the sense that I don't mean the database or whatever persistence store is used under the hood by redis, I mean all redis traffic, essentially the persister service.

You should not be able to visit the management console without the correct credentials and be able to see anything at all about the persister, even basic information. Securing a database sounds too low-level to me, although I am not that familiar with redis.

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2023-02-07T14:52:20Z

---

But I agree that we want to secure the management console in the first instance, not something behind it.

[epag]

---

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-02-07T14:58:50Z

---

Not talking past each other. I'm just not using precise language.

I think we may be talking past each other in the sense that I don't mean the database or whatever persistence store is used under the hood by redis, I mean all redis traffic, essentially the persister service.

That's what I meant.

Hank

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2023-02-07T15:07:29Z

---

OK. Just need to work out how to do that, then, but I think you're right that it's suboptimal if we cannot secure the management console itself.

[epag]

---

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-02-07T19:27:52Z

---

I emailed ITSG asking about how to do what I wanted: limit access to RedisInsight. He recommended ssh tunneling using the @ssl -L@ option so that I can reach it as if I'm on the localhost. Though that allows me to connect to RedisInsight as if I am on the -dev03, i.e., using localhost:8001, I still have to expose 8001 outside of the container, and, by default, that means exposing it the outside world.

Further research led me to test this which is supposed to limit access to the 8001 port to the localhost:

    redisinsight:
        ports:
         - '127.0.0.1:8001:8001'

Theoretically, if I had a browser on the -dev03, I should be able to connect to @localhost:8001@ using that setting. A @docker container ls@ shows this:

a9be8a57c3df redislabs/redisinsight:latest "bash ./docker-entry…" About an hour ago Up About an hour 127.0.0.1:8001->8001/tcp

Unfortunately, that does not mean ssh tunneling will allow me access. My browser cannot see it if I include the 127.0.0.1 prefix.

It certainly appears as though RedisInsight cannot be properly limited to only admin users if I were to use that tool. If we need a Redis monitoring tool, it may need to use a different capability, perhaps the prometheus-Grafana route you mentioned. Given how this is really not critical, however, it might be enough just to stick with @redis-cli@, but perhaps identify the commands I need to perform additional actions beyond just listing keys.

Hank

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2023-02-07T19:44:52Z

---

That isn't the way to go, certs or some other authentication (e.g., user/pass) is needed. Restricting use to anyone with access to a machine is quite different from authenticating to a service.

But, as always, the question has usually been posed before:

https://stackoverflow.com/questions/71057840/how-to-secure-redisinsights-on-a-docker-container

Doesn't look promising. As noted in the comment, though, slapping a reverse proxy in front of it would be an option.

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2023-02-07T19:49:51Z

---

I prefer that solution to securing low-level data feeds - only the intended users should gain access to the gui.

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2023-02-07T19:51:27Z

---

https://docs.redis.com/latest/ri/using-redisinsight/proxy/

[epag]

---

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-02-07T19:56:38Z

---

I must have left something key out of my search. That didn't come up for me. :)

I was willing to live with access restricted to folks with access to the entry COWRES machine, because its a very restricted pool of individuals who are generally trustworthy. I wouldn't have been happy if this were a critical, operations-supporting service, but, let's face it, that isn't the WRES. Still, I agree a localhost only solution is not as clean as mutual-TLS or user-password. Those solution, however, would require additional maintenance, and that's when it it becomes hard to justify deploying RedisInsight.

Let me review what you found,

Hank

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2023-02-07T20:03:01Z

---

I don't much care about what option we go with right now for the reasons you mention, but it's good to document the issues here so that future readers can do better when the time comes to do better. I take your point about machines at nwcal, but defense in depth and all that :-) Or, in other words, if doesn't justify doing properly, it probably ain't that important (right now and/or relative to other things) and we should focus our efforts elsewhere.

[epag]

---

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-02-07T20:10:40Z

---

Took at look at both URLs and it makes sense. Still, I'm leaning toward not installing RedisInsight, but, rather, keeping it in our back pocket if such a tool ever proves to be more important than it is right now. As you said, " if doesn't justify doing properly, it probably ain't that important (right now and/or relative to other things) and we should focus our efforts elsewhere." It was good learning experience, though. Still lots to learn about Docker and Redis.

I'll leave the ticket on-hold and slip it back into the Backlog. To summarize where I left off, here is what the RedisInsight entry in the compose .yml looks like at the moment:

    redisinsight:
        ports:
         - '8001:8001'
        image: redislabs/redisinsight:latest
        volumes:
         - /home/Hank/redisinsight:/db
        networks:
            wres_net:

To view the @persister@ Redis instance, you need to visit RedisInsight at http://[COWRES entry machine]:8001. Click on Add Redis Database and Connect to a Redis Database, and specify the host as "persister", port as "6379", and give it a reasonable name.

That's it for now. Thanks,

Hank

[epag]

---

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-02-15T16:03:30Z

---

RedisInsight was, indeed, valuable in debugging #112668, so there is perhaps some value in spending the time needed to add it and secure the endpoint, likely via a reverse proxy. For now, I'm going to note the .yml configuration in #88805-63 and keep that in my back pocket if it becomes useful to stand it up in staging to debug an issue, but we haven't configured the reverse proxy to secure the endpoint. We can add it temporarily to the .yml and spin up the container on the fly.

Hank

[epag]

---

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-02-15T17:27:56Z

---

Rather than keep it in my back pocket, I setup a RedisInsight @.yml@ in staging, @/mn/wres_share/deployment/compose-redisInsight.yml@, that can be brought up by this command:

cd /mnt/wres_share/deployment

docker-compose -f compose-redisInsight.yml up -d

Without the @-d@, the container logs directly to the terminal and the prompt won't come back.

The @.yml@ at the moment includes the RedisInsight @service@ previously noted, but also the definition of @wres_net@ pulled from the current entry @.yml@. If I don't include the @wres_net@ definition, it can't talk to the network and fails to come up. However, with it included, when the RedisInsight is brought @down@, it tries to bring down @wres_net@ and fails because its still in use; I'd rather it not even try to bring it down. I'm going to look into alternative solutions for pointing RedisInsight at the @wres_net@ network defined in another .yml.

Note that if anyone brings up RedisInsight in this way, they must be sure to bring the container back down and should also remove the RedisInsight image from the local Docker image collection.

Thanks,

Hank

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2023-02-15T17:30:40Z

---

If we're going to use it, then I think it needs to be part of the overall service composition (within the main compose file), not separate.

[epag]

---

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-02-15T17:41:30Z

---

Long term, yes, but standing it up and securing it behind a reverse proxy will take a bit of effort to figure out. I'm planning to shift back to the WRES GUI (new Features panel) once we deploy 6.11 and won't have the time to fight through setting up the proxy. So, I figured I would create a quick, one-off tool that can be spun up and torn down to support looking into the persister without having to use @redis-cli@. I'm not even sure I'll put it in the repo; more for my own sake.

I found @external: true@ here:

https://stackoverflow.com/questions/38088279/communication-between-multiple-docker-compose-projects

Unfortunately, when I attempt that, I get this:

[Hank@nwcal-wres-ti02 deployment]$ docker-compose -f compose-redisInsight.yml up -d
ERROR: The Compose file './compose-redisInsight.yml' is invalid because:
services.redisinsight.networks.wres_net contains unsupported option: 'external'

D'oh! Our Docker must be too old.

Lunch is coming, so I'll probably leave it at that. Yes, its a crude way to stand it up, but, as I said, I'm just planning to use it to spy into redis for a few minutes before disposing. I'm also thinking about this more as a tool in staging; that is, reproduce issues in staging, and only then bring up RedisInsight to take a look so as to avoid impacting production.

Note that, if 6.11 will take a few more days to start deployment, then I might go ahead and dig into the reverse proxy solution. I'm just worried it may take a while to figure that out even with the examples available online.

Hank

[epag]

---

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-02-15T18:13:34Z

---

The compose .yml that works is below. My previous problem with @external@ was because of a misunderstanding.

Stepping away, again, but might return to work on the reverse proxy if time allows. Again, I need to return to the WRES GUI and work on a new Features panel, but plan to do so after 6.11.

Hank

===================================================

[Hank@nwcal-wres-ti02 deployment]$ more compose-redisInsight.yml 
version: '2.3'
networks:
    deployment_wres_net:
        external: true

services:
    redisinsight:
        ports:
         - '8001:8001'
        image: redislabs/redisinsight:latest
        volumes:
         - /home/Hank/redisinsight:/db
        networks:
            deployment_wres_net:

[epag]

---

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-02-15T20:00:47Z

---

Per the dev call last week, I'm repurposing this development ticket.

This ticket will now be about installing a monitoring tool that allows us to see the running, queued, and past, "remembered" jobs by examining the persister. We don't really see the value, at this point, of having the WRES GUI display that information for the user. The concern is that the information we could provide may not be all that helpful; i.e., it may not be what they really need to see, which is information that can tell them how long it will be until their evaluation completes. There is already a ticket for that (can't recall the number) which will require under the hood COWRES changes, beyond just implementing a monitoring tool.

The monitoring tools I'm proposing is RedisInsight, which has already assisted us in debugging issues with the persister, as noted above. Also as noted above, the problem is that the endpoint cannot be readily secured: by exposing RedisInsight, we are allowing every user to see and manipulate the persister contents. Thus, we need to put it behind some sort of security, and a reverse proxy may solve that; again, see above comments.

Thanks,

Hank

[epag]

---

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-02-17T18:49:53Z

---

I had some spare time today, since I don't want to start on something new with just a few hours left on a Friday, and was able to figure out the reverse proxy using nginx, though it likely needs more work to make it "operational".

As it stands, I'm able to force a request posted to http://[entry machine]:9000/redisinsight to a standard, NGINX user/password login dialog. With a specific name and password, it will forward the request to RedisInsight. Otherwise, no dice. And if I try to go straight to port 8001, 9000, or 8001/redisinsight, it does not provide the RedisInsight interface, but, instead, a blank screen or forbidden.

The files to make this happen are as follows:

• Configuration in @docker-redisInsight.yml@:

version: '2.3'
networks:
    deployment_wres_net:
        external: true

services:
    redisinsight:
        ports:
          - '8001:8001'
        image: redislabs/redisinsight:latest
        environment:
          - RITRUSTEDORIGINS=http://localhost:9000 # This is the proxy origin from brow
          - RIPROXYENABLE=t                        # Enable Subpath Proxy
          - RIPROXYPATH=/redisinsight/        # Set static proxy subpath
        volumes:
          - [omitted]/redisinsight:/db
        networks:
          - deployment_wres_net 

    nginx-subpath-proxy:
        image: nginx
        volumes: # Your nginx config
          - ./nginx-subpath-proxy.conf.template:/etc/nginx/templates/nginx-subpath-proxy.conf.template
          - ./.htpasswd:/etc/nginx/.htpasswd
        ports:
          - "9000:9000"
        environment:
          - NGINX_PORT=9000
          - NGINX_PROXY_PATH=/redisinsight/       
        depends_on:
          - redisinsight
        networks:
          - deployment_wres_net
</code>

• NGINX configuration file, @nginx-subpath-proxy.conf.template@:

server {
 listen ${NGINX_PORT} default_server;

 root /etc/nginx;
 index index.html;

 location ${NGINX_PROXY_PATH} {                        # Subpath
     proxy_pass             http://redisinsight:8001/; # Assumes redisinsight runs in this host
     auth_basic            "Administrator’s Area";
     auth_basic_user_file  ./.htpasswd;
     proxy_read_timeout     900;
     proxy_set_header       Host $host;
 }
}

• An .htpasswd file containing the username and password. Its a text file with a single line, "wres-admin:[encrypted password]", with the password encrypted via @openssl passwd -apr1@ (the "-apr1" specifies the compatibility).

Before operationalizing, I will need to add TLS, so that its https, and perhaps make that mutual in order to remove the need for a user-name and passwd, though we have a precedence for using either approach: the broker uses user-name password, while the events broker uses mutual TLS. I'm also going to create a directory called @/mnt/wres_share/redisInsight@ to contain all of the required files.

Hank

[epag]

---

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-02-17T19:43:22Z

---

Another tricky part...

When you first go into RedisInsight, its accesses a configured local directory checking for files that indicate previous use. If it does not find any, then it will ask the user to review terms and signal agreement. When doing so behind a proxy, however, it fails to save your agreement and reports, "if you are behind a proxy, please set the environment variables." (paraphrasing) The problem is, I have set the environment variables and the message is unclear as to what variables are missing. The redisInsight docker log doesn't log the issue, either. I wasn't able to figure it out.

So, to get past that, I needed to redeploy without the reverse proxy, click agreement to terms, and then redeploy again using the reverse proxy. From then on, it behaves as expected.

One other thing I learned: the redis database can be username-password protected, and redisInsight can be setup to ask for that username and password. That would have been one way to protect our database if we deployed without a reverse proxy.

I think that's all I'm going to do today. Getting closer every time a spend a couple of hours in research, but still not quite there.

Hank