Grafana selinux policy

[sfeifer]

Hi!

I am working on developing a selinux policy for grafana and this seemed like a good spot to start. Is there interest in collaborating on this?

[georou]

G'day mate,

Good to hear you're interested in working with SELinux. I'm more than happy if you want to extend/fork the policy from my work here.

Currently, I don't have Grafana in my stack and so the policy might be a bit behind current releases. I imagine it needs a bit of work bringing it up to speed.

With my limited free time presently, I'm not sure when I could personally get around to looking into.

Is there any aspect you particularly want to focus on or having trouble with?

Cheers.

[sfeifer]

No problem. I'll fork and work on the policy from this starting point.

Thanks for offering to answer questions! I am very new to selinux policy development, so I still have some work to be done in terms of learning more background information. When I do encounter points where I'm having trouble should I make a new issue in this repository like I did to begin this thread?

[georou]

No worries. You can just reply to this thread if that's easier.

If you're after immediate help, I'd also suggest hopping on to the #selinux IRC on LiberaChat - plenty of users there are also happy to answer questions or provide feedback.

[sfeifer]

Great! Thanks

[sfeifer]

Hi,

I've been exploring more with selinux, looking at the policies for other tools, and testing this one. I'm wondering why the type is named "grafanad" instead of grafana. Could I get some insight on this?

Thanks.

[georou]

Hi,

It's basically the naming convention I chose at the time. The 'd' representing daemon as that's what the policy is confining.

If you look here in the contrib area of polices for Fedora/Red Hat. you'll notice similar conventions.

I wouldn't say it's required and it's up to the developer's choice from memory. Although, more and more are dropping it.

[natoscott]

It's common for the policy name to match the name of the daemon though or, failing that, the package name. Since there isn't a grafanad (the grafana daemon binary is named "grafana-server"), it may make more sense in this case to match the package name. This is the strategy redis policy uses, for example, with its redis-server binary.

[sfeifer]

Hi,

I've renamed all references of "grafanad" to "grafana" in addition to changing the names of the files and updating the README. I also added a bit to the grafana.te file while debugging avc logs. I did this in my forked repository and want to merge those changes into the original, this repository. Should I go ahead and open up that pull request?

[georou]

You're welcome to open a PR for changing granafad to grafana.

Looking over the additions, there's some questions as to why those are needed.

Firstly, the reverse proxy boolean, not sure if you enabled it in the policy and forgot it but the accepted way is to have it disabled in the policy and toggle the boolean on a per host basis. Not everyone needs it enabled and the philosophy of least privileges applies. You need to question why something is needed otherwise having a door is pointless if you leave the door unlocked.

Having the interface kernel_rw_unlabeled_files() looks like you've got files somewhere incorrectly labeled. You need to find out which and why, as this is a bit of a show stopper. kernel_signull_unlabeled() and files_delete_isid_type_files() is also related.

The socket file in /tmp might need more permissions. The symlink could also use the read_lnk_files_pattern macro. In general, using macros and interfaces cleans up the policy and you don't have an ugly require{}. Look at contrib examples how this is achieved.

As a good guide, if audit2allow suggests something and you're not entirely sure if you should use it. Go to the contrib page I linked above and search the other policies if they contain the interface/macro. So for example, if you searched for _kernel_rw_unlabeledfiles, without noticing that the name is already a red flag, you'll see no other policies use it.

If you're having trouble looking at the audit logs and not seeing the full picture, I highly suggest enabling full auditing mode. Dan Walsh has a guide for the steps. Note though, the audit.rules path might be outdated. Should be /etc/audit/rules.d/audit.rules.

You did catch the wild boolean for alertmanager, I'll need to improve that. Thank you.

Having RH Insights in there seems like a use case for your stack. It's difficult for me to say that this should be included as that's a custom requirement. This is generally aimed at being a canvas to build on. Again, I'm not an authority on a policy for grafana - I'm just an enthusiast on Github trying their best to share some knowledge. I'm leaning more to taking PRs that just get the basics of Grafana working with a policy and then leaving it up to the user to customize for themselves, privately.

As for a coding style guide when you're comfortable, check out the Gentoo SELinux style guide. It should help you put items in the proper order.

I fully appreciate the time you're taking if I sound off putting. SELinux is a bit of a beast, especially when audit2allow throws out a best guess and if you're unfamiliar it leaves large security holes and negates even having SELinux on. That's the biggest hurdle I can see for its future. It requires a specialist and sadly for example, if you're a web dev and under time pressure, you'll just disable SELinux and move on with your day. Simple as that.

[georou]

I've installed Grafana on a CentOS Stream 9 box and thrown in the policy and noticed that it starts unconfined (ps auxZ | grep grafana).

Looking at the systemd unit file, it starts a binary in a different location (/usr/share/grafana/bin/grafana). Moving to Grafana's git, it looks like they changed this in a commit from Nov, 2022.

So at the very least, the .fc needs updating with the new location and the new folder path needs investigating (/usr/share/grafana) for appropriate labels. A rough as guts change, I labelled the binary and added two new permissions and so far no denies just to start and login to Grafana.

I'll look into it more over the coming weeks and push an update out since it's broken in its current state.

[sfeifer]

Thanks for taking a look at this! I just opened the PR to rename grafanad to grafana with a new branch I made where the only change is the naming.

I'll also keep working with this to see if I can improve on anything and learn more about SELinux. Once the bug you mentioned above is fixed, could you tag a release for it?

Thanks again for the feedback.

[sfeifer]

I'm looking into working on the plugins part of the policy. Any suggestions on where I should look for examples and general info on that?

[georou]

For plugins example wise - you might need to look at packages in contrib again. From memory, something like nagios had a similar idea in it.

[georou]

But looking over the policy again with 2023 eyes. I'm not sure if it even needs to confine "plugins". I didn't get as far as using any apart from what is included so I'm only going on what I can see in the Grafana docs and my fresh install dashboard - plugins are mostly just data sources or things like graph modifiers?

If that's the case, might not even be needed. From memory I originally put it in just in case - but obviously never expanded on it. If there's something you've seen like a plugin that is a binary and needs various system level access - that would be a use case.

I've pushed the latest changes. That's hopefully working properly now and confined.

[sfeifer]

Thanks for pushing those changes.

I'm trying to get the policy working with plugins that are not preinstalled with Grafana. With the SELinux policy enabled, no plugins besides the preinstalled ones can be installed/used. My specific use case is the Performance Co-Pilot plugin. Any thoughts on how I should go about getting additional plugins working?

I've been looking at the naigos policy and trying to adapt it for Grafana to get those additional plugins working but no luck yet. Will keep looking at this but thought I'd ask if you had any initial ideas.

[georou]

You're welcome.

Looking at Performance Co-Pilot, it looks similar to something like Prometheus/Telegraf/Zabbix - collecting host metrics? In that case, it would be better to then use the plugin context.

So, first steps would be to define the plugin type, build any interfaces (.if) and a possible shared domain for Grafana+Plugins. Similar to Nagios. For rules, it would again be similar in logic to Nagios and/or my Prometheus policy(in my github).

Looking at the install instructions, did you just use the steps to install it? ie: grafana-cli? I can try and take a look at it but currently my plate is full so it'll be a slow road. If you can give me the steps to quickly replicate a working example that would help a lot.

[sfeifer]

Installation to replicate the issue (I'm not sure if the first 3 steps to be done to replicate the issue, but without it installed I foresee more issues will arise while trying to solve the initial issue):

1. Install required packages:

   yum -y install pcp

2. Allow incoming connections on TCP/44321:

   firewall-cmd --permanent --zone=public --add-port=44321/tcp

   firewall-cmd --reload

3. Restart & activate PMCD:

   systemctl restart pmcd

   systemctl enable pmcd

4. Install required packages:

   yum -y install redis grafana grafana-pcp

5. Now we can start the components, and enable them for start after reboots:

   systemctl restart redis pmproxy grafana-server

   systemctl enable redis pmproxy grafana-server

Steps to reproduce issue:

1. Navigate to localhost:3000 in a browser
2. Sign in (admin as username and password works as default)
3. Hover over the gear icon in the bottom left and click "plugins"
4. On the plugins page, in the search bar search for "Performance Co-Pilot" and click on it
5. Click install on the Performance Co-Pilot page (This step causes an error and the installation fails)

Let me know if you get different behavior when trying to do this. I'm still quite new to this SELinux policy stuff.

Thanks for helping -Sam

[sfeifer]

Got some weird formatting because I copy and pasted some lines that started with "#". Disregard the difference in text size and treat all the text as if it were the same size.

[sfeifer]

Forgot to answer your question. Perfromance Co-Pilot is like Prometheus in that it collects host metrics. It can also be used to collect metrics from other systems into a host system that monitors those other systems.

[georou]

Thank you, those detailed steps should help me out. I'll try to get a look at it over the coming week(s).

That's good to hear it's like Prometheus. Should be similar in scope to my Prometheus policy.

[sfeifer]

I've been working on the plugin part both to learn more about SELinux maybe make some progress. While testing, I'm having an issue with the restorecon command in the README.

When I run that restorecon command, I get the following message: restorecon: lstat(/usr/share/grafana/bin) failed: No such file or directory

I looked in the /usr/share/grafana directory and notice 3 directories; conf, plugins-bundled, and public. Is one of these the intended directory for that command?

I'm running fedora 38 on my machine. I doubt that is what is causing the error but thought I should mention it. The rest of the directories work fine and do not cause an error.

[natoscott]

@sfeifer @georou PCP ships its own selinux policy, so I think the policy we'll need here is to help on the Grafana side, to allow the grafana-pcp plugin to connect and talk to pmproxy(1) which provides the PCP REST API. It's listening for HTTP traffic on port 44322.

[georou]

Curiously, grafana-pcp changes the /usr/share/grafana directory and the binary goes back to /usr/sbin/grafana-server. Also curious, both pcp and redis have policies.

So, looks like the point of focus is the file/usr/share/performancecopilot-pcp-app/datasources/redis/pcp_redis_datasource_linux_amd64. As you've said, Natoscott and the docs read that it is an interface between pmproxy and Grafana.

The question is, if Grafana should transition executing that file or not - and if to transition, to what domain? grafana_t/grafana_plugin_t or pcp_pmproxy_t for example.

As a quick and dirty test: setenforce 0

Add to grafana.te: allow grafana_t grafana_exec_t:file execute_no_trans; allow grafana_t self:unix_stream_socket connectto;

and then chcon -t grafana_exec_t /usr/share/performancecopilot-pcp-app/datasources/redis/pcp_redis_datasource_linux_amd64

systemctl restart grafana-server.service

If you continue to use/setup pcp in Grafana, get as far as you can using the it. Then maybe throw a gist of the audit log you get from when you restarted Grafana: ausearch -m avc,user_avc,selinux_err -ts 21:00

P.S Is the PCP plugin suppose to show up as unsigned in the Grafana plugins page?

[natoscott]

P.S Is the PCP plugin suppose to show up as unsigned in the Grafana plugins page?

From memory, because we rebuild both Grafana and grafana-pcp in Fedora/RHEL the plugin is no longer signed as it would be when downloaded from the grafana.com plugins catalog.

[andreasgerstmayr]

P.S Is the PCP plugin suppose to show up as unsigned in the Grafana plugins page?

From memory, because we rebuild both Grafana and grafana-pcp in Fedora/RHEL the plugin is no longer signed as it would be when downloaded from the grafana.com plugins catalog.

Yep, exactly.

This is the workflow to get a Grafana Plugin signed:

• compute the hashes of all files in the plugin zip file
• send them to a grafana.com API with a private API Key
• the grafana.com API returns a cryptographically signed textfile of all hashes
• include this textfile (MANIFEST.txt) in the plugin zip file

You can see this file in the community version: https://github.com/performancecopilot/grafana-pcp/releases/tag/v5.1.1 More info on plugin signing: https://grafana.com/docs/grafana/latest/developers/plugins/publish-a-plugin/sign-a-plugin/

Unfortunately we cannot call the grafana.com API during the Fedora/RHEL build process, as we would need internet connectivity and also store a private API key somewhere, therefore the Fedora/RHEL builds of the Grafana plugin are unsigned.

[sfeifer]

Hi.

I'm having an AVC denial that I'm confused on. The pcp plugin currently works, even with the avc denial, so I think it could be related to something else.

The denial is as follows: type=AVC msg=audit(1694718950.565:5766): avc: denied { unlink } for pid=1 comm="systemd" name="plugin325928490" dev="tmpfs" ino=13146 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:grafana_tmp_t:s0 tclass=sock_file permissive=0

From what I can tell, this looks to be related to the newly added grafana_tmp_t type. As I currently understand it, adding a rule like "allow init_t grafana_tmp_t:sock_file unlink;" would not be valid since the init_t type is not defined in the .te file.

One possible solution would be to relabel what's being labeled as grafana_tmp_t to something that is already accessible by init_t (e.g. user_tmp_t, init_tmp_t, or tmp_t), but I'm not sure if we want to do this. I also do not see where files are being labeled as grafana_tmp_t....so do not know where I would relabel them if I were to go this route.

Is there a better solution to this AVC denial that I have not yet found? If not, how would I go about relabeling that sock_file to something init_t can already unlink?

Thanks -Sam

p.s. I'll keep looking into this denial, but wanted to ask someone with more SELinux experience.

[natoscott]

| ... the init_t type is not defined in the .te file

I think you can do something like this @sfeifer ...?

require {
        type init_t;
}
allow grafana_tmp_t init_t:sock_file { unlink };

[georou]

Hi Sam,

That should be a cause from the Grafana service file using the systemd sandboxing feature; PrivateTmp.

Throw into the .te file:

optional_policy(`
    systemd_private_tmp(grafana_tmp_t)
')

That should help you along. I'll actually add it to the policy here since it should be there if using PrivateTmp. Nice catch.

If you're having trouble finding the /tmp files, because of the sandboxing, they're hiding in: /tmp/systemd-private-4c2092420eaa45f28e38fa0c004bfbc8-grafana-server.service-q3bydy/tmp *Random numbers disregarded.

[sfeifer]

Hi.

Thanks for that, looks to have done the trick. I'm now having an AVC denial that appears when I remove the policy, rather than when I have the policy enabled.

After removing the policy and running the same restorecon commands I ran after installing the policy, I got an AVC denial related to something with trawcon="system_u:object_r:grafana_var_run_t:s0". This makes me think the label of some file is not being correctly returned to what it was prior to the policy being loaded. I should note I do not see any different functionality with grafana in my browser when that AVC denial is present.

The denial is: type=AVC msg=audit(1695325130.671:10422): avc: denied { unlink } for pid=1 comm="systemd" name="grafana-server.pid" dev="tmpfs" ino=17047 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=0 trawcon="system_u:object_r:grafana_var_run_t:s0"

I noticed /var/run/grafana-server.pid gets labeled as grafana_var_run_t in grafana.fc, so thought this might be a good starting point. I'm not sure when this label gets applied, though, as I do not run restorecon on that path after installing the policy; however, it seems the label is still being applied. I guess this is good as we want the label to be applied.

When the policy is removed, though, it does not look like the label for this file is returned to the correct value. Additionally, while trying to do restorecon on that file, I found the file's path is /var/run/grafana/grafana-server.pid. What is the reason for this path not matching the path in grafana.fc?

Do you have insight about what could be causing the AVC denial and how I could fix it?

Thanks -Sam

[georou]

Hey Sam, no worries, glad that helped. As you can tell, things change in the package or in the kernel/SELinux and it's a constant battle to maintain these little issues. Worse still if you don't use the software and don't track the changes.

Looking at the grafana .pid file, looks like it might have that file context because it was using the init startup file where it has PID_FILE=/var/run/$NAME.pid. That might have been a mix up or it was like that before it moved to a systemd unit file. I can't recall. But now that we're using the systemd unit file, the run time directory is listed under a grafana directory. See unit file.

Changing that .pid line in the .fc file to: /var/run/grafana(/.*)? gen_context(system_u:object_r:grafana_var_run_t,s0) should fix your issue. Again, I'll add it to my commit along with the privatetmp above.

For when it get's applied and how, see L:101. There's a transition rule there that if the domain creates a file with the label var_run_t, to then label it grafana_van_run_t instead. That's why even though the file context in the .fc is wrong, it still labels correctly when the process (grafana) runs and does its thing.

[sfeifer]

That explanation is very helpful, thanks!

Through more experimentation, I've found that the AVC denial does not occur when I run systemctl stop grafana-server before removing the selinux policy. Therefore, I do not think the AVC denial is related to the actual policy after all, but rather the order of removing the selinux policy and restarting grafana.

Initially, I had been removing the selinux policy with grafana-server still running and then running systemctl restart grafana-server after the policy was removed. This caused the AVC denial from my message above, and additional AVC denials I found today.

To avoid those AVC denials, I now first systemctl stop grafana-server. I then delete the port, remove the policy, and restore the contexts of the files. Once that is done, running systemctl restart grafana-server does not produce any AVC denials.

Does this difference in behavior based on the order of removing the policy and restarting grafana-server make sense?

Thanks -Sam

[georou]

No worries, Sam.

That's the correct work flow. Removing a policy on a running process as you've found, doesn't work well. It's also correct in reverse when working on a policy - make policy changes and then restart the service. Just good habit.

Cheers.

[sfeifer]

Hi.

Could we transition the naming of the default branch from "master" to "main". This change is being gradually rolled out across Github. Some more information on it is here, https://github.com/github/renaming.

The change is motivated by the word "master" being offensive to some people due to its history. I think this change would be a good idea as it helps promote inclusion in open source.

Thanks -Sam

[georou]

Hey Sam,

Sounds good, I'll queue it up. Got a few oldish repos I'll need to tackle too, in that case.

Change is now live. Some handy steps from the github page on how to update your existing branch:

git branch -m master main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a

Cheers!