ldms streams fails to implement authorization

morrone commented 3 days ago

Could someone please explain how ldms streams authorization works? I'm having trouble finding anything about it in documentation.

morrone commented 1 day ago

Here is what I understand:

We need to keep in mind that authentication only tells us who a user is. It does not tell us what they are authorized to do.

Any user that can authenticate with an ldmsd has free access to publish and/or subscribe to anything in streams that they wish. So in common deployments where munge is used to authenticate users on a cluster and in ldmsd, all cluster users will have free reign to publish and access any streams they wish. A user's stream data will not be isolated from other users.

Many ldmsd running in deployments are running as root, because samplers require root access to gather their data.

Right now those same root ldmsd can have streams data published to them from any authenticating user (typically any user on the system). This is poor practice, because we should limit to our greatest possible extent the amount of data that can be injected into the daemon by users. The more we allow rusers to push data into the root daemon, the more likely we are to have the ldmsd exploited in the future. Best practice in most cases would be to avoid enabling streams when the ldmsd is running as a privileged user. Separate non-privileged ldmsd should be used for handling user data streams.

Next, there is no isolation of data between the users using the streams service. Anyone on the cluster can access anything. We need to protect a user's data, and only expose it to the other users explicitly permitted to do so.

So in the longer term, we need to design and implement a proper authorization system for ldms streams.

But in the near term we must:

Add an ldmsd configuration option to enable/disable ldms streams.
Make the default disabled.
Clearly document the implications of enabling streams. Something like "WARNING: All authenticated users are permitted to publish anything to any stream, and may also subscribe to any stream. There is no data protection after authentication. In most cases, it is also best practice to avoid enabling streams when the ldmsd runs as a privileged user."

tom95858 commented 1 day ago

@morrone, all of these issues are resolved in 4.5.1 (top of tree). Just an FYI. And I think you statements above are a bit too strong, but let's talk about it and decide what to do with 4.4.5. I like the idea of the stream disable command.

morrone commented 23 hours ago

If I misunderstand something about the implementation I am happy to make corrections. But if my understanding is correct, I think it is important that we are clear and direct about the security implications.

Can you give pointer to where the issues are resolved on main?

tom95858 commented 23 hours ago

There are uid/gid/perm information in the stream message headers. Please look at the ldms stream implementation as opposed to the ldmsd stream implementation.

On Thu, Nov 21, 2024, 6:39 PM Christopher J. Morrone < @.***> wrote:

If I misunderstand something about the implementation I am happy to make corrections. But if my understanding is correct, I think it is important that we are clear and direct about the security implications.

Can you give pointer to where the issues are resolved on main?

— Reply to this email directly, view it on GitHub https://github.com/ovis-hpc/ldms/issues/1527#issuecomment-2492695867, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABVTPXHYWK3DXZ6KGZBOEOT2B2DM3AVCNFSM6AAAAABSDCIV5CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJSGY4TKOBWG4 . You are receiving this because you commented.Message ID: @.***>

morrone commented 19 hours ago

Ok, it looks like only main has ldms_stream.[ch], while both have ldmsd_stream.[ch]. On first skim of the code, it looks main broke out some of the ldmsd_stream.[ch] content into ldms_stream.[ch] and then expanded the functioality.

There are virtually no code comments, and I haven't found much in the way of other documentation yet. It looks like ldms_stream.h is missing function declarations that should probably be there too. Tomorrow I'll see if I can figure out what what is going on.

tom95858 commented 6 hours ago

Ok, it looks like only main has ldms_stream.[ch], while both have ldmsd_stream.[ch]. On first skim of the code, it looks main broke out some of the ldmsd_stream.[ch] content into ldms_stream.[ch] and then expanded the functioality.

It broke out all of the capability and added more. The only reason that ldmsd_stream still exist is to make the configuration compatible with 4.4.5. If we don't want that, we can completely remove ldmsd_xxxx version of streams. And I'm beginning to think that is a good idea

There are virtually no code comments,

They are written in C. ;- Point taken.

... and I haven't found much in the way of other documentation yet.

I think we should review the documentation for both main and b4.4.

... It looks like ldms_stream.h is missing function declarations that should probably be there too. Tomorrow I'll see if I can figure out what what is going on.

@narategithub could you take a look at this?

narategithub commented 6 hours ago

ldmsd_strea.[ch] are there so that top-of-tree ldmsd can still process old ldmsd_stream messages from old ldmsd. Going forward, all application should use ldms_stream.

If you are an application, just include ldms.h will provide all functions you need to interact with ldms_stream.

To publish to the peer, after connected, simply call ldms_stream_publish() (see https://github.com/ovis-hpc/ldms/blob/main/ldms/src/core/ldms.h#L1376) to publish the data.

To subscribe for data, there are two steps:

subscribe the stream locally in the process: client = ldms_stream_subsribe("job.*", 1, cb_fn, "job stream") (see https://github.com/ovis-hpc/ldms/blob/main/ldms/src/core/ldms.h#L1474). Whenever a stream data with the stream name matching "job.*" reached our process, the cb_fn() will be called.
ask the remote process to send relevant stream to us: ldms_stream_remote_subscribe(x, "job.*", 1, NULL, NULL, LDMS_UNLIMITED) (see https://github.com/ovis-hpc/ldms/blob/main/ldms/src/core/ldms.h#L1507).

I agree that we should have a man page for this, with examples (both Python and C).

Example of simple stream client w/ Python: https://github.com/ovis-hpc/ldms-test/blob/master/python/ldms_stream_client.py

Simple example of stream publisher:

from ovis_ldms import ldms 
x = ldms.Xprt(name = "sock", auth = "munge")
x.connect("somehost", 411)
x.stream_publish("mystreamname", "some_stream_data")

tom95858 commented 5 hours ago

@narategithub I think @morrone is talking about configuration documentation. Do we have that?

narategithub commented 5 hours ago

We have prdcr_subscribe config command. It has its help string, e.g.

(not connected)> ?prdcr_subscribe

        Subscribe for stream data from all matching producers
        Parameters:
        regex=     A regular expression matching PRODUCER names
        stream=    The stream name or regular expression
        [rx_rate=] The recv rate (bytes/sec) limit for the matching streams. The
                   default is -1 (unlimited).

and another description in ldmsd_controller.man:

   Subscribe to a stream on matching producers
       prdcr_subscribe attr=<value>

              regex PRDCR_REGEX
                     A regular expression matching PRODUCER names

              stream STREAM_NAME_OR_REGEX
                     The stream name or regular expression

              [rx_rate BYTES_PER_SECOND]
                     The recv rate (bytes/sec) limit for the matching streams. The default is -1 (unlimited).

ovis-hpc / ldms

ldms streams fails to implement authorization #1527