Test adding a user and access control rule to a particular MQTT topic

[bajiat]

From @bajiat on December 16, 2016 11:26

On our EMQ deployment, we need to be able to add new users and publish to various topics. We will also want to restrict topics, so they are only accessible for specific users.

See #124 for suggestion on where to start testing.

User story

As an API Manager I want to proxy my MQTT traffic through the Apinf platform so that I can view analytics, perform rate limiting, etc

As an API Manager I want to be the only authorized user to publish messages to a given topic so that all messages on that topic come only from my real-time API

Copied from original issue: Digipalvelutehdas/APIKA#161

[bajiat]

@frenchbread Can you take this task? If yes, please keep us updated, because we need this to move forward.

[bajiat]

From @frenchbread on December 20, 2016 1:6

@bajiat Self-assigned

[bajiat]

@frenchbread Any updates on this task? Can you keep us updated regularly, e.g. daily? You can add you status update e.g. as comments to the issue. Or post to Apinf channel in RC.

[bajiat]

From @frenchbread on December 21, 2016 10:8

@bajiat I was just about to. Doing a research on EMQTTD since, as noted, it doesn't provide open REST interface to add entities (e.g. users, subscriptions, messages etc.) and everything related to adding stuff to there supposed to be done through user interface, MQTT dashboard. EMQTTD also doesn't have a detailed documentation for developers that allows them to extend the broker internally, only by adding plugins. So I had to dig into the code, and understand what's going on there.

Database

To add users to MQTT, some storage is needed. By default MQTT uses mnesia as a core database to store data in. Since we need to somehow access it, add entities, we need some REST interface which is not provided. The only idea I came up with so far is to code it ourselves. Because there is no other way to add user to emqtt, and existing REST interface is for only for listing data, not adding.

[bajiat]

From @frenchbread on December 21, 2016 10:15

The suggestion is to create a separate plugin, or extend existing one emqttd_plugin_apinf to have rest endpoint that will allow to write/read items similarly to these:

[bajiat]

@frenchbread Remember that this should be an upstream contribution and start with users. You can give your recommendation about storage mechanism. Keep in mind we want to define ACL rules for the users to publish messages to EMQ topics.

[bajiat]

@frenchbread Can you also check the research comments from #124 before writing any code?

[bajiat]

@frenchbread Why does the picture you have added in one of the comments above contain 'Y' (= yes) for POST for /api/users? Isn't POST for adding?

[bajiat]

From @frenchbread on December 21, 2016 21:50

'Y' (= yes) for POST for /api/users? Isn't POST for adding?

@bajiat It is, but still it doesn't allow to add entities. (There is no was to pass params to that POST req). Did not find anything about that in code as well.

[bajiat]

From @frenchbread on December 23, 2016 4:31

Added an endpoint for adding user, tested it with postman and it worked. Working in https://github.com/apinf/emq_dashboard repo (forked). It works in a way, that you have to pass username and password key/value in the headers. Later will add same endpoints for topics, acls etc.

[bajiat]

From @phanimahesh on December 23, 2016 17:57

It looks like you may be reinventing the wheel. If you need a user and acl management system, you can store it outside emqttd and use one of its plugins. Have you considered any of these?

• https://github.com/emqtt/emq-auth-http : Expose HTTP endpoints (REST) in your app that will be used by emqttd to check whether to allow connections and pubsub.
• https://github.com/emqtt/emq-auth-mongo : Ask emqttd to read authentications and acl from mongodb and create a separate api layer in node to manage entries in this mongodb
• https://github.com/emqtt/emq-auth-mysql : Same as above, but with mysql
• https://github.com/emqtt/emq-auth-pgsql : Same as above, but with postgres
• https://github.com/emqtt/emq-auth-redis : Same as above but with redis
• https://github.com/emqtt/emq-auth-ldap : Same as above but with LDAP

PS: Chandra requested me to have a look at this and comment here.

[bajiat]

@frenchbread Will you be able to carry this forward? Can I assign this to you? Please see the comments above from @phanimahesh. Does this need re-estimation based on the comments? We can also re-define the scope with @brylie

[bajiat]

From @frenchbread on January 9, 2017 8:6

@bajiat Yes, I'll continue this one. Estimation is the same.

[bajiat]

@frenchbread Let's have a meeting about this task to check what should be the deliverable.

[bajiat]

From @brylie on January 10, 2017 12:46

@phanimahesh our understanding of the HTTP API is that it is mainly useful for getting data, but does not allow the creation of data.

For example, we would like to use the REST API to create the following:

• users for submitting messages to topics (using whichever authentication method is appropriate)
• topics for publishing and subscribing to messages
• ACL rules to control which users can publish/subscribe to which topics

To this end, we are tracking the following tasks separately:

• https://github.com/apinf/emq_dashboard/issues/1
• https://github.com/apinf/emq_dashboard/issues/2

/cc @bajiat @frenchbread

[bajiat]

From @phanimahesh on January 10, 2017 13:23

The idea is to manage this data out of emqttd. Consider this like any other application data you track, pick an appropriate storage like postgres, redis, mongodb etc, create a new table/collection in an existing or new database, and create an api for getting and creating data in this database. For instance, this can be a new collection alongside other apinf data, with api for creation, and web interface managed by apinf dashboard.

Now, the final piece of the puzzle is to make emqttd use this data. Here we have two options.

1. Create apis to check user's password and permissions, and use emq-auth-http pointed at those apis. Then for every incoming mqtt packet, emqttd will make a request to the api and use the response to decide whether connections and publish/subscribe requests should be allowed. It does not matter how the api server internally manages the data.
2. Alternatively, make emqttd directly talk to underlying datastore, using the appropriate plugin among emq-auth-{mongo,pgsql,mysql,redis} etc.

In other words:

• For every new connection
• emqttd will ask you whether it should be allowed. In option 1, you'll get a request, and have to respond with a yes or no. In option 2, we tell emqttd to go look for the data in a database directly. In both cases, emqttd does not manage the data, it only reads it. You manage that data using a separate web interface and assosciated apis, probably a part of apinf dashboard.
• Emqttd will use the response (or data in the database) to make the decision whether to allow a connection or not.

Similar flow happens for access control rules. Every subscription request and publish request is checked against access control rules.

There is no need to create topics, just like there is no need to create urls on a server. They just exist. Topics in emqttd are created the first time they are encountered, and cleaned up after long periods of disuse. A topic is usable only when there is an access control rule that allows either publishes or subscriptions to atleast some clients. Make sure to check the default or catch-all mode applied when no other matching acl is found, it should ideally be deny.

Hope that clears things up.

[bajiat]

From @frenchbread on January 10, 2017 14:48

Opened an upstream issue, suggestion about extending the list of existing API endpoints.

[bajiat]

From @frenchbread on January 11, 2017 8:49

As of our discussion with @brylie and @bajiat, I added a draft code to https://github.com/apinf/emq_dashboard that allows to get request method from the request headers (e.g. POST, GET, UPDATE, DELETE) and check with what action to proceed with specific request.

[bajiat]

From @brylie on January 11, 2017 9:45

make emqttd directly talk to underlying datastore, using the appropriate plugin among emq-auth-{mongo,pgsql,mysql,redis} etc.

@phanimahesh that sounds close to our goal. I.e. we are considering how to view, create, update, and delete EMQ ACL rules, regardless of underlying datastore, via a REST API.

E.g. we would like to be able to call /api/acl, via HTTP, to do the following:

• get a list of ACL rules
• get a single ACL rule
• create an ACL rule
• update an ACL rule
• delete an ACL rule

For the timebeing, we could agree on a specific datastore, such as EMQ internal, and target that for the initial REST endpoint. In the long run, it would ideally be datastore agnostic, so that an integrator only thinks in terms of REST.

[bajiat]

From @brylie on January 11, 2017 9:51

There is no need to create topics, just like there is no need to create urls on a server. They just exist. Topics in emqttd are created the first time they are encountered, and cleaned up after long periods of disuse. A topic is usable only when there is an access control rule that allows either publishes or subscriptions to atleast some clients. Make sure to check the default or catch-all mode applied when no other matching acl is found, it should ideally be deny.

We will remove the requirement for 'defining' topics from our task.

[bajiat]

From @brylie on January 11, 2017 10:3

If you need a user and acl management system, you can store it outside emqttd and use one of its plugins.

This offers great flexibility as well.

My understanding is that a REST API would still be useful, once the datastore is configured, so that we can do the following via HTTP calls to the /api/users endpoint:

• get a list of users
• get a single user
• create a user
• update a user
• delete a user

I may be misunderstanding our goal, but it seems like we want a common interface (REST) for managing users, agnostic of any chosen datastore.

[bajiat]

From @brylie on January 11, 2017 10:4

/cc @bajiat @ccsr

[bajiat]

From @phanimahesh on January 11, 2017 20:33

@brylie I recommend storing the users and acl rules completely outside emqttd. Working with mnesia without good experience with erlang and mnesia is not a good idea IMO.

If you'd like concrete suggestions, with most choices made for you based on what I know about the team:

1. Create a new project - Probably call it Apinf MQTT user and access control manager
2. Make it a nodejs based one with postgres for the datastore
3. Create a REST API in this project to manage (CRUD) users and ACL rules
4. Create a database view that looks more or less like the example in emq-auth-pgsql's README. You can skip this step if your internal database schemas are directly what emq-auth-pgsql accepts.
5. On the emqttd instance, install emq-auth-pgsql and configure it to talk to a database shared by this and the above mentioned project.

(I suggested postgres instead of mongo because I honestly believe mongo is a bad data store, and trusting auth and acl data to it doesn't feel right. You can substitute mongodb for postgres if you desire, but be warned that mongo's design makes it unreliable for data storage in clustered mode, and postgres is considered objectively superior by many.)

If you want to communicate strictly over http without sharing the database directly, implement two additional http endpoints as required by emq-auth-http. See its README for more details.

We may be slightly talking past each other. Allow me to clarify.

The authentication and access control data should be owned by a entity separate from emqttd.

My understanding is that a REST API would still be useful, once the datastore is configured, so that we can do the following via HTTP calls to the /api/users endpoint:

The REST API is provided by this new project. It can be made a part of an existing one, if you prefer.

I may be misunderstanding our goal, but it seems like we want a common interface (REST) for managing users, agnostic of any chosen datastore.

My understanding is that a REST API is needed so that a nice UI can be built around it, and maybe you may like to expose this API. The API is "technically" datastore agnostic because communicating with the datastore is handled fully by this new project, and any internal details about the datastore are not exposed in the API.

Postgres can be swapped with mysql for instance with just a configuration change in case you use a ORM. (Assuming you use emq-auth-http, and retain same routes as before. Or else, you'll also need to switch emq-auth-pgsql with emq-auth-mysql) If an ORM is not being used, depending on how it is written, I expect it to be a week's effort or less to switch.

For the timebeing, we could agree on a specific datastore, such as EMQ internal, and target that for the initial REST endpoint.

As noted above, I suggest a completely external datastore unless you have strong reasons otherwise. It will be much easier in the long run that way.

In the long run, it would ideally be datastore agnostic, so that an integrator only thinks in terms of REST.

I may be missing something, what is this Integrator? Is this the part that links apinf dashboard and this emqttd auth/acl service? In that case, it is already handled by my above suggestion. Please correct me if I'm not understanding this correctly.

[bajiat]

From @phanimahesh on January 11, 2017 20:50

Regarding the upstream issue adding support for user management to emqttd dashboard itself, it is planned to work with emqttd auth username module, which is not ideal for this usecase. Having an external datastore offers greater flexibility.

[bajiat]

From @brylie on January 11, 2017 23:30

what is this Integrator?

Basically, we are the integrators - developers integrating the systems.

Creating a stand-alone project sounds like a viable idea.

I would like to wait for a bit to see how the EMQ management API progresses, before committing to a new integration project.

/cc @bajiat

[bajiat]

From @ccsr on January 12, 2017 6:26

@phanimahesh What is the timeline for the user management API to be available?

I would like to wait for a bit to see how the EMQ management API progresses, before committing to a new integration project.

@brylie do you mean we wait for this feature to be available for this phase? I dont recommend this has been in the most requested feature.

[bajiat]

From @ccsr on January 12, 2017 6:27

Sorry closed this issues by accident. I reopned it again

[bajiat]

From @brylie on January 12, 2017 8:59

@ccsr a day ago, @emqplus said:

"We are working on the the management/monitor API these days."

https://github.com/emqtt/emq-dashboard/issues/90#issuecomment-271820809

My recommendation is to clarify the progress and scope of the EMQ Management API.

We can see if the EMQ Management API might meet our needs before committing to build and maintain a stand-alone application to integrate Apinf with EMQ.

/cc @frenchbread @bajiat

[brylie]

@frenchbread has a work in progress task #2005

[frenchbread]

2005 updated. closing this one