Add new queries to ClientSerieModule for accessing tag and field keys/values for a measurement

[meverett]

At present it is possible to query measurement tag keys, tag values, and field keys by just writing the following raw queries and executing them:

SHOW TAG KEYS FROM "measurement"

SHOW TAG VALUES FROM "measurement" WITH KEY = "tagKey"

SHOW FIELD KEYS FROM "measurement"

However it would be a lot more convenient if this interaction was promoted to native methods on the SerieClientModule:

IEnumerable<string> GetTagKeysAsync(string dbName, string measurementName);

IEnumerable<TagValue> GetTagValuesAsync(string dbName, string measurementName, string tagName);

IEnumerable<FieldKey> GetFieldKeysAsync(string dbName, string measurementName);

These methods are really useful for things such as populating values in a drop down in UI-driven applications: query tag keys -> bind to drop down -> value changed -> query tag values -> bind to list view. Otherwise this requires a bit of hard-coded response parsing every place you would want to explore these values/the schema of a measurement.

I have a feature branch in my fork where I've implemented these methods and the two new needed models TagValue and FieldKey. Let me know if you'd like a pull request. My changes around fixed Diagnostics parsing and Uptime parsing are included in this branch as well.

I would also like to add a UserClientModule and implement Show Users, Create User, Create Admin User, and Drop User. With these methods and the ones I've added above InfluxData.Net would implement more queries than the deprecated InfluxDB Web Admin Interface has in its Query Templates feature. These queries seem to be the last missing ones for completeness. Thoughts?

[meverett]

Besides these new Serie module features, I did end up implementing a User module and all relevant user API requests that I could find. I updated the documentation in the README to reflect all of these additions. Let me know if you'd like a PR for it: https://github.com/meverett/InfluxData.Net/tree/feature/UsersModule

[tihomir-kit]

Hi man,

first of all - it makes me really happy when I see someone else is willing to contribute to the lib with code, so thanks! In general - you don't have to ask if I'm willing to accept a PR, of course I am. :D A bit of planning and talking about how best to implement something like this is fine, but yes, I will happily accept PR's.

That being said - sure, make a PR and I'll merge it in.

The tags and fields, it does sound like a good idea, and I also think it would be pretty convenient. Regarding the user module, I was thinking of adding that sometime in the future, but I never did because I didn't have the need for it, so yes, that sounds great as well.

I played last night a bit with implementing batched writes, I'll wrap that up today and then resolve PR's.

Great work, thanks! :)

[meverett]

I hear you on the planning front. In my case I'm working on a GUI application that basically breaks out all of InfluxData.Net functionality for managing and interacting with InfluxDB servers: something like robomongo, but for InfluxDB. I wanted to add user management for completeness and I had time to push on the project this week so I just sort of went for it and implemented it to my needs. It was an understandably non-coordinated solo effort, which is why I wasn't sure you'd be interested in a PR (or at least felt like I should ask), but that said I feel like my implementation is pretty solid.

My basic overview of the work is this:

I added 4 new response models and 1 new enumeration:

User

This model is pretty straight forward. Just a string username would almost be enough but SHOW USERS returns 2 pieces of data, the user name and the admin flag, so I just tried to match that.

public class User
{
    public string Name { get; set; }
    public bool IsAdmin { get; set; }
}

Privileges

This is an enumeration that captures the supported InfluxDB privileges (read, write, all) plus a "none" that allows to initialize a privilege value in "unknown" state instead of initializing to a assumed value.

public enum Privileges
 {
    None = 0,
    Read = 1,
    Write = 2,
    All = 4,
}

Grant

This is a model that represents the returned response when calling SHOW GRANTS FOR "username". It uses the Privileges enumeration to hold the actual privilege value.

public class Grant
{
    public string Database { get; set; }
    public Privileges Privilege { get; set; }
}

TagValue

This model stores the results of the query SHOW TAG VALUES ON "database" WITH KEY = "tagName". Name is the tag name, and Value is the tag value.

public class TagValue
{
    public string Name { get; set; }
    public string Value { get; set; }
}

FieldKey

This model stores the results of the query SHOW FIELD KEYS ON "database" FROM 'measurement'. Name is the field name, and Type is the data type of the field (integer, string, etc.).

public class FieldKey
{
    public string Name { get; set; }
    public string Type { get; set; }
}

Final Note

A final note worth mentioning, and possibly where your input/insight is most important: I received a lot of "deprecated" warnings when running some of the queries I added (user queries especially). Basically the warnings from InfluxDB were that using HTTP GET is deprecated and that the queries should use POST.

Although the underlying classes let the HTTP method be specified before making the underlying HttpClient request, the methods I wanted to use from the UserClientModule (like GetAndValidateQueryAsync) always assumed GET and hence the warnings in the server response.

So in IInfluxDbRequestClient/InfluxDbRequestClient/ClientModuleBase I added some overrides to QueryAsync/GetAndValidateQueryAsync so that you can alternatively specify the HTTP method you'd like to use and pass that down to RequestClientBase. This allowed me to keep all the existing code as it was, but also to call the overrides from the client module methods I was using that were receiving the warning. Doing so removed the warnings and all of the requests worked with POST testing against InfluxDB 1.0.2.

The only remaining unknown since I don't have a ton of different versions of InfluxDB setup to test against, is that going backwards in time, perhaps some versions only support GET and not POST in which case my UserClientModule methods would have to be updated to use the right HTTP method based on version.

So that's the work in a nutshell with all the major considerations I could think of. I haven't added any tests yet. I noticed when I ran the tests locally that they expected various InfluxDB servers set up on the local network with specific IPs and I didn't have the time to invest in replicating the testing setup locally. But if you can point me to some tests that don't require a server or could be mocked, I could always try to take it a bit further.

Let me know what you think!

[tihomir-kit]

That sounds interesting. Are you making it as a service or as some kind of open-source app web or desktop? It crossed my mind to perhaps try and make something like this because the InfluxDb admin webapp seems to be left in a bit of a limbo. :D But I never got around to actually doing it.

Anyway, I'll now check your latest PR and merge it in.

I am also aware of the deprecated warnings (see #13 and #14, you might already have seen it anyway). My plan was to include these warnings as a part of IInfluxApiResponse objects. This would then be easily testable. I could just attach and additional check to each integration test which says - if Influx API returns a warning - fail the test. And the plan was to then go and update the influx/kapacitor templates that are used by the library.

I have v0.9.6 in a VM and I can test against that. I think I'll have to override some of what you made to make it work with that again but ok. I am pretty confident that pre-v1 only supported GET's for everything and I think I remember reading somewhere on their github that other people complained about it as well.

Regarding your tests - I dunno, do you have a local VM with the latest influx version perhaps? If so, you can just change the app.config in the InfluxData.Net.Tests project and run the tests for your version. It should work. For testing purposes, I installed influx and kapacitor inside 2 separate ubuntu VM's and I set the username and password both for everything to "root" to make my life easier. If you don't feel like setting this up, I can understand that :D although it's not really a lot of work with virtualbox. If you are - great.

[meverett]

The GUI app is a Windows.Forms desktop application. All the InfluxDB stuff I'm doing presently is related to .NET. I'm using InfluxData.Net in the cloud to relay some ingested IoT data into InfluxDB and that's how I first stumbled upon the project. Since I'm going to be administering InfluxDB now I was sort of shocked there wasn't an existing tool (for example I mentioned Robomongo which I use for MongoDB work) so I decided to make one.

I do plan on releasing it as open source and I had picked WinForm hoping somehow Mono support might allow the application to be truly cross platform. But my initial tests (in Mac OS X at least) are a little disappointing. I'm not sure how realistic it is to have Mono compatibility without a ton of work.

As soon as all of this new InfluxData.Net work is released and the nuget package is updated I will go back to referencing InfluxData.Net as a nuget package in the GUI app and not a custom project reference to a custom branch with my additions to it :D - and then I'll stick it up on github for anyone to grab.

I'm really close. I'm part way through adding Continuous Query editing workflows and then I just need to do Retention Policy workflows (and perhaps a little snazzy UI for stats). After that the first version should be feature complete with every one of the InfluxData.Net API methods exposed in UI workflows and controls (not Kapacitor though), including all the new work I just added (user management, etc.).

I'll post a link when it's up if you're interested.

[tihomir-kit]

Oh yeah, for sure - let me know. We can also add it to this README.md file once it's ready. Other people might discover it sooner then.

[tihomir-kit]

I merged all this stuff in, did a small bit of refactoring with the HTTP GET/POST stuff, but otherwise I left it as it was. I did make some other breaking changes though. This was needed to fix a few bugs and to improve older version support. So I'll be releasing this under a new major version. I'll probably play with it for a few more days, make a few more improvements over at the Kapacitor project, write a few more tests and then make the new nuget. Thanks for the good work. :)

[meverett]

Glad it worked out! I'm looking forward to the next release :) Also the work on Batch Writing looks great.

[meverett]

Also FWIW I noticed in the new documentation/README.md that the code block/sample/c# for the AddPoint() method of the batch writer is incorrect and is a duplicate copy and paste from batch.stop() a few lines above it.

[tihomir-kit]

Ah, thought I fixed that. Will fix, thnx.

[tihomir-kit]

Just to let you know, the latest release is up on nuget.org. I tested it locally and it all seems fine, all tests are passing. I didn't write any tests for the user module, don't have enough free time to burn on this now, I'll do it over christmas holidays probably. If you notice anything strange, please let me know. Cheers and thanks!

[meverett]

I'm still putting the documentation together and I still have a few features to add, but the GUI tool is up if you want to check it out: https://github.com/CymaticLabs/InfluxDBStudio

[tihomir-kit]

Well that looks pretty good! I'll give it a go over the weekend. And I'll add a link to the readme, I bet other people will be interested as well. Nice one.