Unify configuration file format

[serpi90]

Could you consider a standard file format for the configuration files, it's kind of a mess sometimes.

It could be either XML, JSON, Yaml or any standard format.

If you choose XML, please define schemas (XSD) for validation (it ensures a correct file format, and you can define default values there). And you can use the Xerces library for writing / parsing.

[RElesgoe]

That's a good idea. Right now, the files are in a human friendly format. XML format is more for computers so we should also have gui program that can modify these files

[HarpyWar]

json may be preferred. It's readable both for a human and a computer.

PvPGN is a server side app and I don't think that we need a special GUI tool for that (but it would be nice if someone create a web app to control a server that can include a visual config editor).

[anonprophet]

+1 json

[Kelwing]

I agree with JSON, and suggest jsoncpp for the parsing library.

[cen1]

The more important issue here is that we need to open a port and implement a small http server into pvpgn if you want it to be configurable from the web. It needs to be able to handle a REST API. Configuration should be represented as a class so it can be easily serializable. A quick look at boost serialization library seems to be able to do both: serialize and deserialize. If that actually works you could push whole JSON config from the web and apply it (deserialize it) into pvpgn config object.

This is how java folks do it and these are considered best practices. I never did such a thing in c++ tho and since c++ lacks reflection it should first be investigated. Perhaps write a simple class and try to serialize and deserialize it from/into json.

All config files could also be joint into one since you can do trees with json to easily separate different pvpgn subsystems.

A lot of stuff to think about here.

[Kelwing]

No one here mentioned a web conf interface. We can do JSON through a config file. It's very simple to write an interface using jsoncpp to deserialize JSON. As far as a web interface, it could be implemented quickly and easily using libmicrohttpd, but its entirely not necessary. You can just edit JSON config files using a text editor.

[Kelwing]

jsoncpp is extremely well documented: http://open-source-parsers.github.io/jsoncpp-docs/doxygen/index.html

[cen1]

Do you know if it's possible to serialize and deserialize this way? http://stackoverflow.com/questions/17549906/c-json-serialization The idea is not to do it by hand but it seems not to be possible in c++ due to missing reflection. If we used some kind of a map then I think it would be possible but that is a tradeoff then because you are not using the class itself and rely on string keys. I've been doing too much java with jackson library lately which can do this automatically so I'm curious to see if it would be possible in c++. :)

[HarpyWar]

Now we have a lot of different formats and each configuration file has own file reader/writer wrapper (tournaments, icons, topics, bnmaps and others). Benefits from json will be in put in one place reader/writer of all configuration files. And other parts of code can use this internal config layer to access config items.

By a web app I meant user friendly interface to stop/start server, several config sections. For instance to upload banners and edit their rotate options, add icons, edit normal config values and then serialize it into json files.

Small http web server looks interesting but it may be very hard to implement and it adds a new complexity. I thought about real-time control features and it may be done with a persistent connected external bot that can have REST API interface to send text commands to a server. A bot can be written on Java by someone. Sometimes real-time changes is preferred instead of direct changes in a database. For instance it can be built-in into a web site with a feature to change a user password (otherwise password may not be changed or with a delay due to pvpgn cache layer if we change it in a database).

[cen1]

User's password is not a configuration matter, that is a separate thing that has to do with database persistence. It can already be changed live using a chat bot and /chpass (that's how we do it in production actually). If something can't be changed live with a bnet command then yes, we need a new protocol but that should be separate from configuration. It can still be a REST API though. A client in the middle seems like unnecessary complication.. if you can receive commands from a local client you might as well receive them from the web.

Changing config live is actually doable, just with some caveats.

1. Frontend sends new JSON config to REST API using http POST
2. Pvpgn deserializes JSON into config object
3. You reload the config using the new object and switch the pointer to it. Added benefit if you keep the old config object in memory so you can simply switch back to it if the new config is bad.

Obviously, not all config can be changed like that on live system. Some attributes would have to be protected from being modified live.

I suggest we write some proof of concept code first before a single line of code is changed just to see how things would actually work. I will hack something together myself.

[RElesgoe]

I'm looking at JsonCPP right now and it seems like it will work out but I'm concerned about generating an amalgamated source since it requires Python. Is it possible to generate an amalgamated source and upload the files to this repository so that everyone doesn't have to generate it themselves?

[Neubivljiv]

It's a good idea to use a universal and readable format configuration. As for me, it is not necessary (because it's mainly configured only once). For web applications, plain text could be easily parsed (even better if it's json). I already did this for ghost bot for web application, where it's possible to edit huge configs in plain text (each line is parsed and created input/dropdown field for HTML output). I also added auto backup for original config so you can always undo configuration. About REST API I would suggest to use sockets, so you can easily connect via php or any other third party application for remote control (we already did this for ghost++ where bot streams live games to user chat web application via sockets). Also, remote control is very easy to buildin. If you ask me, avoid Java for many reasons.

Here is few screenshots of old GamePanel for Ghost bots, and Steam games (we are using this for a years): chat log (can load huge log file very fast because it partially reads file).

This can be easily done for PVPGN, but I already tried and build small interface for start/stop server. Than I added option to user can login/change password/register via web. But it's not working, because PVPGN server won't check every time user login (I need to restart server after each registration or pw change), so currently I gave up.

[cen1]

One big problem with JSON which we did not talk about yet is that you can't have comments or documentation in it. At the moment we have huge amount of helpful documentation and tips about each variable and all this has to go away if we use JSON.

@Neubivljiv Sockets are fine for simple tasks but using a small http server gives you standard protocol support (http with all the headers) and return codes. If you use sockets you either need to reinvent the wheel or you end up with some custom protocol.

[Neubivljiv]

Well, why not using JSON data as a comment?

{ "_info": "This is a comma delimited list of hostnames that the server should listen on. It might be useful to make an internal-only server on a gateway machine for example...", "servaddrs" : ":" }, { "_info" : "W3 Play Game router address. Just put your server address in here or use 0.0.0.0:6200 for server to bind to all interfaces, but make sure you set up w3trans if you do.", "w3routeaddr" : "0.0.0.0:6200" }

[Kelwing]

@xboi209 On Linux, most if not all distributions have it as a binary package, so you can just use the system package instead of including the source files directly in our code. Once you have the system library installed, its as simple as #include <json/json.h>

Windows is another issue, however.

[cen1]

@Neubivljiv I guess. It's not very nice but it would allow us to also push comments to the frontend which we would need anyway.

[RElesgoe]

@Kelwing Yeah we will need to support Windows. Maybe there's a pre-compiled DLL somewhere like zlib

[Kelwing]

I know it supports compilation under Visual Studio, I just haven't done it myself yet.

[RElesgoe]

The readme suggests to create an amalgamated source if you're going to integrate it into a project, which will require Python.

[Kelwing]

Right, be we only need to do that once. I can do that on my laptop if we need. I have Python on it.

[Kelwing]

Here is a gzip'd tarball with the amalgamated source: https://kelnet.org/owncloud/index.php/s/LDngqOoFgurtqBy

[cen1]

After a few hours of research and playing with different libraries I managed to hack together a simple proof of concept code. It can be found in my repo together with VS 2015 project and all libs and binaries (so you don't have to compile anything yourself). Just change include paths (from my absolutes) plus some file paths in code and you are good to go: https://github.com/cen1/pvpgn-conf-poc

The example code shows: -a simple REST API using a http server from Poco library. It is really straightforward, you just initiate HttpServer, give it a request handler and then you can register REST endpoints by checking the URI path that was requested -config object serialization and deserialization using cereal library. You simply define a template class and tell which attributes to serialize and it does it automatically. No parsing by hand! Pure awesomeness if you ask me. :)

Result:

In the following days I'll clean up the code and abstract it a bit, it should be done by the end of it. I'll also include AngularJS frontend example to show how easily we can bind frontend forms with received data and send it back to the server as a single json object.

I am willing to spend a few hours a week to work on the actual implementation but feedback is welcome first.

[RElesgoe]

I've decided to use JSON for Modern C++ for this task.

[xpeh-owns]

If you decide to JSON over XML, use hjson or json5 format since json is not very user friendly to edit.

[RElesgoe]

I believe that standard JSON should be used since it's more widely adopted.

[xpeh-owns]

JSON extension have advantages:

• comments. Who needs config files without comments?
• trailing comma
• some more...

[serpi90]

If you want to use a non standard format, just use the current one as it works. Configuration parameters description should be on a document, a CONFIGURATION.md detailing the configuration files. And/Or using JSON data as comments as said before.

[xpeh-owns]

@serpi90 old conf files worked too...

[serpi90]

@xpeh-owns that was what i meant

[RElesgoe]

Documentation files would be better suited instead of comments since you could format information in a clean and organized way. Standard JSON syntax does lack some features, but if we're looking at JSON extensions, we might as well also consider entirely different formats such as YAML.

[xpeh-owns]

@serpi90 ah ok :)

@RElesgoe don't know yaml, but you can google json5 and hjson - they are backwards compatible and almost as simple as json. And if you introduce new config formats, shouldn't it be an experimental version of pvpng?

[xpeh-owns]

So I think xml/json should be useful for nested configs, i.e. not for flat key-value config like bnetd.conf.

• versioncheck.conf should it be in json format? Probably I can covert it into json and make a script that converts it to conf till json configs would be added. It has very weird format with no syntax highlight.

[xpeh-owns]

@RElesgoe comments are not only for documentation, it's also a possibility to temporarily disable an entry. I can't imagine a config without such a feature.

What's config encoding currently?

[serpi90]

{  
  "configuration":{  
    "some":"strff"
  }
}

you can comment like this:

{  
  "#configuration":{  
    "some":"strff"
  }
}

[xpeh-owns]

@serpi90 but if your config format support comments you have syntax highlihting. Even in notepad++ you can comment out selected text with a shortcut.

[RElesgoe]

@xpeh-owns versioncheck.conf will be converted to JSON soon, but if you'd like to convert it for me, I'd be happy to accept contributions. I would like the format to be like this: https://gist.github.com/RElesgoe/ecbcfb1af96953c8ffeb169ad6d38921

Changes to PvPGN in this repository is often going to be not thoroughly tested, this whole repository is practically experimental.

The encoding for all files should be UTF-8 without BOM.

[xpeh-owns]

Well, do you have stable releases at all? I assume people don't want to change their configs or even db structure because some of developers want to test some feature.

Another feature of json extensions is hex notation for integer.

[RElesgoe]

There aren't regular stable releases since development is irregular, but you can look here: https://github.com/pvpgn/pvpgn-server/releases The code that I'm writing to load JSON configuration files should work well as far as I know. It might be beneficial to write a script that can convert between the old and new formats.

[xpeh-owns]

Another feature of json extensions is hex notation for integer so I could write "hash": 0xb52bad87, and have syntax highlighting instead of "hash": "0xb52bad87",

They must not be regular, they just had to be stable.

[xpeh-owns]

Got some results on converting versioncheck. How to upload?

[RElesgoe]

Use https://gist.github.com