Parse a Database connection as a rule source

[kleineroscar]

It would be nice if we can parse a database to use as a source for the rules instead of xml/zip files. This streamlines to process when working with the web maintenance tool.

I already have a version that technically works but lacks some features. For example it does not parse the Mapping collection for actions. Also the new methods i'm calling are kinda hacky. There is also no way to use this constructor from the command line atm.

I also have a version for the JaRe server, but this one is even more hacky.

I think we also need some way to mark a project as ready to be used in the database. A flag in the project table would be the easiest I think. Then we can simply check this flag when parsing the database and retuirn no rules if it is not marked as ready. Soem changes to the web maintenance tool would also be needed though.

[kleineroscar]

You can find my branch here kleineroscar/jare/tree/parse_db_as_rule_source

[uwegeercken]

thanks for your comments. give me some time to understand all of it.

[kleineroscar]

Hey great to hear from you! In the current version there are still some minor mistakes. I'll clean up the code a bit and push a new version to the branch next week

[uwegeercken]

cool. Would you mind to also:

• provide an example
• update the documentation

I haven't looked yet at the code: will this be generic for different database types?

[uwegeercken]

in regards to the project flag: are you thinking of a sort of "is complete" indicator? Or one could also see it as a type of approval: one person adds the rules and another checks them and gives the ok. I've been thinking about this... I also though about adding functionallity to push the project, groups, rules, etc to git from the web app. This way e.g. a Jenkins job could pick it up via a hook and do something. I am mentioning this, because at the company I work we use the ruleengine in a microservice in a docker container. Jenkins reacts on git and creates new docker images and then disploys them.

[kleineroscar]

Yeah that's exactly what I hoped for. We're currently in the process of deploying a Pentaho Workflow which integrates the rule engine into a larger automation. This way we don't have to do any file handling.

[uwegeercken]

ah, cool. You know I have created a PDI plugin for the ruleengine? For working with files or also with the server. If you choose the server version, you have the rules on the server side and outside pdi.

[kleineroscar]

Yes we're also using your plugin! I altered it to also be able to parse a database connection. I'll also create a fork and push my changes, but I'm currently still investigating a problem where it didn't parse everything correctly.

[uwegeercken]

I looked at the code. Thanks for all the work. When I went through the code changes for the xml parser I thought about the following:

• the parser class is really for parsing (xml) and the db connection would be for collecting from a database. So I would rather create a separate class for it and naming it accordingly.
• your code has a lot of hard coded database column names. This will make maintenance difficult. I think it would be best to create a library which contains the details. This way one could use it for this project but also for the web application and thus have a single place for the maintenance in case something in the database changes.
• rulegroups have a valid from/to date. so the code would need to take care that rule group which are not valid anymore don't get imported. But those which are valid in the future have to be imported (maybe - I have to think about this).

I will further look into this in the next two weeks.

[kleineroscar]

Thanks I'll answer some of your points below:

will this be generic for different database types?

I thought about this again and I think it's enough for us to support a mysql and mariadb database as the maintenance tool is currently the only way to insert the rules into a database.

the parser class is really for parsing (xml) and the db connection would be for collecting from a database. So I would rather create a separate class for it and naming it accordingly.

Yeah that's a valid point. I'll refactor it into it's own class.

your code has a lot of hard coded database column names. This will make maintenance difficult. I think it would be best to create a library which contains the details. This way one could use it for this project but also for the web application and thus have a single place for the maintenance in case something in the database changes.

What do you mean by library? Maybe we could solve it with a maven parent project from which both descend. I'm currently developing the maintenance tool with a maven environment. So It should be possible.

rulegroups have a valid from/to date. so the code would need to take care that rule group which are not valid anymore don't get imported. But those which are valid in the future have to be imported (maybe - I have to think about this).

Isn't this already handled by the com.datamelt.rules.core.RuleGroup#isValid() method?

[uwegeercken]

ref "library". yes a maven parent project for both would be good. The column names should be available as constants in one place so maintenance is easier and in one place.

ref "valid from/to". yes just wanted to mention it. you are obviously very familiar with the code but I wanted to mention it for completeness.

[uwegeercken]

ref "mysql/mariadb". ok agreed. for the moment that is enough.

[kleineroscar]

Sorry for being MIA. I pushed my changed to the branch. Maybe you can take a look at it. I don't really know how mappings are implemented in the rule engine and more importantly saved in the db, so some guidance would be great

[kleineroscar]

Hey, as it turns out I misunderstood mappings. So they don't need to be parsed from a project. Have a look at the latest versions at jare and jareplugin_src . May I also suggest that instead of separate repos for jareplugin and jareplugin_src we just one repo?