search

oracle-samples / clara-rules

Forward-chaining rules in Clojure(Script)
http://www.clara-rules.org
Apache License 2.0
1.2k stars 115 forks source link

Clara rules throws a Null Pointer Exception even though there is no Null values #437

Open stLalo opened 5 years ago

stLalo commented 5 years ago

Description

I am building a ETL Software and I use clara rules to clean data by applying certain constraints (rules) to shape the data as I want. Rules always work well, until I make my ETL Software a multi-threading parallel synchronous processor. After this, I started running stress testing on the software and I was able to caught an exception that is weird because Clara is screaming at me for trying to divide a null value.

Steps to reproduce

EthanEChristian commented 5 years ago

@stLalo, Its not recommended to do state-based manipulation in the LHS or RHS of rules. Clara makes no guarantees on the number of times the LHS/RHS is executed, only that the steady state is factually correct, meaning that due to truth maintenance the RHS might execute many times. Additionally, the way that clara manages logical retractions would likely lead to less than ideal scenarios when using stateful actions in the RHS.

As to why the null pointer is thrown, do you have the state of the atom when the RHS is fired? From the error it looks like that either:

  1. evnts doesn't contain the event id in question
  2. the event within evnts doesn't contain a : volume or :rate value.
stLalo commented 5 years ago

@EthanEChristian During the weekend, I explore more my situation and I figured out the problem. You said,

  1. evnts doesn't contain the event id in question And this is correct.

As the ETL Software is a Parallel Synchronous multi-thread processor, the state was lost somewhere in between threads. I think my proper solution could be the following steps:

  1. For every rule, insert the Event into Clara rules in-memory database
  2. Query for the events that fitted the Clara rules
  3. Apply corresponding changes for each Clara rule type
  4. Repeat step 1 to 4 until there is no more Records to apply the rules too.

Please bare with me as I am refactoring legacy code and it is the first time encountering this problem.

EthanEChristian commented 5 years ago

@stLalo, I agree with:

  1. For every rule, insert the Event into Clara rules in-memory database
  2. Query for the events that fitted the Clara rules

Without knowing the constraints of your codebase I can't say for sure, but I feel that all events could be processed in the same session for:

  1. Apply corresponding changes for each Clara rule type
  2. Repeat step 1 to 4 until there is no more Records to apply the rules too.

But regardless of 3&4, I think that the Insert & Query pattern should achieve what you need.

stLalo commented 5 years ago

So what I have in mind would be something like

  1. Insert all my events in one session 
    (-> events
  (mk-session 'some.namespace.rules)
  (insert-all events)
  (fire-rules))
  2. Matching rules will be inserted such as this 
    (defrule do-something-coll
[?event <- Event (= type :tulip)]
=>
(insert! (->Event ?event))
  3. (defquery get-tulips
[]
(Event (= :tulip (:type ?event))))
  4. Apply changes to type = tulip 
    (defn thingamagiq
[Event]
(assoc Event :howdy "howdy"))

Repeat the same steps with other rules and queries. I based my approach from this example

EthanEChristian commented 5 years ago 
(defrule do-something-coll
[?event <- Event (= type :tulip)]
=>
 (insert! (->Event ?event))

Rules that bind and insert the same fact type can be dangerous as they are prone to infinite looping.

For queries, you could use parameters to be more flexible in the event that you need other event types:

(defquery get-events-by-type
[:?type]
[?event <- Event (= ?type type)])
stLalo commented 5 years ago

So to continue with this thread, I went back and did the following to my rules.

  1. Create a new Record with similar attributes from the main Event Record.

  2. 
  (mk-session 'some.namespace.rules)
  (insert-all events)
  (fire-rules)) 
  3. 
(defrule do-something-coll
       [?event <- Event (= type :tulip)]
      =>
        (insert! (map->NewEvent (assoc ?event :color "blue"))
  4. 
(defquery get-events
   []
    [?newevents <- NewEvent])

This works wonderful for only one rule. However, I have cases where Event Records will be true for several Rules. Inserting! the true fact into NewEvent, will cause the creation of duplicates of the same events but with different values here and there. I was thinking about using Retract! on the rules after the first one, but it seems not to return the previous NewEvent

EthanEChristian commented 5 years ago

@stLalo, I think by modeling this slightly differently you could get away from the need to have explicit retractions. Again retractions can lead to unintended looping behavior.

From some of the things that you posted so far, it looks like you are trying to apply a series of changes to the original model. The way I was thinking would be something like this:

Facts:

(defrecord Event [id type color stem-length])
(defrecord FinalEvent [id type color stem-length])
(defrecord Correction [id field val])

Correction Rules:

(defrule correction-one 
  [?event <- Event (= type :tulip) (= ?id id)]
  =>
  (insert! (->Correction ?id :color "blue")))

(defrule correction-two 
  [?event <- Event (= type :tulip) 
    (= ?id id)
    (> stem-length 15)]
  =>
  (insert! (->Correction ?id :stem-length 15)))

Consolidation:

(defrule final-event
  [?event <- Event (= ?id id)]
  [?corrections <- (acc/all) :from [Correction (= id ?id)]]
  =>
  (let [final-event (reduce (fn [final correction]
                              (assoc final (:field correction) (:val correction)))
                            ?event
                            ?corrections)]
    (insert! (map->FinalEvent final-event))))

Query:

(defquery get-events
    []
   [?final-event <- FinalEvent])

This side steps the duplicate event behavior by adding an accumulator and consolidation rule.

EthanEChristian commented 5 years ago

I suppose if the Session had enough scope it might include rules that wanted to correct the same field. That might require a hierarchy of what corrections need to be applied in what order.

The approach above could also be done where the correction fact would contain a fn instead of key/value pairs.