Refact: StreamTask, OATask - order of processing/adaption on new/obsolete instances

[detlefarend]

Motivation Currently, methods StreamTask.run() and OATask.adapt() get instances to be processed in two different parameters

• p_inst_new : list[Instance]
• p_inst_del : list[Instance]

As a result, both lists are processed separately and block-wise.

Since an instance has a unique time stamp (see issue #987) the processing/adaptation should take place in an overall order by time stamp.

One possibility is to replace parameters p_inst_new, p_inst_del by a new parameter

• p_inst : dict[tstamp,tuple(new_del_flag, Instance)]...

Benefit Processing/adaptation is much more plausible than before.

Are the proposed changes backward compatible?

• [ ] Yes
• [x] No

Related development objects

• [x] bf.streams
  • [x] bf.streams.StreamTask
  • [x] bf.streams.StreamWorkflow
  • [x] bf.streams.StreamScenario
  • [x] bf.stream.tasks.Window
    • [x] 2D plot
    • [x] 3D plot
    • [x] nD plot
  • [x] bf.streams.tasks.Rearranger
  • [x] bf.streams.tasks.Deriver
• [ ] oa.streams
  • [x] oa.streams.OATask
  • [x] oa.streams.tasks.BoundaryDetector
    • [x] nD Plot
  • [ ] oa.streams.tasks.Normalizer (Laxmikant)
    • [ ] NormalizerMinMax
      • [x] 2D plot Plot raw/normalized should be identical (except scaling)...
      • [x] 3D plot Plot raw/normalized should be identical (except scaling)...
      • [x] nD Plot Plot raw/normalized should be identical (except scaling)...
    • [ ] NormalizerZTrans
      • [x] 2D plot
      • [x] 3D plot
      • [x] nD Plot
  • [x] oa.streams.tasks.ClusterAnalyzer
  • [x] oa.streams.tasks.AnomalyDetector (Syam)
• [ ] oa.systems (Laxmikant)
  • [ ] Refactoring of all methods using p_inst_new, p_inst_del (needs to be replaced by p_inst : InstDict). The best is to search for p_inst_new, p_inst_del in the code. Here are some findings:
    • [x] Method OAFctBroken._run_wf_broken()
    • [x] Method OAFctSTrans.simulate_reaction(): l. 213
    • [x] Method OAFctSTrans._run_wf_strans()
    • [ ] ...
• [x] MLPro-Int-River
• [x] MLPro-Int-scikit-learn

[detlefarend]

@steveyuwono @syamrajsatheesh this heavy design weakness came to my notice during working on SPARCCStream. Unfortunately, the resulting changes are extensive. But better now than after releasing MLPro 2... What is your opinion? Do you see a better way to solve this?

[steveyuwono]

@steveyuwono @syamrajsatheesh this heavy design weakness came to my notice during working on SPARCCStream. Unfortunately, the resulting changes are extensive. But better now than after releasing MLPro 2... What is your opinion? Do you see a better way to solve this?

Hi @detlefarend , what do you mean by block-wise? Regarding the order of processing the instances, I agree with the idea of ordering based on its timestamp rather than first the addition and then the deletion.

Regarding the proposed solution, a dict is a good solution as the new parameter.

[detlefarend]

Hi @steveyuwono, with block-weise I mean that we currently process all new instances in method _adapt() before we process all obsolete instances in method _adapt_reverse().

Ok, I will do the refactoring tomorrow in a separate branch. The best is to merge everything to main before. Any doubts, better ideas, last words?

[steveyuwono]

Okay, sounds good! Nothing else from my side...

[detlefarend]

@laxmikantbaheti I already refactored the normalizers due to the new stream instance management (see oa.streams.tasks.OATask, run() and adapt()). I also added numerous further howtos to validate every single oa task in 2d/3d/nd mode. Two things are not yet solved:

• Normalizer MinMax: parameters are updated during plotting but there are still differences between input signal and scaled plot. I expected an identical plot in both windows. But this is maybe the wrong approach. The normalizer re-normalizes no internal data. Why should it do for the plot data? Let's talk about this.
• Z-Transformation: not yet running at all. Equations need to be reviewed as mentioned in #993

[detlefarend]

@laxmikantbaheti I also started refactoring oa.systems. Here further things are to do. I added a brief description to the issue.