recommend purge/archieve instances

[kstan79]

I'm notice that the workflow instance remain in the mongodb collection after complete the execution. It will cause performance impact when the collection grow large. Base on my experience in Kogito/JBPMN the database will remove the workflow record after complete the process, I'm recommend we can implement similar approach as below:

1. define additional environment variable of bpmnEndStrategy = "delete/keep/archieve"
2. delete mean remove from database, dont want to keep it
3. keep remain as current
4. archieve will store into wf_archieves

[ralphhanna]

How about the configuration.ts parameter:

/**
    *   system will check both days, size conditions if exist
    */

   DeleteStrategy= {
        deleteAfterEnd: { days: 30 ,      /* number of days to delete after completion
                                                0 means right away
                                                -1 means never 
                                            */
                            size: 1000     /* delete after size reaches certain size */
                            },
        archiveOnDelete: true
   };

appDelegate.ts will have a method to handle this on Thanks for the suggestion, great idea, very useful.

BTW, have you done any benchmarks?

[kstan79]

i no yet do any benchmark. but I tried every 5 second and the historical record look too much. yes you may implement delegate, base on your reply I suggest clear cut option.

{
     deleteFromArchieve: {days:30, size:1000}. //0 days mean keep forever, 0 size mean unlimited. default both = 0
}

since we have auto purge archieve options, we no need archiveOnDelete: true anymore, which is:

1. all completed instance directly transfer to archieve, no option available
2. archieve will clean up base on configuration, default configuration store 7 days (like kafka)

[ralphhanna]

OK, got it working now

• New Method server.dataStore.archive(query)
• New Script archive.js daysToArchive daysToCleanArchive
• You run schedule a job to clean up the data

  node dist/scripts/archive.js 30 90 

  Just for the record, here are the results of my benchmark:

  
  ------------- at start after cleaning the data
  -- only 100 instances and 100 items in wait state

start Buy Used Car

mongo.insert:wf_locks: 0.636ms mongo.insert:wf_instances: 0.597ms mongo.update:wf_instances: 0.762ms mongo.remove:wf_locks: 0.667ms mongo.find:wf_instances: 0.038ms mongo.insert:wf_locks: 0.294ms mongo.find:wf_instances: 0.029ms mongo.update:wf_instances: 0.86ms mongo.update:wf_instances: 0.889ms mongo.remove:wf_locks: 0.336ms invoke: 6.654ms mongo.find:wf_instances: 0.033ms mongo.insert:wf_locks: 0.254ms mongo.find:wf_instances: 0.024ms mongo.update:wf_instances: 3.013ms mongo.remove:wf_locks: 0.716ms mongo.find:wf_instances: 0.057ms mongo.insert:wf_locks: 0.43ms mongo.find:wf_instances: 0.064ms mongo.update:wf_instances: 3.861ms mongo.remove:wf_locks: 1.455ms invoke: 13.06ms mongo.find:wf_instances: 0.09ms find: 40.099ms find returned 102 mongo.find:wf_instances: 0.042ms mongo.insert:wf_locks: 0.329ms mongo.find:wf_instances: 0.026ms mongo.update:wf_instances: 2.094ms mongo.update:wf_instances: 1.636ms mongo.remove:wf_locks: 0.414ms invoke: 9.778ms car-0: 84.167ms

start Buy Used Car

mongo.insert:wf_locks: 0.31ms mongo.insert:wf_instances: 0.409ms mongo.update:wf_instances: 1.852ms mongo.remove:wf_locks: 0.408ms mongo.find:wf_instances: 0.034ms mongo.insert:wf_locks: 0.291ms mongo.find:wf_instances: 0.011ms mongo.update:wf_instances: 1.629ms mongo.update:wf_instances: 4.226ms mongo.remove:wf_locks: 0.489ms invoke: 25.918ms mongo.find:wf_instances: 0.017ms mongo.insert:wf_locks: 0.332ms mongo.find:wf_instances: 0.012ms mongo.update:wf_instances: 2.148ms mongo.remove:wf_locks: 0.421ms mongo.find:wf_instances: 0.019ms mongo.insert:wf_locks: 0.291ms mongo.find:wf_instances: 0.011ms mongo.update:wf_instances: 2.885ms mongo.remove:wf_locks: 0.648ms invoke: 15.015ms mongo.find:wf_instances: 0.018ms find: 78.44ms find returned 102 mongo.find:wf_instances: 0.026ms mongo.insert:wf_locks: 0.512ms mongo.find:wf_instances: 0.018ms mongo.update:wf_instances: 4.985ms mongo.update:wf_instances: 3.111ms mongo.remove:wf_locks: 0.753ms invoke: 51.493ms car-7074: 200.918ms

--- with 10K records loaded

start Buy Used Car

mongo.insert:wf_locks: 0.236ms mongo.insert:wf_instances: 0.356ms mongo.update:wf_instances: 1.753ms mongo.remove:wf_locks: 0.401ms mongo.find:wf_instances: 0.018ms mongo.insert:wf_locks: 0.262ms mongo.find:wf_instances: 0.009ms mongo.update:wf_instances: 1.711ms mongo.update:wf_instances: 1.899ms mongo.remove:wf_locks: 0.71ms invoke: 24.353ms mongo.find:wf_instances: 0.021ms mongo.insert:wf_locks: 0.22ms mongo.find:wf_instances: 0.01ms mongo.update:wf_instances: 2.041ms mongo.remove:wf_locks: 0.333ms mongo.find:wf_instances: 0.016ms mongo.insert:wf_locks: 0.19ms mongo.find:wf_instances: 0.008ms mongo.update:wf_instances: 2.282ms mongo.remove:wf_locks: 0.349ms invoke: 15.213ms mongo.find:wf_instances: 0.015ms find: 66.448ms find returned 102 mongo.find:wf_instances: 0.019ms mongo.insert:wf_locks: 0.272ms mongo.find:wf_instances: 0.009ms mongo.update:wf_instances: 2.736ms mongo.update:wf_instances: 2.598ms mongo.remove:wf_locks: 0.506ms invoke: 29.052ms car-10000: 170.839ms STRESS: 23:03.831 (m:ss.mmm)

![image](https://github.com/bpmnServer/bpmn-server/assets/11893416/1083459f-3105-4f16-9f3c-b375ffc72981)

In Summary, MongoDB didn't show any degradation with 10k records

[kstan79]

thanks, i'd update to latest but I dont know how to implement it:

1. I create a schedule bpmn, seems it finish execute remain in instance, I need to call specifc method to move it?
2. Your example indicate that we need run separated node cli, you mean no option available right, we need to schedule cli archive it?
3. Seems there is server.dataStore.archive(query), I can run archive process using mongodb query in source code is it?

[kstan79]

You are the man! tested in 2.1.7, using below strategy:

this.bpmnServer = new BPMNServer(config, new HiddenLogger());
setInterval(async ()=>{
      console.log("Archiving")
      await this.bpmnServer.dataStore.archive({status:'end'})
      console.log("Archive done")
    },10*1000*60)

it work well and great thanks!