add feature to export and import BPMN state, incl. resume capability

[nitram509]

it would be awesome for use cases where long-running processes and are implemented in short living microservices, if lib-bpmn-engine would be able to export and import+resume it's internal state.

• [x] must: the data must be represented as ASCII-save string, to allow maximum compatibility and avoid codepage/unicode issues
• [x] could: the data format should be human readable, e.g. JSON
• [x] optional: data compression
• [x] optional: base64 encoding
• [ ] must: the feature must be agnostic to library upgrades and for example allowing schema changes (optional: split this into another feature)
  • e.g. version info in the serialized file
• [ ] documentation is up to date

[cq-z]

Suggestion: add ”dump“ and ” regain“ api 。So that compatible on the storage implementation The storage implementation can be create tables (events 、tasks...) to save state and data like：https://github.com/antlinker/flow/blob/develop/schema/s_flow.go

[nitram509]

Hi @cq-z thank you for your suggestion. I will try my best to make it somewhat compatible, but will not make it a 1:1 fit. I rather think of optimizing for this lib but allowing an optional "adapter function", which could map the schemas.

@cq-z May I ask: what use case do you have in mind, for making this suggestion?

PS: See also https://github.com/nitram509/lib-bpmn-engine/blob/feature/export_and_import_BPMN_state/pkg/bpmn_engine/engine_marshal.go for my latest work in progress.

[cq-z]

example :
Marshal to model like https://gorm.io/docs/has_many.html, after that model to save。 model Select， after that model Unmarshal to BpmnEngineState

Marshal support [] byte or struct , or like https://github.com/jinzhu/copier implementation BpmnEngineState to model Whether the scalability is better

[cq-z]

support grom like demo

package main

import (
    "fmt"
    "github.com/jinzhu/copier"
    "github.com/nitram509/lib-bpmn-engine/pkg/bpmn_engine"
    "github.com/stretchr/testify/assert"
    "gorm.io/driver/mysql"
    "gorm.io/gorm"
    "testing"
)

type serializedBpmnEngine struct {
    Id                   int64                  `gorm:"foreignKey:autoIncrement"`
    Version              int                    `json:"version"`
    Name                 string                 `json:"name"`
    MessageSubscriptions []*MessageSubscription `json:"MessageSubscriptions,omitempty" gorm:"foreignKey:BpmnId"  copier:"GetMessageSubscriptions"`
    Processes            []ProcessInfo          `json:"Processes,omitempty" gorm:"foreignKey:BpmnId" copier:"GetProcessInstances"`
}
type MessageSubscription struct {
    Id                 int64 `gorm:"foreignKey:autoIncrement"`
    BpmnId             int64
    ElementId          string `json:"ElementId"`
    ElementInstanceKey int64  `json:"ElementInstanceKey"`
    ProcessInstanceKey int64  `json:"ProcessInstanceKey"`
    Name               string `json:"Name"`
    State              string `json:"State"`
    CreatedAt          int64  `json:"CreatedAt"`
}
type ProcessInfo struct {
    Id            int64 `gorm:"foreignKey:autoIncrement"`
    BpmnId        int64
    BpmnProcessId string `json:"BpmnProcessId"` // The ID as defined in the BPMN file
    Version       int32  `json:"Version"`       // A version of the process, default=1, incremented, when another process with the same ID is loaded
    ProcessKey    int64  `json:"ProcessKey"`    // The engines key for this given process with version

    // TODO: make them private again?
    Definitions   string `json:"definitions"`   // parsed file content
    ChecksumBytes string `json:"checksumBytes"` // internal checksum to identify different versions
}

var bpmnByte = `
<?xml version="1.0" encoding="UTF-8"?>
<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL" xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI" xmlns:dc="http://www.omg.org/spec/DD/20100524/DC" xmlns:zeebe="http://camunda.org/schema/zeebe/1.0" xmlns:di="http://www.omg.org/spec/DD/20100524/DI" xmlns:modeler="http://camunda.org/schema/modeler/1.0" id="Definitions_1u3x2yl" targetNamespace="http://bpmn.io/schema/bpmn" exporter="Camunda Modeler" exporterVersion="5.0.0" modeler:executionPlatform="Camunda Cloud" modeler:executionPlatformVersion="1.0.0">
  <bpmn:process id="simple-user-task" name="simple-user-task" isExecutable="true">
    <bpmn:startEvent id="StartEvent_1">
      <bpmn:outgoing>Flow_0xt1d7q</bpmn:outgoing>
    </bpmn:startEvent>
    <bpmn:sequenceFlow id="Flow_0xt1d7q" sourceRef="StartEvent_1" targetRef="user-task" />
    <bpmn:endEvent id="Event_1j4mcqg">
      <bpmn:incoming>Flow_1vz4oo2</bpmn:incoming>
    </bpmn:endEvent>
    <bpmn:sequenceFlow id="Flow_1vz4oo2" sourceRef="user-task" targetRef="Event_1j4mcqg" />
    <bpmn:userTask id="user-task" name="user-task">
      <bpmn:extensionElements>
        <zeebe:assignmentDefinition assignee="assignee" candidateGroups="candicate-groups" />
        <zeebe:formDefinition formKey="form-key" />
      </bpmn:extensionElements>
      <bpmn:incoming>Flow_0xt1d7q</bpmn:incoming>
      <bpmn:outgoing>Flow_1vz4oo2</bpmn:outgoing>
    </bpmn:userTask>
  </bpmn:process>
  <bpmndi:BPMNDiagram id="BPMNDiagram_1">
    <bpmndi:BPMNPlane id="BPMNPlane_1" bpmnElement="simple-user-task">
      <bpmndi:BPMNEdge id="Flow_1vz4oo2_di" bpmnElement="Flow_1vz4oo2">
        <di:waypoint x="370" y="117" />
        <di:waypoint x="432" y="117" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNEdge id="Flow_0xt1d7q_di" bpmnElement="Flow_0xt1d7q">
        <di:waypoint x="215" y="117" />
        <di:waypoint x="270" y="117" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNShape id="_BPMNShape_StartEvent_2" bpmnElement="StartEvent_1">
        <dc:Bounds x="179" y="99" width="36" height="36" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNShape id="Event_1j4mcqg_di" bpmnElement="Event_1j4mcqg">
        <dc:Bounds x="432" y="99" width="36" height="36" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNShape id="Activity_102w2rp_di" bpmnElement="user-task">
        <dc:Bounds x="270" y="77" width="100" height="80" />
        <bpmndi:BPMNLabel />
      </bpmndi:BPMNShape>
    </bpmndi:BPMNPlane>
  </bpmndi:BPMNDiagram>
</bpmn:definitions>
`

func TestGrom(t *testing.T) {

    db, err := gorm.Open(mysql.Open("root:123456@tcp(127.0.0.1:3306)/test"), &gorm.Config{})
    if err != nil {
        panic("failed to connect database")
    }

    db.AutoMigrate(serializedBpmnEngine{}, ProcessInfo{}, MessageSubscription{})

    // setup
    bpmnEngine := bpmn_engine.New("name")
    process, _ := bpmnEngine.LoadFromBytes([]byte(bpmnByte))

    variables := map[string]interface{}{
        "price": -50,
    }

    // when
    bpmnEngine.CreateAndRunInstance(process.ProcessKey, variables)
    processes := bpmnEngine.GetProcessInstances()
    var gormProcesses []ProcessInfo
    err = copier.Copy(&gormProcesses, processes)
    fmt.Println(err)
    messageSubscriptions := bpmnEngine.GetMessageSubscriptions()
    var gormMessageSubscriptions []*MessageSubscription
    err = copier.Copy(&gormMessageSubscriptions, messageSubscriptions)
    fmt.Println(err)
    s := &serializedBpmnEngine{
        Name:                 bpmnEngine.GetName(),
        Processes:            gormProcesses,
        MessageSubscriptions: gormMessageSubscriptions,
    }
    //err = copier.CopyWithOption(s, bpmnEngine, copier.Option{
    //  IgnoreEmpty: true,
    //  DeepCopy:    true,
    //  Converters: []copier.TypeConverter{
    //      {
    //          SrcType: []ProcessInfo{},
    //          DstType: []bpmn_engine.ProcessInfo{},
    //          Fn: func(src interface{}) (interface{}, error) {
    //              s, ok := src.(string)
    //              if !ok {
    //                  return nil, fmt.Errorf("src type not matching")
    //              }
    //              return s, nil
    //          },
    //      },
    //  }})
    fmt.Println(err)
    // then
    db.Create(s)
    s1 := &serializedBpmnEngine{}
    db.Model(&serializedBpmnEngine{}).First(&s1)
    var bpmnEngine1 bpmn_engine.BpmnEngineState
    err = copier.CopyWithOption(&bpmnEngine1, s1, copier.Option{})
    assert.Equal(t, bpmnEngine1, bpmnEngine)
}

[cq-z]

Other solutions： bpmnEngine to json, json to grom struct. grom struct to json , json to bpmnEngine.

[eriknyk]

@nitram509 any idea when feature/export_and_import_BPMN_state will be merged to main branch, and release new version? In terms of generic way to persist the state looks enough.

Regards.

[nitram509]

FYI: I'm working on this and have pushed first working version (main branch).

[cq-z]

Hello @nitram509 , feature/export_and_import_BPMN_state is ready?

[nitram509]

@cq-z unfortunately, still in progress.

I found little spare time during the last weeks/months and plan to continue work the next weeks.

[cq-z]

I can do something for you

[nitram509]

Hi @cq-z , @eriknyk ,

I did some work to implement the marshalling and un-marschalling support. If you're interested, have a look at the recent https://github.com/nitram509/lib-bpmn-engine/releases/tag/v0.3.0-rc1 release. Documentation is missing, but have a look at tests/marshalling_test.go which shows how the feature can be used.

I will continue improving documentation and likely do some more changes towards Getter&Setter API rework as the other open issues for a proper v0.3.0 release indicate.

I would be glad to hear your feedback, and if the marshall & un-marshall works for you.