Tutorials

[tgalery]

Hi this is more of a question. I know that there are api docs available, but I was just wondering if you guys have a tutorial on how to run the server and create a basic workflow.

[evrenesat]

Unfortunately we don't have a tutorial in English, yet. But you can run and inspect our main project ulakbus which is built on zengine. It's even have a pre-built vagrant box for development purposes.

You can run the websocket (tornado) server with ./manage.py runserver command and backend worker with ./manage.py runworker command. Workflow diagrams are stored within diagrams folder which can be opened and modified with Camunda Modeler.

[tgalery]

Thanks @evrenesat I took a look at the documentations you sent, however I'm getting a 404 when looking for the vagrantfile. Could you post a direct link here somewhere ?

[tgalery]

Sorry, I got it now, the translation was messing up with the url.

[evrenesat]

Thiago, since we have trouble with our cluster, buildbot can't function properly to build our docs with sphinx. So please add the following commands manually.

in virtualenv:

$ pip install tornado pika

$ python manage.py migrate --model all

then use this command:

$ python ./manage.py load_fixture fixtures/

sorry for this documentation mess.

[knipknap]

I recently started to think about a containerized REST API for SpiffWorkflow, and would probably have gone along a similar route as zengine (I was thinking of RabbitMQ+CouchDB+uwsgi or Daphne+Swagger, without having done my homework).

Now I am wondering whether zengine already does what I planned. Does your engine provide a REST API? I am having a hard time navigating the ulakbus source, due to a bit of a language barrier. Is there a list of REST methods or AMQP services anywhere?

[kunthar]

@knipknap At first, thank you for what you've created to guide us \o/ . We really would like to know what do you mean with "containerized REST API"? Could you please elaborate on this more. What will be the main purpose of this? We can then properly answer your questions. Regarding ulakbus, you can totally ignore it and create a new project combining zengine and dependencies. We will publish a document describes how to achieve this soon for you.

[knipknap]

Sounds great, Thanks @kunthar! I will try to read into the zengine code soon.

I want to allow for deploying a docker container that provides some sort of API for running and executing workflows. The container needs to have an API (either REST, or RabbitMQ tasks) to allow for uploading specifications and custom tasks, and stepping through instances of a workflow.

In short, I want to solve the problems of deployment, storage, and long-running tasks for SpiffWorkflow.

[alirizakeles]

@knipknap let me explain what zengine can provide you. As a web framework, it has a session management part including user login / logout workflows, authentication / authorization and a session cache mechanism. zengine creates permissions from workflow and workflow steps (one permission for each step) and authorization system works with permissions.

Zengine provides a server deamon for http and websocket. http is for public workflows only, since websocket is for authenticated and authorized communication between client and zengine.

Zengine provides another deamon called worker. Worker simply is the core of which your great job spiff is the one of the most important part. It gets message from server, check it, runs wf engine to iterate it, runs module / script / service corresponding current task, gets results and sent it back. Server and worker communicates each other over AMQP.

We decided to use BPMN2 to represent wf diagrams, we have own parser and we store them in db. Zengine depends on Riak KV (please see pyoko orm project which we develop for Riak), this can be changed but i think it is not in near future. Also zengine have some forms, views, UI widgets etc which can be found in many web framework.

I understand what you wrote that you want to make spiff a service. We have had an idea to separate engine part and make it work as a service which we call it "workflow engine as a service". Please notice that it is not "workflow as a service", it is workflow engine as a service. As you know there are many workflow service platforms providing some predefined business logic. We do not mean that. This one is neutral just an engine. It serves both amqp and http, it has specs and responsible for all engine related questions: get specs, get state, what is next, etc..

Lets continue to talk more and detailed. Maybe we can help each other about same / similar intents.

[knipknap]

Awesome, Thanks! Your explanation helped me make sense of the source code.

I see that your worker is implemented mostly in zengine.wf_daemon and zengine.engine, and the worker is doing what I was looking for. The only extra part that would be needed is the ability to register custom tasks:

What is interesting to me is that instead of creating specialized tasks derived from SpiffWorkflow.spec.TaskSpec, you are assigning your code to ZEngine.wf_activities, which is essentially loaded from disk by importing Python modules, and then execute this as the workflow progresses. I had not considered doing it this way, but it is perfectly fine and frees you from having to create a specialized serializer for each task.

Trying to bottle all this up in a docker container may be challenging however; I have no clear idea how to properly register the custom tasks. The worker would probably need to allow for uploading custom Python files, and execute them in yet another sandbox.

One thing I haven't seen yet is where you handle authorization and permissions; I am guessing this happens exclusively in the view, not on the worker?