Allow "generic" resource deployment to zeebe

marstamm commented 1 month ago

Is your feature request related to a problem? Please describe. We are working on a RPA integration for Camunda. For this, we need to manage the script files that the process has access to. Currently, only files with a specific type (.bpmn, .dmn and .form) can be deployed, anything else is rejected.

Describe the solution you'd like Allow any file type to be part of the deployment and access it through the deployment.

Describe alternatives you've considered

Store script files outside of zeebe, e.g. on S3: We want to present RPA as a core Camunda Feature. The RPA scripts will be the same for all instances of the process, so I think it should be handled the same as other linked resources, like DMN or Forms.
Only support .robot files instead of a generic resource: This would work for our use-case, as only .robot files are supported. However, it makes it harder for users and customers to implement their custom solution, as they need to solve the problems for storage and referencing again. It would be great to have a simple solution for "versionBinding: deployment" for any resource.

Additional context This was also discussed on Slack. This thread contains the discussion and some more background: https://camunda.slack.com/archives/CSQ2E3BT4/p1722330790414699

RPA Epic: https://github.com/camunda/product-hub/issues/2412

megglos commented 1 month ago

@aleksander-dytko as this is part of Boat, we should align on how to incorporate this.

@marstamm what's deadline until you need this?

marstamm commented 1 month ago

It would be great if we could get this into 8.6, so we can present it at Camunda Con New York.

megglos commented 1 month ago

@marstamm we don't have capacity to include this into 8.6 still, is there any way you could unblock the demo for Camunda Con without this change in Zeebe?

cc @aleksander-dytko

megglos commented 1 month ago

@aleksander-dytko @bastiankoerber can you please align on a timeline for this going forward?

bastiankoerber commented 1 month ago

@marstamm & @nikku will follow up here

marstamm commented 1 month ago

We will be able to build RPA v0.1 without deployment to zeebe (cf. Discussion).

For v1.0, we might want this or a similar feature. We can postpone the discussion for this until then, potentially with a contribution from the RPA team.

This Feature can be de-prioritized for now

nikku commented 1 month ago

Some further insights into what our investigation found:

User Problem

I want to deploy a resource with Zeebe, with the purpose to access what is deployed as part of my job worker.

Assessment

This only works if we have a solid story how to access them from within a job worker in place, right now this story, to my knowledge, does not exist.

What folks would do today if we had "generic resource deployment" is to (1) either listen to Zeebe exporter stream for "resource deployed" events, or (2) ask operate for the resources (where in operate, this information may be available delayed).

Potential Solution

We could offer a simple API (REST or zeebe) to retrieve a resource local to a deployment (versionBinding=deployment), or via key from the job worker.

Example Scenario

I deploy an external configuration (.json file) with my process
I execute a job worker
During execution I want to access the JSON file to understand "what to do"
I need to get from job -> process instance -> process definition -> deployment and ask operate for a particular resource deployed with the process (?)
I get access to the resource, and can follow-up
:arrow_right: I'd like to be able to do this in a simple manner
For certain known and Zeebe managed resources with versionBinding=latest I may want to get access to the resource key that Zeebe assigned to it, so I can fetch exactly what I should.

camunda / camunda