CRUD pipelines

[jnehring]

When designing the pipelines feature we talked about adding the possibility to create / read / update / delete pipelines on the server. I think this should be part of FREME 0.4.

So we add POST / GET / PUT / DELETE endpoints to the pipelining module. Pipelines will be stored in the MySQL database. Each pipeline has label and description so others can re-use pipelines.

@ghsnd What do you think of this feature?

[ghsnd]

This feature is possible, however the same issues as templating with e-Link arise:

• Everybody can create, update and delete pipelines, so there will be plenty of them.
• Some pipelines won't work if not correct, some will be duplicates of others, ...
• What if someone modifies or deletes a pipeline used by somebody else?
• What if the description is not precise?

How do we keep control over these issues?

I was thinking of 2 options:

1. Using the java code from the pipelines repo makes it easy to create good pipelines. This eliminates the need to store pipelines on the server (the current system), but this requires some work from the end-user, and it assumes (s)he is using Java which is not always true of course. Also, pipelines cannot be re-used.
2. Implementing CRUD + a validator + having the possibility to create pipelines via the browser (via a form or some other GUI). We still have some of the mentioned issues, but it makes it easier to create valid pipelines.

[jnehring]

Everybody can create, update and delete pipelines, so there will be plenty of them.

For e-link templates we had the idea to implement a life time for templates. That means that the create templates endpoint has a parameter "persist" which has default value false. When persist=false then the templates will be automatically deleted one week after their creation. This will take care for all these test templates.

What if someone modifies or deletes a pipeline used by somebody else?

We will add the access control stuff so you can only edit / delete your own pipelines.

Implementing CRUD + a validator + having the possibility to create pipelines via the browser (via a form or some other GUI). We still have some of the mentioned issues, but it makes it easier to create valid pipelines.

Might be a good idea. However I think this is a bigger task. But it might be really cool and help evelopment of NLP applications. I think that the pipeline module can include other NIF ready APIs apart from FREME out of the box. So this might be really cool for developers of NLP applications.

Maybe we should wait for adding CRUD pipelines stuff. We should wait if someone asks for that feature. I think until now nobody has really used the pipelines. Maybe we should wait for some feedback before putting more work in the development of pipelines.

[ghsnd]

On 11/09/15 12:01, Jan Nehring wrote:

Everybody can create, update and delete pipelines, so there will be
plenty of them.

For e-link templates we had the idea to implement a life time for templates. That means that the create templates endpoint has a parameter "persist" which has default value false. When |persist=false| then the templates will be automatically deleted one week after their creation. This will take care for all these test templates.

What if someone modifies or deletes a pipeline used by somebody else?

We will add the access control stuff so you can only edit / delete your own pipelines.

Implementing CRUD + a validator + having the possibility to create
pipelines via the browser (via a form or some other GUI). We still
have some of the mentioned issues, but it makes it easier to create
valid pipelines.

Might be a good idea. However I think this is a bigger task. But it might be really cool and help evelopment of NLP applications. I think that the pipeline module can include other NIF ready APIs apart from FREME out of the box. So this might be really cool for developers of NLP applications.

Maybe we should wait for adding CRUD pipelines stuff. We should wait if someone asks for that feature. I think until now nobody has really used the pipelines. Maybe we should wait for some feedback before putting more work in the development of pipelines.

OK, let's do that.

— Reply to this email directly or view it on GitHub https://github.com/freme-project/pipelines/issues/16#issuecomment-139504895.

[jnehring]

dublicate of #17 and https://github.com/freme-project/technical-discussion/issues/62