How to use demo

[kavink]

Selinon looks very powerful and exactly what I want it do . But bringing up the demo, I can run flow from swagger API, but im unable to figure out , how do I monitor/see if flow failed, completed or is running.

It returns "dispatcher_id": "4da2e182-fd9d-415c-b692-76299da2edf9"

But there is no results api which I can hit and see what flow did and its results for each stage and params and output etc.

A real life complex demo showing off all features would help a lot in adoption, as of now im trying to dig through docs and code on after triggering tasks how do I see its results/success or failures. And then can I re-run just failed tasks or force re-run the flow.

[fridex]

Hi @kavink,

Selinon looks very powerful and exactly what I want it do . But bringing up the demo, I can run flow from swagger API, but im unable to figure out , how do I monitor/see if flow failed, completed or is running.

It returns "dispatcher_id": "4da2e182-fd9d-415c-b692-76299da2edf9"

If you bring it up using docker-compose up, you should see JSON output printed by workers. Based on that output you can get a notion what's going on in the system - also Celery Flower can give you info about running tasks on Celery level.

But there is no results api which I can hit and see what flow did and its results for each stage and params and output etc.

You can go to Redis commander, PGWeb, Mongo Express, Minio S3 or others (based on storage adapter used) to see what results land in databases as of now.

Thanks for the result API tip. I will incorporate it to demo.

A real life complex demo showing off all features would help a lot in adoption,

+1. Soon I will fix the last known issue to me and release 1.0.0. Based on things we've done with selinon, I would say it's mature enough even today, but lacking real how-to on how to set up environment is definitely something I need to work on. I want to refresh demo for the GeoPython conference where I will have a talk - feel free to attend.

as of now im trying to dig through docs and code on after triggering tasks how do I see its results/success or failures. And then can I re-run just failed tasks or force re-run the flow.

Selinon is stateless when it comes to what flows were run in the past. If you would like to track which tasks failed, you can implement store_error method that should keep track of persistence of failures. Later you can query which tasks failed and trigger selective task flows where you just say which tasks in which flows should be started again. If you have some data dependencies between tasks and you don't want to run precedence tasks of the failed ones, you will need to also implement selective run function that just tels selinon from where results should be gathered to reuse them.

If anything is unclear, feel free to ask. I agree that initial steps of the learning curve can be tricky, but IMHO it can worth it.

[kavink]

@fridex Yes, I already can see its powerful and stateless, Im just hoping to see a complex real world example, So I can learn from example and modify for my use case .

Will there be a recorded video of the conference ?

Basically even an example will be powerful.

• A DAG with flow and tasks/subtasks.
• A task or flow input depending on return value from a previous/task or flow.
• Ability to monitor current or historical status of runs.
• Ability to re-run flow for just failed or force rerun.

Wondering if you can guide me in right direction to do the following.

I already have a celery task (sync) which takes in a JSON, parses JSON and runs the function. Also have task workflow defined in DB/yaml similar to explained here but I can easily use Selinon .

• How do I see Results/State of the triggered flow ?
• How do Store results on success/failure in Postgres for each stage

It might be easy to do ... Still learning about Selinon.

[kavink]

Also Selective runs, how do I make it dynamic so I dont need to write a YAML for rerun, but just write a flask API end point which runs like failed task and downwards, Would be great if you can incorporate that in Demo when you enhance the demo.

[fridex]

@fridex Yes, I already can see its powerful and stateless, Im just hoping to see a complex real world example, So I can learn from example and modify for my use case .

Will there be a recorded video of the conference ?

I hope so.

Basically even an example will be powerful.

A DAG with flow and tasks/subtasks. Selinon flows are not DAG, it can be any graph - no acyclic graph limitations.

A task or flow input depending on return value from a previous/task or flow.

That's already present there - see Fibonacci flow.

Ability to monitor current or historical status of runs.

See tracing mechanism.

Wondering if you can guide me in right direction to do the following.

I already have a celery task (sync) which takes in a JSON, parses JSON and runs the function. Also have task workflow defined in DB/yaml similar to explained here but I can easily use Selinon . How do I see Results/State of the triggered flow ? How do Store results on success/failure in Postgres for each stage It might be easy to do ... Still learning about Selinon.

Please do not duplicate questions. Let's keep one thread open.