search

botify-labs / simpleflow

Python library for dataflow programming.
https://botify-labs.github.com/simpleflow/
MIT License
68 stars 24 forks source link

couldn't get up and running.... #372

Closed 8secz-johndpope closed 4 years ago

8secz-johndpope commented 4 years ago

The decider and poller seem to start fine - but running the extras demo seems to fall over. it wasn't clear I needed to install aws / shell / configure - but once I did I got further along -

not sure why workerid in xtras demo isn't working..... do I need to pass it in?

https://github.com/8secz-johndpope/aws-swf-python-demo

or

I do like the look of this repo - but from the above repo - some highlights I don't see any bootstrapping / registration of domain / workflow. these missing steps seem to make things make it hard to proceed with this libary. consider use is - I want to use this library to interoperate with nodejs / running machine learning stuff on my home pc / while server / nodejs dishes up the work (StepFunction not sure can help here) https://github.com/neyric/aws-swf

simpleflow worker.start --domain TestDomain --task-list quickstart 2020-06-30T09:39:56 INFO [process=MainProcess, pid=14207]: starting <bound method Poller.start of ActivityPoller(domain=TestDomain, task_list=quickstart)> 2020-06-30T09:39:56 INFO [process=Process-1, pid=14212]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-2, pid=14213]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-3, pid=14214]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-4, pid=14215]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-5, pid=14216]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-6, pid=14217]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-7, pid=14218]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-8, pid=14219]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-9, pid=14220]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-10, pid=14221]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-12, pid=14223]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-11, pid=14222]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-13, pid=14224]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-14, pid=14225]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-15, pid=14226]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-16, pid=14227]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-17, pid=14228]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-18, pid=14229]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-19, pid=14230]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-20, pid=14231]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-21, pid=14232]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-22, pid=14233]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-23, pid=14234]: starting ActivityPoller(task_list=quickstart) on domain TestDomain

simpleflow worker.start --domain TestDomain --task-list quickstart 2020-06-30T09:39:56 INFO [process=MainProcess, pid=14207]: starting <bound method Poller.start of ActivityPoller(domain=TestDomain, task_list=quickstart)> 2020-06-30T09:39:56 INFO [process=Process-1, pid=14212]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-2, pid=14213]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-3, pid=14214]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-4, pid=14215]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-5, pid=14216]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-6, pid=14217]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-7, pid=14218]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-8, pid=14219]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-9, pid=14220]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-10, pid=14221]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-12, pid=14223]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-11, pid=14222]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-13, pid=14224]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-14, pid=14225]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-15, pid=14226]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-16, pid=14227]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-17, pid=14228]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-18, pid=14229]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-19, pid=14230]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-20, pid=14231]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-21, pid=14232]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-22, pid=14233]: starting ActivityPoller(task_list=quickstart) on domain TestDomain 2020-06-30T09:39:56 INFO [process=Process-23, pid=14234]: starting ActivityPoller(task_list=quickstart) on domain TestDomain

(gfla) ➜ simpleflow git:(master) ./extras/demo

Euh... What do I do???

HELP! $ simpleflow --help Usage: simpleflow [OPTIONS] COMMAND [ARGS]...

Options: --format TEXT --header / --no-header --color [auto|always|never] --version Show the version and exit. --help Show this message and exit.

Commands: activity.rerun Rerun an activity task locally. binaries.download Downloads some binaries with simpleflow.download... decider.start Start a decider process to manage workflow executions. info Display versions, settings, and environment variables. standalone Execute a workflow with a single process. task.info Informations on a task. worker.start Start a worker process to handle activity tasks. workflow.filter Filter workflow executions. workflow.info Info about a workflow execution. workflow.list Active workflow executions. workflow.profile Profile of a workflow. workflow.restart Workflow associated with WORKFLOW_ID and optionally... workflow.start Start the workflow defined in the WORKFLOW module. workflow.tasks Tasks of a workflow execution. workflow.terminate Workflow associated with WORKFLOW and optionally...

OK. So, now let's start a workflow.

$ simpleflow workflow.start --domain TestDomain --task-list test --input '{"args": [1, 120]}' examples.basic.BasicWorkflow Traceback (most recent call last): File "/home/jp/miniconda3/envs/gfla/lib/python3.6/site-packages/simpleflow-0.21.13-py3.6.egg/swf/querysets/workflow.py", line 126, in get response = self.connection.describe_workflow_type(self.domain.name, name, version) File "/home/jp/miniconda3/envs/gfla/lib/python3.6/site-packages/boto-2.49.0-py3.6.egg/boto/swf/layer1.py", line 1009, in describe_workflow_type 'version': workflow_version} File "/home/jp/miniconda3/envs/gfla/lib/python3.6/site-packages/boto-2.49.0-py3.6.egg/boto/swf/layer1.py", line 118, in json_request return self.make_request(action, json_input, object_hook) File "/home/jp/miniconda3/envs/gfla/lib/python3.6/site-packages/boto-2.49.0-py3.6.egg/boto/swf/layer1.py", line 145, in make_request raise excp_cls(response.status, response.reason, body=json_body) boto.exception.SWFResponseError: SWFResponseError: 400 Bad Request {'__type': 'com.amazonaws.swf.base.model#UnknownResourceFault', 'message': 'Unknown type: WorkflowType=[name=basic, version=example]'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/jp/miniconda3/envs/gfla/lib/python3.6/site-packages/simpleflow-0.21.13-py3.6.egg/swf/querysets/workflow.py", line 228, in get_or_create decision_tasks_timeout=decision_tasks_timeout) File "/home/jp/miniconda3/envs/gfla/lib/python3.6/site-packages/simpleflow-0.21.13-py3.6.egg/swf/querysets/workflow.py", line 129, in get raise DoesNotExistError(e.body['message']) swf.exceptions.DoesNotExistError: Unknown type: WorkflowType=[name=basic, version=example]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/jp/miniconda3/envs/gfla/lib/python3.6/site-packages/simpleflow-0.21.13-py3.6.egg/swf/models/workflow.py", line 200, in save description=self.description File "/home/jp/miniconda3/envs/gfla/lib/python3.6/site-packages/boto-2.49.0-py3.6.egg/boto/swf/layer1.py", line 767, in register_workflow_type 'description': description, File "/home/jp/miniconda3/envs/gfla/lib/python3.6/site-packages/boto-2.49.0-py3.6.egg/boto/swf/layer1.py", line 118, in json_request return self.make_request(action, json_input, object_hook) File "/home/jp/miniconda3/envs/gfla/lib/python3.6/site-packages/boto-2.49.0-py3.6.egg/boto/swf/layer1.py", line 145, in make_request raise excp_cls(response.status, response.reason, body=json_body) boto.exception.SWFResponseError: SWFResponseError: 400 Bad Request {'__type': 'com.amazonaws.swf.base.model#UnknownResourceFault', 'message': 'Unknown domain: TestDomain'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/jp/miniconda3/envs/gfla/bin/simpleflow", line 33, in sys.exit(load_entry_point('simpleflow==0.21.13', 'console_scripts', 'simpleflow')()) File "/home/jp/miniconda3/envs/gfla/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 829, in call return self.main(args, kwargs) File "/home/jp/miniconda3/envs/gfla/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 782, in main rv = self.invoke(ctx) File "/home/jp/miniconda3/envs/gfla/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/jp/miniconda3/envs/gfla/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/jp/miniconda3/envs/gfla/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 610, in invoke return callback(args, **kwargs) File "/home/jp/miniconda3/envs/gfla/lib/python3.6/site-packages/simpleflow-0.21.13-py3.6.egg/simpleflow/command.py", line 201, in start_workflow workflow_type = get_workflow_type(domain, workflow_class) File "/home/jp/miniconda3/envs/gfla/lib/python3.6/site-packages/simpleflow-0.21.13-py3.6.egg/simpleflow/command.py", line 104, in get_workflow_type return query.get_or_create(workflow_class.name, workflow_class.version) File "/home/jp/miniconda3/envs/gfla/lib/python3.6/site-packages/simpleflow-0.21.13-py3.6.egg/swf/querysets/workflow.py", line 242, in get_or_create description=description, File "/home/jp/miniconda3/envs/gfla/lib/python3.6/site-packages/simpleflow-0.21.13-py3.6.egg/swf/querysets/workflow.py", line 383, in create workflow_type.save() File "/home/jp/miniconda3/envs/gfla/lib/python3.6/site-packages/simpleflow-0.21.13-py3.6.egg/swf/models/workflow.py", line 206, in save raise DoesNotExistError(e.body['message']) swf.exceptions.DoesNotExistError: Unknown domain: TestDomain

How is it running?

$ simpl^C (gfla) ➜ simpleflow git:(master) simpleflow --help

Usage: simpleflow [OPTIONS] COMMAND [ARGS]...

dauving commented 4 years ago

Indeed, the documentation is not clear on this subject, you need to have an existing SWF domain for your tests, named "testDomain" in our demo.

If you have awscli installed, it's aws swf register-domain --name testDomain --workflow-execution-retention-period-in-days 1 ---region us-east-1 (change the region as you wish, SWF domain are scoped to the region). Or you can use the "manage domains" button at the top right on the SWF Console.

8secz-johndpope commented 4 years ago

I did manually go in and make this -

this other repo I got further along with https://github.com/MuminovicAnel/aws-swf-python-demo

it seems boto can handle the long polling - does it supercede this codebase? or am I missing something.

import boto3 from botocore.exceptions import ClientError from config import SWFConfig

swf = boto3.client('swf')

Register a workflow

try: swf.register_workflow_type( domain = SWFConfig.DOMAIN, name = SWFConfig.DOMAIN, version = SWFConfig.WORKFLOW_VERSION, description = 'Copie users.csv sur son bucket S3', defaultTaskStartToCloseTimeout=str(3600), defaultExecutionStartToCloseTimeout=str(24*3600), defaultChildPolicy = "TERMINATE", defaultTaskList={ 'name': SWFConfig.TASK_LIST_NAME }, ) print(f"{SWFConfig.DOMAIN} created !") except ClientError as e: print('Error workflow registration: ', e.response.get('Error', {}).get('Code'))

Register an activity type

try: for activity_type in SWFConfig.ACTIVITY_LIST: swf.register_activity_type( domain=SWFConfig.DOMAIN, name=activity_type['name'], version=SWFConfig.WORKFLOW_VERSION, description=f"Activity type {activity_type['version']}", defaultTaskHeartbeatTimeout="900", defaultTaskScheduleToStartTimeout="120", defaultTaskScheduleToCloseTimeout="3800", defaultTaskStartToCloseTimeout="3600", defaultTaskList={"name": SWFConfig.TASK_LIST_NAME } ) print("Worker created") except ClientError as e: print('Error in registering activity: ', e.response.get('Error', {}).get('Code'))