vincentclaes / datajob

Build and deploy a serverless data pipeline on AWS with no effort.
https://pypi.org/project/datajob/
Apache License 2.0
110 stars 19 forks source link

RecursionError: maximum recursion depth exceeded #98

Closed ivankeller closed 3 years ago

ivankeller commented 3 years ago

Last changes in main branch broke our test in Mailswitch:

======================================================================
--
ERROR: test_run_mailswitch_stack_without_errors (test_mailswitch_stack.TestMailswitchStack)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/codebuild/output/src937238439/src/github.axa.com/ds4a/mailswitch/mailswitch_stack.py", line 320, in <module>
filter_qa_labels_from_all_labels >> join_labels >> group_small_classes >> split_train_test >> preprocess_dataset >> train_model_step
File "/codebuild/output/src937238439/src/github.axa.com/ds4a/datajob/datajob/stepfunctions/stepfunctions_workflow.py", line 228, in __exit__
self._build_workflow()
File "/codebuild/output/src937238439/src/github.axa.com/ds4a/datajob/datajob/stepfunctions/stepfunctions_workflow.py", line 139, in _build_workflow
self.chain_of_tasks = self._construct_toposorted_chain_of_tasks()
File "/codebuild/output/src937238439/src/github.axa.com/ds4a/datajob/datajob/stepfunctions/stepfunctions_workflow.py", line 129, in _construct_toposorted_chain_of_tasks
sfn_task = self.add_parallel_tasks(element)
File "/codebuild/output/src937238439/src/github.axa.com/ds4a/datajob/datajob/stepfunctions/stepfunctions_workflow.py", line 90, in add_parallel_tasks
logger.debug(f"adding parallel task {a_task}")
File "/codebuild/output/src937238439/src/github.axa.com/ds4a/datajob/datajob/datajob_base.py", line 73, in __str__
return f"{self}"
File "/codebuild/output/src937238439/src/github.axa.com/ds4a/datajob/datajob/datajob_base.py", line 73, in __str__
return f"{self}"
File "/codebuild/output/src937238439/src/github.axa.com/ds4a/datajob/datajob/datajob_base.py", line 73, in __str__
return f"{self}"
[Previous line repeated 585 more times]
RecursionError: maximum recursion depth exceeded

For the moment we'll overcome this by cloning at a certain commit. Then we'll see how we manage datajob versions. Do you plan to publish releases to PyPI?

ivankeller commented 3 years ago

Actually I see it's on PyPI. We'll use it

vincentclaes commented 3 years ago

@ivankeller hmm interesting not sure how I have to reproduce this

vincentclaes commented 3 years ago

ok got it. @ivankeller can you pull branch hotfix-print-str and check if this issue is resolved?

ivankeller commented 3 years ago

Hi Vincent! Thank you for the quick reaction! I'm just leaving on holiday now but I'll forward to Lorenzo. See you in August! I.

On Thu, 15 Jul 2021 at 15:15, Vincent Claes @.***> wrote:

ok got it. @ivankeller https://github.com/ivankeller can you pull branch hotfix-print-str and check if this issue is resolved?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/vincentclaes/datajob/issues/98#issuecomment-880685763, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB6EJWRRRKKRWAUV6ASOLZTTX3NQ7ANCNFSM5ALWXXTA .