search

flyteorg / flyte

Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
https://flyte.org
Apache License 2.0
5.62k stars 620 forks source link

[BUG] flytekit runs the user code during registration #5586

Open pingsutw opened 2 months ago

pingsutw commented 2 months ago

Describe the bug

(flyte-conformance) ➜  flyte-conformance git:(actor-in-dynamic) pyflyte -vv run --remote workflow/integration_tests.py my_spark
16:18:55.724615 INFO     __init__.py:67 - We won't register bigquery handler for structured dataset because we can't find the packages google-cloud-bigquery-storage and google-cloud-bigquery            
16:18:55.791410 WARNING  type_engine.py:1343 - Unsupported Type typing.Any found, Flyte will default to use PickleFile as the transport. Pickle can only be used to send objects between the exact same   
                         version of Python, and we strongly recommend to use python type that flyte support.                                                                                              
16:18:56.037417 INFO     utils.py:340 - Translate literal to python value. [Time: 0.000027s]                                                                                                              
16:18:56.037968 INFO     base_task.py:731 - Invoking flytesnacks.examples.basics.basics.workflow.slope with inputs: {'x': [-3, 0, 3], 'y': [7, 4, -2]}                                                    
16:18:56.038488 INFO     utils.py:340 - Execute user level code. [Time: 0.000013s]                                                                                                                        
16:18:56.038837 INFO     utils.py:340 - Translate the output to literals. [Time: 0.000013s]                                                                                                               
16:18:56.039543 INFO     utils.py:340 - Translate literal to python value. [Time: 0.000020s]                                                                                                              
16:18:56.039843 INFO     base_task.py:731 - Invoking flytesnacks.examples.basics.basics.workflow.intercept with inputs: {'x': [-3, 0, 3], 'y': [7, 4, -2], 'slope': -1.5}                                 
16:18:56.040302 INFO     utils.py:340 - Execute user level code. [Time: 0.000009s]                                                                                                                        
16:18:56.040634 INFO     utils.py:340 - Translate the output to literals. [Time: 0.000011s]                                                                                                               
16:18:56.041532 INFO     utils.py:340 - Translate literal to python value. [Time: 0.000017s]                                                                                                              
16:18:56.041850 INFO     base_task.py:731 - Invoking flytesnacks.examples.basics.basics.workflow.slope with inputs: {'x': [-3, 0, 3], 'y': [7, 4, -2]}                                                    
16:18:56.042240 INFO     utils.py:340 - Execute user level code. [Time: 0.000012s]

Expected behavior

should not run the user code

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

Have you read the Code of Conduct?

runllm[bot] commented 2 months ago

The issue you're experiencing seems to be related to the behavior of Flyte during the registration process. According to the Flyte documentation, the registration process involves uploading the packaged (serialized) code to the Flyte backend, enabling the workflow to be triggered. However, the user code should not be executed during this process.

Would you like more details on how to properly register workflows and tasks without executing the user code?

Sources:

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @runllm to ask follow-up questions. See runllm.com for more!