search

Azure / MachineLearningNotebooks

Python notebooks with ML and deep learning examples with Azure Machine Learning Python SDK | Microsoft
https://docs.microsoft.com/azure/machine-learning/service/
MIT License
4k stars 2.49k forks source link

ParallelRunStep is failing with SystemExit: 42 #1930

Closed umangkhare closed 10 months ago

umangkhare commented 11 months ago

I'm trying to run ParallelRunStep:

parallel_run_config = ParallelRunConfig(
   source_directory=experiment_folder,
   entry_script='training_file.py',
   mini_batch_size="1KB",
   error_threshold=10,         
   output_action="append_row",
   environment=registered_env,
   compute_target=pipeline_cluster,
   node_count=2,
   process_count_per_node=4)

train_step = ParallelRunStep(
    name="Paralle Run step",
    parallel_run_config=parallel_run_config,
    inputs = [data_dataset.as_named_input('inp_data')],
    output = output_dir,
    allow_reuse=False)

But getting following error:

Traceback (most recent call last):
  File "driver/amlbi_main.py", line 275, in <module>
    main()
  File "driver/amlbi_main.py", line 226, in main
    sys.exit(exitcode_candidate)
SystemExit: 42

Can anyone please help me with this issue?

juichiache commented 11 months ago

I have the same issue. Inference was working a couple months back. Check the log files in output+logs -> user -> stdout -> 1 -> process000.std.txt for detailed error.

LIK2RNG commented 10 months ago

I'm trying to run ParallelRunStep:

parallel_run_config = ParallelRunConfig(
   source_directory=experiment_folder,
   entry_script='training_file.py',
   mini_batch_size="1KB",
   error_threshold=10,         
   output_action="append_row",
   environment=registered_env,
   compute_target=pipeline_cluster,
   node_count=2,
   process_count_per_node=4)

train_step = ParallelRunStep(
    name="Paralle Run step",
    parallel_run_config=parallel_run_config,
    inputs = [data_dataset.as_named_input('inp_data')],
    output = output_dir,
    allow_reuse=False)

But getting following error:

Traceback (most recent call last):
  File "driver/amlbi_main.py", line 275, in <module>
    main()
  File "driver/amlbi_main.py", line 226, in main
    sys.exit(exitcode_candidate)
SystemExit: 42

Can anyone please help me with this issue?

@umangkhare @juichiache i met the same problem. Eventually i could find the error inside output+logs --> logs/, inside the folder there're joberror_xxx.txt & jobresult.txt, which contain the real error message

umangkhare commented 10 months ago

I think this is not an error which results in failing the pipeline. It eventually results in output logs every time when ParallelRunStep is used. I tried after correcting my other problems in code, and it worked out. Thanks for the support.

Meenakshi-subramanian commented 8 months ago

Hi, I am facing the same issue. Kindly help where should we look at to resolve this.

LIK2RNG commented 8 months ago

Hi, I am facing the same issue. Kindly help where should we look at to resolve this.

same for me. Parallel jobs did not start, indeed it was pending for some time and then died. However, the job was shown as Successful eventually

Meenakshi-subramanian commented 8 months ago

How can I see the real error? You have any idea?

umangkhare commented 8 months ago

ParallelRunStep doesn't show the outputs like usual PythonRunStep. I'd suggest to read the file 'readme.txt' under logs folder to understand how the logging works in ParallelRunStep. It's somewhat tricky to read these logs here. Also, you should consider analyzing CPU utilization and memory usage under 'Monitoring' of the step to get a better understanding. You will always get 'SystemExit: 42' error irrespective of your pipeline success or failure. You should try tuning the parameters- 'no of processes' and 'mini_batch_size' to run your pipeline properly. Note- It may take you some time, just have patience.😁