Yarn Cluster

[hamroune]

Hi everyone, I'm trying to execute a small piece of code on Yarn, there is my whole code:

`

with YarnCluster(environment='environment.tar.gz',
                      worker_vcores=2,
                      worker_memory="8GiB") as cluster:

                cluster.scale(2)
                # Connect to the cluster
                client = Client(cluster)

                CSV_INPUT = 'hdfs:///Data/input/input.csv'
                CSV_OUTPUT = 'hdfs://Data/input/output.parquet'

                df = dd.read_csv(CSV_INPUT)
                dd.to_parquet(df, CSV_OUTPUT)

`

unfortunely i have this error

`

distributed.scheduler.KilledWorker: ("('pandas_read_text-read-block-from-delayed-3f488119df76d5b2ba0e2e75ec2bc55b', 0)", <Worker 'tcp://myip:40417', memory: 0, processing: 1>)

` the error is not understandable for me, have any one experienced this issue? thanks

[jcrist]

That error indicates you have a worker that keeps dieing. This could be due to a few things:

• Your task is using too much memory, resulting in it being killed
• Your task may crash the worker. What hdfs library (pyarrow is default, falling back to hdfs3 if pyarrow isn't installed) are you using?

Usually when confronted with an error in a dask-yarn application, the first thing to do is look at the application logs. You can get these by

$ yarn logs -applicationId <application-id>

[jcrist]

Closing due to inactivity. Feel free to reopen if you can provide more information.