can't upload files

[xxxxsk]

When I was deploying my project， I can't import func from my *.py, and there is a FileNotFoundError after client.upload_file('')

The directory structure dask_test

• run.sh
• test.py
• upload_file.py
• run.sh:

  /opt/anaconda3/envs/pymodel/bin/dask-yarn submit \
  --environment python:///opt/anaconda3/envs/pymodel/bin/python \
  --worker-count 5 \
  --worker-vcores 2 \
  --worker-memory 4GiB \
  test.py

test.py:

from dask_yarn import YarnCluster
from dask.distributed import Client
import time
import pandas as pd
import os

cluster = YarnCluster.from_current()
client = Client(cluster)

def square(x):
    return x ** 2
ts = time.time()
client.upload_file('upload_file.py')
A = client.map(square, range(5))
total = client.submit(sum, A)
print(total.result())
print('1cost time :%s'%(time.time()-ts))
client.close()
cluster.close()

I want to konw how to correctly upload file and import ?

Environment:

• Dask version:2.11.0
• Python version:3.6.10
• Install method (conda, pip, source):pip

[quasiben]

I don't think dask-yarn can submit both a dependent file and the script to run. I'd recommend bundling all of scripts into one file. Alternatively, you could install the library in an environment to be bundled with the application: https://yarn.dask.org/en/latest/environments.html#managing-python-environments

[xxxxsk]

I upload my files to hdfs and it works