search

dask / knit

Deprecated, please use https://github.com/jcrist/skein or https://github.com/dask/dask-yarn instead
http://knit.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
53 stars 10 forks source link

Locale error on application start #123

Closed j-bennet closed 6 years ago

j-bennet commented 6 years ago

I'm trying to run a Dask application on YARN and get a setlocale error on container start:

18/04/14 21:38:43 INFO knit.ApplicationMaster$: Running command in container: dvss.zip/dvss/bin/python dvss.zip/dvss/bin/dask-worker --nprocs=1 --nthreads=5 --memory-limit=500000000.0 --no-bokeh tcp://10.21.0.219:42867 
18/04/14 21:38:43 INFO knit.ApplicationMaster$: Setting container language to 'C.UTF-8'
18/04/14 21:38:43 INFO knit.ApplicationMaster$: RESOURCE: hdfs:/user/hadoop/.knitDeps/dvss.zip archive=true
18/04/14 21:38:43 INFO knit.ApplicationMaster$: TOKEN ADDED: Kind: YARN_AM_RM_TOKEN, Service: 10.21.0.219:8030, Ident: (appAttemptId { application_id { id: 2 cluster_timestamp: 1523740467851 } attemptId: 1 } keyId: 1185419776)
18/04/14 21:38:43 WARN token.Token: Cannot find class for token kind kms-dt
18/04/14 21:38:43 INFO knit.ApplicationMaster$: TOKEN ADDED: Kind: kms-dt, Service: 10.21.0.219:9700, Ident: 00 06 68 61 64 6f 6f 70 04 79 61 72 6e 00 8a 01 62 c6 18 78 1d 8a 01 62 ea 24 fc 1d 02 02
18/04/14 21:38:43 INFO knit.ApplicationMaster$: Launching container Container: [ContainerId: container_1523740467851_0002_01_000004, NodeId: ip-10-21-0-244.ec2.internal:8041, NodeHttpAddress: ip-10-21-0-244.ec2.internal:8042, Resource: <memory:512, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.21.0.244:8041 }, ]
18/04/14 21:38:43 INFO impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1523740467851_0002_01_000004
18/04/14 21:38:43 INFO impl.ContainerManagementProtocolProxy: Opening proxy : ip-10-21-0-244.ec2.internal:8041
18/04/14 21:38:43 INFO knit.ApplicationMaster$: Container started container_1523740467851_0002_01_000004
18/04/14 21:38:44 INFO knit.ApplicationMaster$: User: Some(hadoop) StagingDir: /user/hadoop/.knitDeps
18/04/14 21:38:44 INFO knit.ApplicationMaster$: Running command in container: dvss.zip/dvss/bin/python dvss.zip/dvss/bin/dask-worker --nprocs=1 --nthreads=5 --memory-limit=500000000.0 --no-bokeh tcp://10.21.0.219:42867 
18/04/14 21:38:45 INFO knit.ApplicationMaster$: Container completed ContainerStatus: [ContainerId: container_1523740467851_0002_01_000005, State: COMPLETE, Diagnostics: Container released by application, ExitStatus: -100, ].getContainerId
18/04/14 21:38:54 INFO knit.ApplicationMaster$: Container completed ContainerStatus: [ContainerId: container_1523740467851_0002_01_000003, State: COMPLETE, Diagnostics: Exception from container-launch.
Container id: container_1523740467851_0002_01_000003
Exit code: 1
Exception message: bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
/mnt/yarn/usercache/hadoop/appcache/application_1523740467851_0002/container_1523740467851_0002_01_000003/launch_container.sh: line 17: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
/mnt/yarn/usercache/hadoop/appcache/application_1523740467851_0002/container_1523740467851_0002_01_000003/launch_container.sh: line 17: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)

Stack trace: ExitCodeException exitCode=1: bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
/mnt/yarn/usercache/hadoop/appcache/application_1523740467851_0002/container_1523740467851_0002_01_000003/launch_container.sh: line 17: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
/mnt/yarn/usercache/hadoop/appcache/application_1523740467851_0002/container_1523740467851_0002_01_000003/launch_container.sh: line 17: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)

    at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
    at org.apache.hadoop.util.Shell.run(Shell.java:479)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Container exited with a non-zero exit code 1
, ExitStatus: 1, ].getContainerId

Just for reference, what I'm running is a tiny test application that reads some parquet and prints out a few rows. Full code:

import dask.dataframe as dd
from dask.distributed import Client
from dask_yarn import DaskYARNCluster

if __name__ == '__main__':
    cluster = DaskYARNCluster(env='/home/hadoop/reqs/dvss.zip')
    cluster.start(n_workers=3, memory=500, cpus=5, checks=True)
    client = Client(address=cluster)
    print('-' * 20)
    print(cluster.knit.conf)
    read_path = 's3://parsely-public/jbennet/daskvsspark/events/100-24/*/*/*/*/*/*.parquet'
    df = dd.read_parquet(read_path)
    print('-' * 20)
    print(df.head(3))
martindurant commented 6 years ago

The keyword lang= exists for this issue. Python requires the shell language to be set, but in some systems, apparently yours, the language is already set with an incompatible value. I mean the language of the shell within which processes are executed by yarn. You should find from the configuration or sysadmin what the right language to set is.

(this is a strange situation, because all shells should have a language set, so this mechanism should not be necessary, and all shells should allow their language to be set to the universal UTF8, but unfortunately this is not the case)

j-bennet commented 6 years ago

Setting cluster = DaskYARNCluster(env='/home/hadoop/reqs/dvss.zip', lang='en_US.UTF-8') helped. Thank you.

martindurant commented 6 years ago

Always glad to see successful uses of knit. You may want to join us on https://gitter.im/dask/dask to describe your experience.