search

yukinko-iwasaki / CS744_CourseProject

0 stars 0 forks source link

Hadoop error: Cannot set priority of datanode process #6

Closed GindaChen closed 3 years ago

GindaChen commented 3 years ago

I am stuck with this error:

start-dfs.sh

will throw

WARNING: Use of this script to start HDFS daemons is deprecated.
WARNING: Attempting to execute replacement "hdfs --daemon start" instead.
ERROR: Cannot set priority of datanode process 6199

with the log

10:02:39 root@node0 logs → cat privileged-root-datanode-node0.xgboost.uwmadison744-f20-pg0.wisc.cloudlab.us.out
JSVC_HOME is not set or set incorrectly. jsvc is required to run secure
or privileged daemons. Please download and install jsvc from
http://archive.apache.org/dist/commons/daemon/binaries/
and set JSVC_HOME to the directory containing the jsvc binary.
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 768131
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 768131
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
GindaChen commented 3 years ago

After some serious digging, I found some knobes that could be the potential cause:

  1. Hadoop version vs Google protobuf message version. See the comments in question of this post. The author switches back to hadoop 2.7 and the things is solved.
  2. pyarrow version. Using HDFS is successful, but pyarrow remains a problem. This could be caused by hdfs library used by pyarrow.
GindaChen commented 3 years ago

See #5. Indeed, it is the version unmatch. No where in the doc has explain this prior.

IapHubery commented 2 years ago

Recently，I had encountered with the similar problem, and pls tell me now what shall I do to take advantage of this problem? My hadoop version is 3.1.2

GindaChen commented 2 years ago

@IapHubery - I may have out-dated memory about this hadoop setup problem. I vaguely remember I solve this by checking all the installed packages share the same hadoop version (2.10.1). See this comment.

If in your project there is no specific hadoop version you must use, perhaps the best way to approach is to switch to Hadoop 2.10.1?