search

gitava / Big-Data-Study

For studying big data
2 stars 0 forks source link

hadoop - hive job failure during insert. #21

Closed gitava closed 4 years ago

gitava commented 4 years ago 
hive> insert into htest values("xiaoxu","20");
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. tez, spark) or using Hive 1.X releases.
Query ID = vagrant_20200628115827_9ac7ffe6-f4da-4c22-ba9c-2bd2922b78ef
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1593342618576_0004, Tracking URL = http://hdp-node-01:8088/proxy/application_1593342618576_0004/
Kill Command = /home/vagrant/hadoop/bin/hadoop job  -kill job_1593342618576_0004
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2020-06-28 11:58:44,235 Stage-1 map = 0%,  reduce = 0%
2020-06-28 11:59:44,618 Stage-1 map = 0%,  reduce = 0%
2020-06-28 11:59:57,012 Stage-1 map = 100%,  reduce = 0%
Ended Job = job_1593342618576_0004 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1593342618576_0004_m_000000 (and more) from job job_1593342618576_0004

Task with the most failures(4):
-----
Task ID:
  task_1593342618576_0004_m_000000

URL:
  http://hdp-node-01:8088/taskdetails.jsp?jobid=job_1593342618576_0004&tipid=task_1593342618576_0004_m_000000
-----
Diagnostic Messages for this Task:

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
hive>
gitava commented 4 years ago

check http://192.168.33.101:8088/cluster for job error info.

gitava commented 4 years ago

http://192.168.33.102:8042/node/containerlogs/container_1593342618576_0004_01_000001/root

host name won't work since mac doesn't have dns info in hosts file.

gitava commented 4 years ago

http://192.168.33.102:8042/node/containerlogs/container_1593342618576_0004_01_000001/root/stderr/?start=-4096


Logs for container_1593342618576_0004_01_000001
--
 
ResourceManagerNodeManagerToolsConfigurationLocal logsServer stacksServer metrics | Jun 28, 2020 11:58:41 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver as a provider class Jun 28, 2020 11:58:41 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class Jun 28, 2020 11:58:41 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices as a root resource class Jun 28, 2020 11:58:41 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM' Jun 28, 2020 11:58:41 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton" Jun 28, 2020 11:58:42 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton" Jun 28, 2020 11:58:42 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices to GuiceManagedComponentProvider with the scope "PerRequest" log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
gitava commented 4 years ago

http://192.168.33.102:8042/node/containerlogs/container_1593342618576_0004_01_000001/root/syslog/?start=-4096

Showing 4096 bytes. Click here for full log

/hdp-node-01:9000/tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1593342618576_0004_conf.xml_tmp
2020-06-28 11:59:56,768 INFO [Thread-71] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://hdp-node-01:9000/tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1593342618576_0004.summary_tmp to hdfs://hdp-node-01:9000/tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1593342618576_0004.summary
2020-06-28 11:59:56,771 INFO [Thread-71] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://hdp-node-01:9000/tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1593342618576_0004_conf.xml_tmp to hdfs://hdp-node-01:9000/tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1593342618576_0004_conf.xml
2020-06-28 11:59:56,774 INFO [Thread-71] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://hdp-node-01:9000/tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1593342618576_0004-1593345513414-root-insert+into+htest+values%28%22xiaoxu%22%2C%2220%22-1593345596556-0-0-FAILED-default-1593345522894.jhist_tmp to hdfs://hdp-node-01:9000/tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1593342618576_0004-1593345513414-root-insert+into+htest+values%28%22xiaoxu%22%2C%2220%22-1593345596556-0-0-FAILED-default-1593345522894.jhist
2020-06-28 11:59:56,774 INFO [Thread-71] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop()
2020-06-28 11:59:56,775 INFO [Thread-71] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Setting job diagnostics to Task failed task_1593342618576_0004_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

2020-06-28 11:59:56,775 INFO [Thread-71] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: History url is http://hdp-node-02:19888/jobhistory/job/job_1593342618576_0004
2020-06-28 11:59:56,805 INFO [Thread-71] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Waiting for application to be successfully unregistered.
2020-06-28 11:59:57,809 INFO [Thread-71] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:0 RackLocal:1
2020-06-28 11:59:57,810 INFO [Thread-71] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://hdp-node-01:9000 /tmp/hadoop-yarn/staging/root/.staging/job_1593342618576_0004
2020-06-28 11:59:57,815 INFO [Thread-71] org.apache.hadoop.ipc.Server: Stopping server on 41269
2020-06-28 11:59:57,818 INFO [IPC Server listener on 41269] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 41269
2020-06-28 11:59:57,818 INFO [Ping Checker] org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: TaskAttemptFinishingMonitor thread interrupted
2020-06-28 11:59:57,818 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted
2020-06-28 11:59:57,818 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2020-06-28 12:00:02,820 INFO [Thread-71] org.apache.hadoop.ipc.Server: Stopping server on 36944
2020-06-28 12:00:02,831 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2020-06-28 12:00:02,838 INFO [IPC Server listener on 36944] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 36944
2020-06-28 12:00:02,855 INFO [Thread-71] org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:0
2020-06-28 12:00:02,858 WARN [489945456@qtp-976213793-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:43837] org.apache.hadoop.http.HttpServer2: HttpServer Acceptor: isRunning is false. Rechecking.
2020-06-28 12:00:02,858 WARN [489945456@qtp-976213793-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:43837] org.apache.hadoop.http.HttpServer2: HttpServer Acceptor: isRunning is false
gitava commented 4 years ago

task details http://192.168.33.101:19888/jobhistory/task/task_1593342618576_0004_m_000000

image

gitava commented 4 years ago

http://192.168.33.101:19888/jobhistory/logs/hdp-node-03:40007/container_1593342618576_0004_01_000002/attempt_1593342618576_0004_m_000000_0/root

Aggregation is not enabled. Try the nodemanager at hdp-node-03:40007
Or see application log at http://hdp-node-03:40007/node/application/application_1593342618576_0004
gitava commented 4 years ago

as per https://blog.csdn.net/knowledgeaaa/article/details/30253505

<property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
</property>

into hadoop/etc/hadoop/yarn-site.xml to enable log aggregation.

gitava commented 4 years ago

This issue fixed but a new error coming up

hive> insert into htest values("xiaoxu","20");
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. tez, spark) or using Hive 1.X releases.
Query ID = vagrant_20200628124140_7c6fc8f5-49fe-4c2c-9ec9-0a7482a369e0
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1593348026195_0001, Tracking URL = http://hdp-node-01:8088/proxy/application_1593348026195_0001/
Kill Command = /home/vagrant/hadoop/bin/hadoop job  -kill job_1593348026195_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2020-06-28 12:42:06,383 Stage-1 map = 0%,  reduce = 0%
2020-06-28 12:42:17,501 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 1.21 sec
MapReduce Total cumulative CPU time: 1 seconds 210 msec
Ended Job = job_1593348026195_0001
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to directory hdfs://hdp-node-01:9000/user/hive/warehouse/hdb.db/htest/.hive-staging_hive_2020-06-28_12-41-40_167_1335969122270533854-1/-ext-10000
Loading data to table hdb.htest
FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.StatsTask. PermGen space
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1   Cumulative CPU: 1.21 sec   HDFS Read: 4078 HDFS Write: 75 SUCCESS
Total MapReduce CPU Time Spent: 1 seconds 210 msec
hive> select * from htest;
Exception in thread "main" java.lang.OutOfMemoryError: PermGen space
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at org.apache.hadoop.hive.common.FileUtils.deleteDirectory(FileUtils.java:778)
    at org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1564)
    at org.apache.hadoop.hive.cli.CliSessionState.close(CliSessionState.java:66)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:762)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:244)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:158)
Exception in thread "Thread-1" java.lang.OutOfMemoryError: PermGen space
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at org.apache.hadoop.util.ShutdownHookManager.getShutdownHooksInOrder(ShutdownHookManager.java:273)
    at org.apache.hadoop.util.ShutdownHookManager.executeShutdown(ShutdownHookManager.java:121)
    at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:95)
[vagrant@hdp-node-01 ~]$
gitava commented 4 years ago

yarn.xml

<?xml version="1.0"?>
<!-- Licensed under the Apache License, Version 2.0 (the "License"); you 
    may not use this file except in compliance with the License. You may obtain 
    a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless 
    required by applicable law or agreed to in writing, software distributed 
    under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES 
    OR CONDITIONS OF ANY KIND, either express or implied. See the License for 
    the specific language governing permissions and limitations under the License. 
    See accompanying LICENSE file. -->
<configuration>

    <!-- Site specific YARN configuration properties -->
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hdp-node-01</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>192.168.33.101:8088</value>
    </property>
    <!-- reducer取数据的方式是mapreduce_shuffle -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <!-- fix to error : Aggregation is not enabled -->
    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property> 
</configuration>