search

cloudera / cdh-twitter-example

Example application for analyzing Twitter data using CDH - Flume, Oozie, Hive
288 stars 343 forks source link

Issues with the json serde. #1

Open mortenbpost opened 12 years ago

mortenbpost commented 12 years ago

Hi,

I'm using the json serde in hive for parsing another set of json files I have from valentines day. I noticed that there is no option to ignore malformed json, and there seems to be some problems with deserializing all json.

This tweet is causing the error:

{"text":"@KimKardashian happy valentines day, hope it's a good one","retweet_count":0,"geo":{"type":"Point","coordinates":[38.7313358,-108.05278695]},"in_reply_to_status_id_str":null,"in_reply_to_user_id":25365536,"source":"\u003Ca href=\"http:\/\/twitter.com\/download\/android\" rel=\"nofollow\"\u003ETwitter for Android\u003C\/a\u003E","in_reply_to_user_id_str":"25365536","id_str":"169483808003989505","entities":{"user_mentions":[{"indices":[0,14],"screen_name":"KimKardashian","id_str":"25365536","name":"Kim Kardashian","id":25365536}],"urls":[],"hashtags":[]},"in_reply_to_status_id":null,"place":{"url":"http:\/\/api.twitter.com\/1\/geo\/id\/6a7e7dbf9d6c7ac4.json","place_type":"city","country_code":"US","attributes":{},"full_name":"Delta, CO","bounding_box":{"type":"Polygon","coordinates":[[[-108.104644,38.71503],[-108.021863,38.71503],[-108.021863,38.769794],[-108.104644,38.769794]]]},"name":"Delta","id":"6a7e7dbf9d6c7ac4","country":"United States"},"in_reply_to_screen_name":"Ki{"text":"@bbrandivirgo too bad I dont have the number. Happy valentines day tho :)","retweet_count":0,"geo":{"type":"Point","coordinates":[33.77406404,-84.39270512]},"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"source":"\u003Ca href=\"http:\/\/mobile.twitter.com\" rel=\"nofollow\"\u003EMobile Web\u003C\/a\u003E","in_reply_to_user_id_str":null,"id_str":"169497701241716736","entities":{"user_mentions":[],"urls":[],"hashtags":[]},"in_reply_to_status_id":null,"place":{"url":"http:\/\/api.twitter.com\/1\/geo\/id\/8173485c72e78ca5.json","place_type":"city","country_code":"US","attributes":{},"full_name":"Atlanta, GA","bounding_box":{"type":"Polygon","coordinates":[[[-84.54674,33.647908],[-84.289389,33.647908],[-84.289389,33.887618],[-84.54674,33.887618]]]},"name":"Atlanta","id":"8173485c72e78ca5","country":"United States"},"in_reply_to_screen_name":null,"favorited":false,"truncated":false,"created_at":"Tue Feb 14 19:06:15 +0000 2012","contributors":null,"user":{"contributors_enabled":false,"profile_background_image_url":"http:\/\/a3.twimg.com\/profile_background_images\/376284279\/yyyyyyyyyyyyyyyyyyyy.jpg","url":"http:\/\/facebook.com\/cperk3","profile_link_color":"0084B4","followers_count":773,"profile_image_url":"http:\/\/a3.twimg.com\/profile_images\/1792490671\/000011110000_normal.jpg","default_profile_image":false,"show_all_inline_media":true,"statuses_count":3271,"profile_background_color":"C0DEED","description":"Ga Tech Athlete-Student.. Black&Samoan...Follow me as I follow Jesus-","location":"Atlanta, GA","profile_background_tile":true,"favourites_count":1,"profile_background_image_url_https":"https:\/\/si0.twimg.com\/profile_background_images\/376284279\/yyyyyyyyyyyyyyyyyyyy.jpg","time_zone":"Quito","profile_sidebar_fill_color":"DDEEF6","screen_name":"Cpeezy21","id_str":"312682111","lang":"en","geo_enabled":true,"profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/1792490671\/000011110000_normal.jpg","verified":false,"notifications":null,"profile_sidebar_border_color":"04080a","protected":false,"listed_count":5,"created_at":"Tue Jun 07 14:14:34 +0000 2011","name":"Charles Perkins III","is_translator":false,"follow_request_sent":null,"following":null,"profile_use_background_image":true,"friends_count":223,"id":312682111,"default_profile":false,"utc_offset":-18000,"profile_text_color":"333333"},"retweeted":false,"id":169497701241716736,"coordinates":{"type":"Point","coordinates":[-84.39270512,33.77406404]}}

I'm getting this error when processing some sample twitter data:

2012-09-26 15:15:39,059 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 2012-09-26 15:15:39,215 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library 2012-09-26 15:15:39,372 INFO org.apache.hadoop.mapred.TaskRunner: Creating symlink: /mapred/local/taskTracker/distcache/-624804405132306423_-2027207125_45603557/hadoop1.domain.com/tmp/hive-root/hive_2012-09-26_15-15-33_715_8669028640552125101/-mr-10004/af319f96-99f0-4f06-8fba-3fbf5b880148 <- /mapred/local/taskTracker/root/jobcache/job_201209252321_0010/attempt_201209252321_0010_m_000000_0/work/HIVE_PLANaf319f96-99f0-4f06-8fba-3fbf5b880148 2012-09-26 15:15:39,380 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /mapred/local/taskTracker/root/jobcache/job_201209252321_0010/jars/job.jar <- /mapred/local/taskTracker/root/jobcache/job_201209252321_0010/attempt_201209252321_0010_m_000000_0/work/job.jar 2012-09-26 15:15:39,388 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /mapred/local/taskTracker/root/jobcache/job_201209252321_0010/jars/.job.jar.crc <- /mapred/local/taskTracker/root/jobcache/job_201209252321_0010/attempt_201209252321_0010_m_000000_0/work/.job.jar.crc 2012-09-26 15:15:39,451 WARN org.apache.hadoop.conf.Configuration: session.id is deprecated. Instead, use dfs.metrics.session-id 2012-09-26 15:15:39,452 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId= 2012-09-26 15:15:39,767 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0 2012-09-26 15:15:39,773 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@484845aa 2012-09-26 15:15:40,065 WARN org.apache.hadoop.hive.conf.HiveConf: hive-site.xml not found on CLASSPATH 2012-09-26 15:15:40,222 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library is available 2012-09-26 15:15:40,222 INFO org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library loaded 2012-09-26 15:15:40,232 WARN mapreduce.Counters: Counter name MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group name and BYTES_READ as counter name instead 2012-09-26 15:15:40,236 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0 2012-09-26 15:15:40,242 INFO ExecMapper: maximum memory = 119341056 2012-09-26 15:15:40,243 INFO ExecMapper: conf classpath = [file:/var/run/cloudera-scm-agent/process/93-mapreduce-TASKTRACKER/, file:/usr/java/jdk1.6.0_31/lib/tools.jar, file:/usr/lib/hadoop-0.20-mapreduce/, file:/usr/lib/hadoop-0.20-mapreduce/hadoop-core-2.0.0-mr1-cdh4.0.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/activation-1.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/ant-contrib-1.0b3.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjrt-1.6.5.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjtools-1.6.5.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/avro-1.5.4.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/avro-compiler-1.5.4.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-1.7.0.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-core-1.8.0.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-cli-1.2.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-codec-1.4.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-collections-3.2.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-configuration-1.6.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-digester-1.8.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-el-1.0.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-httpclient-3.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-io-2.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-lang-2.5.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-logging-1.1.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-logging-api-1.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-math-2.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-net-3.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/core-3.1.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/guava-11.0.2.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/hadoop-fairscheduler-2.0.0-mr1-cdh4.0.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/hsqldb-1.8.0.10.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-core-asl-1.8.8.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-mapper-asl-1.8.8.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-xc-1.8.8.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-compiler-5.5.23.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-runtime-5.5.23.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-api-2.2.2.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-impl-2.2.3-1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jdiff-1.0.9.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-core-1.8.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-json-1.8.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-server-1.8.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jets3t-0.6.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jettison-1.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-6.1.26.cloudera.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-util-6.1.26.cloudera.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jsch-0.1.42.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/json-simple-1.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jsr305-1.3.9.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.2.2.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.3.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/log4j-1.2.16.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/oro-2.0.8.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/paranamer-2.3.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/protobuf-java-2.4.0a.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/servlet-api-2.5.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.6.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/snappy-java-1.0.3.2.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/stax-api-1.0.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/xmlenc-0.52.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-2.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-api-2.1.jar, file:/usr/share/cmf/lib/plugins/tt-instrumentation-4.0.4.jar, file:/usr/share/cmf/lib/plugins/event-publish-4.0.4-shaded.jar, file:/usr/lib/hadoop-hdfs/lib/avro-1.5.4.jar, file:/usr/lib/hadoop-hdfs/lib/paranamer-2.3.jar, file:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar, file:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar, file:/usr/lib/hadoop-hdfs/lib/slf4j-api-1.6.1.jar, file:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.4.0a.jar, file:/usr/lib/hadoop-hdfs/lib/snappy-java-1.0.3.2.jar, file:/usr/lib/hadoop-hdfs/lib/jline-0.9.94.jar, file:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.3.jar, file:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar, file:/usr/lib/hadoop-hdfs/lib/zookeeper-3.4.3-cdh4.0.1.jar, file:/usr/lib/hadoop-hdfs/lib/log4j-1.2.15.jar, file:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.0.1.jar, file:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.0.1.jar, file:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.0.1-tests.jar, file:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar, file:/usr/lib/hadoop/lib/commons-codec-1.4.jar, file:/usr/lib/hadoop/lib/jets3t-0.6.1.jar, file:/usr/lib/hadoop/lib/json-simple-1.1.jar, file:/usr/lib/hadoop/lib/guava-11.0.2.jar, file:/usr/lib/hadoop/lib/avro-1.5.4.jar, file:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar, file:/usr/lib/hadoop/lib/commons-configuration-1.6.jar, file:/usr/lib/hadoop/lib/asm-3.2.jar, file:/usr/lib/hadoop/lib/paranamer-2.3.jar, file:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar, file:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar, file:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar, file:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar, file:/usr/lib/hadoop/lib/commons-cli-1.2.jar, file:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.1.jar, file:/usr/lib/hadoop/lib/commons-lang-2.5.jar, file:/usr/lib/hadoop/lib/kfs-0.3.jar, file:/usr/lib/hadoop/lib/hue-plugins-2.0.0-cdh4.0.1.jar, file:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar, file:/usr/lib/hadoop/lib/jettison-1.1.jar, file:/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar, file:/usr/lib/hadoop/lib/jsch-0.1.42.jar, file:/usr/lib/hadoop/lib/stax-api-1.0.1.jar, file:/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar, file:/usr/lib/hadoop/lib/jsr305-1.3.9.jar, file:/usr/lib/hadoop/lib/snappy-java-1.0.3.2.jar, file:/usr/lib/hadoop/lib/jsp-api-2.1.jar, file:/usr/lib/hadoop/lib/oro-2.0.8.jar, file:/usr/lib/hadoop/lib/jersey-server-1.8.jar, file:/usr/lib/hadoop/lib/commons-digester-1.8.jar, file:/usr/lib/hadoop/lib/commons-math-2.1.jar, file:/usr/lib/hadoop/lib/jline-0.9.94.jar, file:/usr/lib/hadoop/lib/core-3.1.1.jar, file:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar, file:/usr/lib/hadoop/lib/commons-el-1.0.jar, file:/usr/lib/hadoop/lib/jersey-core-1.8.jar, file:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar, file:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar, file:/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.1.jar, file:/usr/lib/zookeeper/zookeeper-3.4.3-cdh4.0.1.jar, file:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar, file:/usr/lib/hadoop/lib/commons-net-3.1.jar, file:/usr/lib/hadoop/lib/servlet-api-2.5.jar, file:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar, file:/usr/lib/hadoop/lib/commons-io-2.1.jar, file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar, file:/usr/lib/hadoop/lib/commons-logging-api-1.1.jar, file:/usr/lib/hadoop/lib/xmlenc-0.52.jar, file:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar, file:/usr/lib/hadoop/lib/activation-1.1.jar, file:/usr/lib/hadoop/lib/jersey-json-1.8.jar, file:/usr/lib/hadoop/lib/aspectjrt-1.6.5.jar, file:/usr/lib/hadoop/lib/log4j-1.2.15.jar, file:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.0.1.jar, file:/usr/lib/hadoop/hadoop-auth-2.0.0-cdh4.0.1.jar, file:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.0.1.jar, file:/usr/lib/hadoop/hadoop-annotations-2.0.0-cdh4.0.1.jar, file:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.0.1-tests.jar, file:/usr/lib/hadoop/hadoop-annotations-2.0.0-cdh4.0.1.jar, file:/usr/lib/hadoop/hadoop-auth-2.0.0-cdh4.0.1.jar, file:/mapred/local/taskTracker/root/jobcache/job_201209252321_0010/jars/classes, file:/mapred/local/taskTracker/root/jobcache/job_2012092523210010/jars/job.jar, file:/mapred/local/taskTracker/root/distcache/4260026189093522549-70309741_45603944/hadoop1.domain.com/user/root/.staging/job_201209252321_0010/libjars/hive-builtins-0.8.1-cdh4.0.1.jar, file:/mapred/local/taskTracker/root/distcache/-6339710882011042599_2132445101_45603979/hadoop1.domain.com/user/root/.staging/job_2012092523210010/libjars/hive-serdes-1.0-SNAPSHOT.jar, file:/mapred/local/taskTracker/root/distcache/7269667103068590023-978189584_45604014/hadoop1.domain.com/user/root/.staging/job_201209252321_0010/libjars/hive-contrib-0.8.1-cdh4.0.1.jar, file:/mapred/local/taskTracker/root/jobcache/job_201209252321_0010/attempt_201209252321_0010_m_000000_0/work/] 2012-09-26 15:15:40,243 INFO ExecMapper: thread classpath = [file:/var/run/cloudera-scm-agent/process/93-mapreduce-TASKTRACKER/, file:/usr/java/jdk1.6.0_31/lib/tools.jar, file:/usr/lib/hadoop-0.20-mapreduce/, file:/usr/lib/hadoop-0.20-mapreduce/hadoop-core-2.0.0-mr1-cdh4.0.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/activation-1.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/ant-contrib-1.0b3.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjrt-1.6.5.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjtools-1.6.5.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/avro-1.5.4.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/avro-compiler-1.5.4.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-1.7.0.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-core-1.8.0.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-cli-1.2.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-codec-1.4.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-collections-3.2.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-configuration-1.6.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-digester-1.8.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-el-1.0.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-httpclient-3.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-io-2.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-lang-2.5.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-logging-1.1.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-logging-api-1.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-math-2.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/commons-net-3.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/core-3.1.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/guava-11.0.2.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/hadoop-fairscheduler-2.0.0-mr1-cdh4.0.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/hsqldb-1.8.0.10.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-core-asl-1.8.8.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-mapper-asl-1.8.8.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-xc-1.8.8.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-compiler-5.5.23.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-runtime-5.5.23.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-api-2.2.2.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-impl-2.2.3-1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jdiff-1.0.9.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-core-1.8.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-json-1.8.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-server-1.8.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jets3t-0.6.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jettison-1.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-6.1.26.cloudera.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-util-6.1.26.cloudera.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jsch-0.1.42.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/json-simple-1.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jsr305-1.3.9.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.2.2.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.3.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/log4j-1.2.16.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/oro-2.0.8.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/paranamer-2.3.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/protobuf-java-2.4.0a.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/servlet-api-2.5.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.6.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/snappy-java-1.0.3.2.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/stax-api-1.0.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/xmlenc-0.52.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-2.1.jar, file:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-api-2.1.jar, file:/usr/share/cmf/lib/plugins/tt-instrumentation-4.0.4.jar, file:/usr/share/cmf/lib/plugins/event-publish-4.0.4-shaded.jar, file:/usr/lib/hadoop-hdfs/lib/avro-1.5.4.jar, file:/usr/lib/hadoop-hdfs/lib/paranamer-2.3.jar, file:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar, file:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar, file:/usr/lib/hadoop-hdfs/lib/slf4j-api-1.6.1.jar, file:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.4.0a.jar, file:/usr/lib/hadoop-hdfs/lib/snappy-java-1.0.3.2.jar, file:/usr/lib/hadoop-hdfs/lib/jline-0.9.94.jar, file:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.3.jar, file:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar, file:/usr/lib/hadoop-hdfs/lib/zookeeper-3.4.3-cdh4.0.1.jar, file:/usr/lib/hadoop-hdfs/lib/log4j-1.2.15.jar, file:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.0.1.jar, file:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.0.1.jar, file:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.0.1-tests.jar, file:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar, file:/usr/lib/hadoop/lib/commons-codec-1.4.jar, file:/usr/lib/hadoop/lib/jets3t-0.6.1.jar, file:/usr/lib/hadoop/lib/json-simple-1.1.jar, file:/usr/lib/hadoop/lib/guava-11.0.2.jar, file:/usr/lib/hadoop/lib/avro-1.5.4.jar, file:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar, file:/usr/lib/hadoop/lib/commons-configuration-1.6.jar, file:/usr/lib/hadoop/lib/asm-3.2.jar, file:/usr/lib/hadoop/lib/paranamer-2.3.jar, file:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar, file:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar, file:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar, file:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar, file:/usr/lib/hadoop/lib/commons-cli-1.2.jar, file:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.1.jar, file:/usr/lib/hadoop/lib/commons-lang-2.5.jar, file:/usr/lib/hadoop/lib/kfs-0.3.jar, file:/usr/lib/hadoop/lib/hue-plugins-2.0.0-cdh4.0.1.jar, file:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar, file:/usr/lib/hadoop/lib/jettison-1.1.jar, file:/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar, file:/usr/lib/hadoop/lib/jsch-0.1.42.jar, file:/usr/lib/hadoop/lib/stax-api-1.0.1.jar, file:/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar, file:/usr/lib/hadoop/lib/jsr305-1.3.9.jar, file:/usr/lib/hadoop/lib/snappy-java-1.0.3.2.jar, file:/usr/lib/hadoop/lib/jsp-api-2.1.jar, file:/usr/lib/hadoop/lib/oro-2.0.8.jar, file:/usr/lib/hadoop/lib/jersey-server-1.8.jar, file:/usr/lib/hadoop/lib/commons-digester-1.8.jar, file:/usr/lib/hadoop/lib/commons-math-2.1.jar, file:/usr/lib/hadoop/lib/jline-0.9.94.jar, file:/usr/lib/hadoop/lib/core-3.1.1.jar, file:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar, file:/usr/lib/hadoop/lib/commons-el-1.0.jar, file:/usr/lib/hadoop/lib/jersey-core-1.8.jar, file:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar, file:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar, file:/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.1.jar, file:/usr/lib/zookeeper/zookeeper-3.4.3-cdh4.0.1.jar, file:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar, file:/usr/lib/hadoop/lib/commons-net-3.1.jar, file:/usr/lib/hadoop/lib/servlet-api-2.5.jar, file:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar, file:/usr/lib/hadoop/lib/commons-io-2.1.jar, file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar, file:/usr/lib/hadoop/lib/commons-logging-api-1.1.jar, file:/usr/lib/hadoop/lib/xmlenc-0.52.jar, file:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar, file:/usr/lib/hadoop/lib/activation-1.1.jar, file:/usr/lib/hadoop/lib/jersey-json-1.8.jar, file:/usr/lib/hadoop/lib/aspectjrt-1.6.5.jar, file:/usr/lib/hadoop/lib/log4j-1.2.15.jar, file:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.0.1.jar, file:/usr/lib/hadoop/hadoop-auth-2.0.0-cdh4.0.1.jar, file:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.0.1.jar, file:/usr/lib/hadoop/hadoop-annotations-2.0.0-cdh4.0.1.jar, file:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.0.1-tests.jar, file:/usr/lib/hadoop/hadoop-annotations-2.0.0-cdh4.0.1.jar, file:/usr/lib/hadoop/hadoop-auth-2.0.0-cdh4.0.1.jar, file:/mapred/local/taskTracker/root/jobcache/job_201209252321_0010/jars/classes, file:/mapred/local/taskTracker/root/jobcache/job_2012092523210010/jars/job.jar, file:/mapred/local/taskTracker/root/distcache/4260026189093522549-70309741_45603944/hadoop1.domain.com/user/root/.staging/job_201209252321_0010/libjars/hive-builtins-0.8.1-cdh4.0.1.jar, file:/mapred/local/taskTracker/root/distcache/-6339710882011042599_2132445101_45603979/hadoop1.domain.com/user/root/.staging/job_2012092523210010/libjars/hive-serdes-1.0-SNAPSHOT.jar, file:/mapred/local/taskTracker/root/distcache/7269667103068590023-978189584_45604014/hadoop1.domain.com/user/root/.staging/job_201209252321_0010/libjars/hive-contrib-0.8.1-cdh4.0.1.jar, file:/mapred/local/taskTracker/root/jobcache/job_201209252321_0010/attempt_201209252321_0010_m_000000_0/work/] 2012-09-26 15:15:40,253 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Adding alias tweets to work list for file hdfs://hadoop1.domain.com:8020/uploads 2012-09-26 15:15:40,256 INFO org.apache.hadoop.hive.ql.exec.MapOperator: dump TS structtext:string,user:struct 2012-09-26 15:15:40,256 INFO ExecMapper:

Id =3 Id =0 Id =1 Id =2 Id = 1 null<\Parent> <\FS> <\Children> Id = 0 null<\Parent> <\SEL> <\Children> Id = 3 null<\Parent> <\TS> <\Children> <\MAP> 2012-09-26 15:15:40,257 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Initializing Self 3 MAP 2012-09-26 15:15:40,257 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing Self 0 TS 2012-09-26 15:15:40,257 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: Operator 0 TS initialized 2012-09-26 15:15:40,257 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing children of 0 TS 2012-09-26 15:15:40,257 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing child 1 SEL 2012-09-26 15:15:40,257 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing Self 1 SEL 2012-09-26 15:15:40,262 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: SELECT structtext:string,user:struct 2012-09-26 15:15:40,262 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Operator 1 SEL initialized 2012-09-26 15:15:40,262 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing children of 1 SEL 2012-09-26 15:15:40,262 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: Initializing child 2 FS 2012-09-26 15:15:40,262 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: Initializing Self 2 FS 2012-09-26 15:15:40,293 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: Operator 2 FS initialized 2012-09-26 15:15:40,293 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: Initialization Done 2 FS 2012-09-26 15:15:40,293 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Initialization Done 1 SEL 2012-09-26 15:15:40,293 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: Initialization Done 0 TS 2012-09-26 15:15:40,293 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Initialization Done 3 MAP 2012-09-26 15:15:40,298 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Processing path hdfs://hadoop1.domain.com:8020/uploads/twitter.txt 2012-09-26 15:15:40,298 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Processing alias tweets for file hdfs://hadoop1.domain.com:8020/uploads 2012-09-26 15:15:40,497 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 3 forwarding 1 rows 2012-09-26 15:15:40,497 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 forwarding 1 rows 2012-09-26 15:15:40,497 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 1 forwarding 1 rows 2012-09-26 15:15:40,497 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: Final Path: FS hdfs://hadoop1.domain.com:8020/tmp/hive-root/hive_2012-09-26_15-15-33_715_8669028640552125101/_tmp.-ext-10002/000000_0 2012-09-26 15:15:40,498 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: Writing to temp file: FS hdfs://hadoop1.domain.com:8020/tmp/hive-root/hive_2012-09-26_15-15-33_715_8669028640552125101/_task_tmp.-ext-10002/_tmp.000000_0 2012-09-26 15:15:40,498 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: New Final Path: FS hdfs://hadoop1.domain.com:8020/tmp/hive-root/hive_2012-09-26_15-15-33_715_8669028640552125101/_tmp.-ext-10002/000000_0 2012-09-26 15:15:40,560 INFO ExecMapper: ExecMapper: processing 1 rows: used memory = 24284800 2012-09-26 15:15:40,577 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 3 forwarding 10 rows 2012-09-26 15:15:40,577 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 forwarding 10 rows 2012-09-26 15:15:40,577 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 1 forwarding 10 rows 2012-09-26 15:15:40,577 INFO ExecMapper: ExecMapper: processing 10 rows: used memory = 24860552 2012-09-26 15:15:40,705 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 3 forwarding 100 rows 2012-09-26 15:15:40,705 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 forwarding 100 rows 2012-09-26 15:15:40,705 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 1 forwarding 100 rows 2012-09-26 15:15:40,705 INFO ExecMapper: ExecMapper: processing 100 rows: used memory = 28885000 2012-09-26 15:15:41,499 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 3 forwarding 1000 rows 2012-09-26 15:15:41,499 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 forwarding 1000 rows 2012-09-26 15:15:41,499 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 1 forwarding 1000 rows 2012-09-26 15:15:41,499 INFO ExecMapper: ExecMapper: processing 1000 rows: used memory = 7598072 2012-09-26 15:15:42,992 FATAL ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"text":"@KimKardashian happy valentines day, hope it's a good one","retweet_count":0,"geo":{"type":"Point","coordinates":[38.7313358,-108.05278695]},"in_reply_to_status_id_str":null,"in_reply_to_user_id":25365536,"source":"\u003Ca href=\"http:\/\/twitter.com\/download\/android\" rel=\"nofollow\"\u003ETwitter for Android\u003C\/a\u003E","in_reply_to_user_id_str":"25365536","id_str":"169483808003989505","entities":{"user_mentions":[{"indices":[0,14],"screen_name":"KimKardashian","id_str":"25365536","name":"Kim Kardashian","id":25365536}],"urls":[],"hashtags":[]},"in_reply_to_status_id":null,"place":{"url":"http:\/\/api.twitter.com\/1\/geo\/id\/6a7e7dbf9d6c7ac4.json","place_type":"city","country_code":"US","attributes":{},"full_name":"Delta, CO","bounding_box":{"type":"Polygon","coordinates":[[[-108.104644,38.71503],[-108.021863,38.71503],[-108.021863,38.769794],[-108.104644,38.769794]]]},"name":"Delta","id":"6a7e7dbf9d6c7ac4","country":"United States"},"in_reply_to_screen_name":"Ki{"text":"@bbrandivirgo too bad I dont have the number. Happy valentines day tho :)","retweet_count":0,"geo":{"type":"Point","coordinates":[33.77406404,-84.39270512]},"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"source":"\u003Ca href=\"http:\/\/mobile.twitter.com\" rel=\"nofollow\"\u003EMobile Web\u003C\/a\u003E","in_reply_to_user_id_str":null,"id_str":"169497701241716736","entities":{"user_mentions":[],"urls":[],"hashtags":[]},"in_reply_to_status_id":null,"place":{"url":"http:\/\/api.twitter.com\/1\/geo\/id\/8173485c72e78ca5.json","place_type":"city","country_code":"US","attributes":{},"full_name":"Atlanta, GA","bounding_box":{"type":"Polygon","coordinates":[[[-84.54674,33.647908],[-84.289389,33.647908],[-84.289389,33.887618],[-84.54674,33.887618]]]},"name":"Atlanta","id":"8173485c72e78ca5","country":"United States"},"in_reply_to_screen_name":null,"favorited":false,"truncated":false,"created_at":"Tue Feb 14 19:06:15 +0000 2012","contributors":null,"user":{"contributors_enabled":false,"profile_background_image_url":"http:\/\/a3.twimg.com\/profile_background_images\/376284279\/yyyyyyyyyyyyyyyyyyyy.jpg","url":"http:\/\/facebook.com\/cperk3","profile_link_color":"0084B4","followers_count":773,"profile_image_url":"http:\/\/a3.twimg.com\/profile_images\/1792490671\/000011110000_normal.jpg","default_profile_image":false,"show_all_inline_media":true,"statuses_count":3271,"profile_background_color":"C0DEED","description":"Ga Tech Athlete-Student.. Black&Samoan...Follow me as I follow Jesus-","location":"Atlanta, GA","profile_background_tile":true,"favourites_count":1,"profile_background_image_url_https":"https:\/\/si0.twimg.com\/profile_background_images\/376284279\/yyyyyyyyyyyyyyyyyyyy.jpg","time_zone":"Quito","profile_sidebar_fill_color":"DDEEF6","screen_name":"Cpeezy21","id_str":"312682111","lang":"en","geo_enabled":true,"profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/1792490671\/000011110000_normal.jpg","verified":false,"notifications":null,"profile_sidebar_border_color":"04080a","protected":false,"listed_count":5,"created_at":"Tue Jun 07 14:14:34 +0000 2011","name":"Charles Perkins III","is_translator":false,"follow_request_sent":null,"following":null,"profile_use_background_image":true,"friends_count":223,"id":312682111,"default_profile":false,"utc_offset":-18000,"profile_text_color":"333333"},"retweeted":false,"id":169497701241716736,"coordinates":{"type":"Point","coordinates":[-84.39270512,33.77406404]}} at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:393) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.mapred.Child.main(Child.java:264) Caused by: org.apache.hadoop.hive.serde2.SerDeException: org.codehaus.jackson.JsonParseException: Unexpected character ('t' (code 116)): was expecting comma to separate OBJECT entries at [Source: java.io.StringReader@366ef7ba; line: 1, column: 999] at com.cloudera.hive.serde.JSONSerDe.deserialize(JSONSerDe.java:128) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:508) ... 9 more Caused by: org.codehaus.jackson.JsonParseException: Unexpected character ('t' (code 116)): was expecting comma to separate OBJECT entries at [Source: java.io.StringReader@366ef7ba; line: 1, column: 999] at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1291) at org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385) at org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:306) at org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:285) at org.codehaus.jackson.map.deser.MapDeserializer._readAndBind(MapDeserializer.java:220) at org.codehaus.jackson.map.deser.MapDeserializer.deserialize(MapDeserializer.java:165) at org.codehaus.jackson.map.deser.MapDeserializer.deserialize(MapDeserializer.java:25) at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2402) at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1602) at com.cloudera.hive.serde.JSONSerDe.deserialize(JSONSerDe.java:126) ... 10 more 2012-09-26 15:15:42,993 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 3 finished. closing... 2012-09-26 15:15:42,993 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 3 forwarded 4551 rows 2012-09-26 15:15:42,993 INFO org.apache.hadoop.hive.ql.exec.MapOperator: DESERIALIZE_ERRORS:1 2012-09-26 15:15:42,993 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 finished. closing... 2012-09-26 15:15:42,993 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 forwarded 4551 rows 2012-09-26 15:15:42,993 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 1 finished. closing... 2012-09-26 15:15:42,993 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 1 forwarded 4551 rows 2012-09-26 15:15:42,993 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: 2 finished. closing... 2012-09-26 15:15:42,993 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: 2 forwarded 0 rows 2012-09-26 15:15:43,066 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: TABLE_ID_1_ROWCOUNT:4551 2012-09-26 15:15:43,066 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 1 Close done 2012-09-26 15:15:43,066 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 Close done 2012-09-26 15:15:43,066 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 3 Close done 2012-09-26 15:15:43,066 INFO ExecMapper: ExecMapper: processed 4551 rows: used memory = 17571376 2012-09-26 15:15:43,074 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 2012-09-26 15:15:43,077 WARN org.apache.hadoop.mapred.Child: Error running child java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"text":"@KimKardashian happy valentines day, hope it's a good one","retweet_count":0,"geo":{"type":"Point","coordinates":[38.7313358,-108.05278695]},"in_reply_to_status_id_str":null,"in_reply_to_user_id":25365536,"source":"\u003Ca href=\"http:\/\/twitter.com\/download\/android\" rel=\"nofollow\"\u003ETwitter for Android\u003C\/a\u003E","in_reply_to_user_id_str":"25365536","id_str":"169483808003989505","entities":{"user_mentions":[{"indices":[0,14],"screen_name":"KimKardashian","id_str":"25365536","name":"Kim Kardashian","id":25365536}],"urls":[],"hashtags":[]},"in_reply_to_status_id":null,"place":{"url":"http:\/\/api.twitter.com\/1\/geo\/id\/6a7e7dbf9d6c7ac4.json","place_type":"city","country_code":"US","attributes":{},"full_name":"Delta, CO","bounding_box":{"type":"Polygon","coordinates":[[[-108.104644,38.71503],[-108.021863,38.71503],[-108.021863,38.769794],[-108.104644,38.769794]]]},"name":"Delta","id":"6a7e7dbf9d6c7ac4","country":"United States"},"in_reply_to_screen_name":"Ki{"text":"@bbrandivirgo too bad I dont have the number. Happy valentines day tho :)","retweet_count":0,"geo":{"type":"Point","coordinates":[33.77406404,-84.39270512]},"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"source":"\u003Ca href=\"http:\/\/mobile.twitter.com\" rel=\"nofollow\"\u003EMobile Web\u003C\/a\u003E","in_reply_to_user_id_str":null,"id_str":"169497701241716736","entities":{"user_mentions":[],"urls":[],"hashtags":[]},"in_reply_to_status_id":null,"place":{"url":"http:\/\/api.twitter.com\/1\/geo\/id\/8173485c72e78ca5.json","place_type":"city","country_code":"US","attributes":{},"full_name":"Atlanta, GA","bounding_box":{"type":"Polygon","coordinates":[[[-84.54674,33.647908],[-84.289389,33.647908],[-84.289389,33.887618],[-84.54674,33.887618]]]},"name":"Atlanta","id":"8173485c72e78ca5","country":"United States"},"in_reply_to_screen_name":null,"favorited":false,"truncated":false,"created_at":"Tue Feb 14 19:06:15 +0000 2012","contributors":null,"user":{"contributors_enabled":false,"profile_background_image_url":"http:\/\/a3.twimg.com\/profile_background_images\/376284279\/yyyyyyyyyyyyyyyyyyyy.jpg","url":"http:\/\/facebook.com\/cperk3","profile_link_color":"0084B4","followers_count":773,"profile_image_url":"http:\/\/a3.twimg.com\/profile_images\/1792490671\/000011110000_normal.jpg","default_profile_image":false,"show_all_inline_media":true,"statuses_count":3271,"profile_background_color":"C0DEED","description":"Ga Tech Athlete-Student.. Black&Samoan...Follow me as I follow Jesus-","location":"Atlanta, GA","profile_background_tile":true,"favourites_count":1,"profile_background_image_url_https":"https:\/\/si0.twimg.com\/profile_background_images\/376284279\/yyyyyyyyyyyyyyyyyyyy.jpg","time_zone":"Quito","profile_sidebar_fill_color":"DDEEF6","screen_name":"Cpeezy21","id_str":"312682111","lang":"en","geo_enabled":true,"profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/1792490671\/000011110000_normal.jpg","verified":false,"notifications":null,"profile_sidebar_border_color":"04080a","protected":false,"listed_count":5,"created_at":"Tue Jun 07 14:14:34 +0000 2011","name":"Charles Perkins III","is_translator":false,"follow_request_sent":null,"following":null,"profile_use_background_image":true,"friends_count":223,"id":312682111,"default_profile":false,"utc_offset":-18000,"profile_text_color":"333333"},"retweeted":false,"id":169497701241716736,"coordinates":{"type":"Point","coordinates":[-84.39270512,33.77406404]}} at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:393) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.mapred.Child.main(Child.java:264) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"text":"@KimKardashian happy valentines day, hope it's a good one","retweet_count":0,"geo":{"type":"Point","coordinates":[38.7313358,-108.05278695]},"in_reply_to_status_id_str":null,"in_reply_to_user_id":25365536,"source":"\u003Ca href=\"http:\/\/twitter.com\/download\/android\" rel=\"nofollow\"\u003ETwitter for Android\u003C\/a\u003E","in_reply_to_user_id_str":"25365536","id_str":"169483808003989505","entities":{"user_mentions":[{"indices":[0,14],"screen_name":"KimKardashian","id_str":"25365536","name":"Kim Kardashian","id":25365536}],"urls":[],"hashtags":[]},"in_reply_to_status_id":null,"place":{"url":"http:\/\/api.twitter.com\/1\/geo\/id\/6a7e7dbf9d6c7ac4.json","place_type":"city","country_code":"US","attributes":{},"full_name":"Delta, CO","bounding_box":{"type":"Polygon","coordinates":[[[-108.104644,38.71503],[-108.021863,38.71503],[-108.021863,38.769794],[-108.104644,38.769794]]]},"name":"Delta","id":"6a7e7dbf9d6c7ac4","country":"United States"},"in_reply_to_screen_name":"Ki{"text":"@bbrandivirgo too bad I dont have the number. Happy valentines day tho :)","retweet_count":0,"geo":{"type":"Point","coordinates":[33.77406404,-84.39270512]},"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"source":"\u003Ca href=\"http:\/\/mobile.twitter.com\" rel=\"nofollow\"\u003EMobile Web\u003C\/a\u003E","in_reply_to_user_id_str":null,"id_str":"169497701241716736","entities":{"user_mentions":[],"urls":[],"hashtags":[]},"in_reply_to_status_id":null,"place":{"url":"http:\/\/api.twitter.com\/1\/geo\/id\/8173485c72e78ca5.json","place_type":"city","country_code":"US","attributes":{},"full_name":"Atlanta, GA","bounding_box":{"type":"Polygon","coordinates":[[[-84.54674,33.647908],[-84.289389,33.647908],[-84.289389,33.887618],[-84.54674,33.887618]]]},"name":"Atlanta","id":"8173485c72e78ca5","country":"United States"},"in_reply_to_screen_name":null,"favorited":false,"truncated":false,"created_at":"Tue Feb 14 19:06:15 +0000 2012","contributors":null,"user":{"contributors_enabled":false,"profile_background_image_url":"http:\/\/a3.twimg.com\/profile_background_images\/376284279\/yyyyyyyyyyyyyyyyyyyy.jpg","url":"http:\/\/facebook.com\/cperk3","profile_link_color":"0084B4","followers_count":773,"profile_image_url":"http:\/\/a3.twimg.com\/profile_images\/1792490671\/000011110000_normal.jpg","default_profile_image":false,"show_all_inline_media":true,"statuses_count":3271,"profile_background_color":"C0DEED","description":"Ga Tech Athlete-Student.. Black&Samoan...Follow me as I follow Jesus-","location":"Atlanta, GA","profile_background_tile":true,"favourites_count":1,"profile_background_image_url_https":"https:\/\/si0.twimg.com\/profile_background_images\/376284279\/yyyyyyyyyyyyyyyyyyyy.jpg","time_zone":"Quito","profile_sidebar_fill_color":"DDEEF6","screen_name":"Cpeezy21","id_str":"312682111","lang":"en","geo_enabled":true,"profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/1792490671\/000011110000_normal.jpg","verified":false,"notifications":null,"profile_sidebar_border_color":"04080a","protected":false,"listed_count":5,"created_at":"Tue Jun 07 14:14:34 +0000 2011","name":"Charles Perkins III","is_translator":false,"follow_request_sent":null,"following":null,"profile_use_background_image":true,"friends_count":223,"id":312682111,"default_profile":false,"utc_offset":-18000,"profile_text_color":"333333"},"retweeted":false,"id":169497701241716736,"coordinates":{"type":"Point","coordinates":[-84.39270512,33.77406404]}} at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143) ... 8 more Caused by: org.apache.hadoop.hive.serde2.SerDeException: org.codehaus.jackson.JsonParseException: Unexpected character ('t' (code 116)): was expecting comma to separate OBJECT entries at [Source: java.io.StringReader@366ef7ba; line: 1, column: 999] at com.cloudera.hive.serde.JSONSerDe.deserialize(JSONSerDe.java:128) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:508) ... 9 more Caused by: org.codehaus.jackson.JsonParseException: Unexpected character ('t' (code 116)): was expecting comma to separate OBJECT entries at [Source: java.io.StringReader@366ef7ba; line: 1, column: 999] at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1291) at org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385) at org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:306) at org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:285) at org.codehaus.jackson.map.deser.MapDeserializer._readAndBind(MapDeserializer.java:220) at org.codehaus.jackson.map.deser.MapDeserializer.deserialize(MapDeserializer.java:165) at org.codehaus.jackson.map.deser.MapDeserializer.deserialize(MapDeserializer.java:25) at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2402) at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1602) at com.cloudera.hive.serde.JSONSerDe.deserialize(JSONSerDe.java:126) ... 10 more 2012-09-26 15:15:43,081 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task