search

infochimps-labs / wonderdog

Bulk loading for elastic search
http://infochimps.com
Apache License 2.0
186 stars 56 forks source link

Can't import config/plugins directory anymore :( #7

Closed rjurney closed 11 years ago

rjurney commented 12 years ago

In the code I read about using Hadoop's cache... well, I'm testing in Pig local mode, so I can't use Hadoop's cache. Poop :( I was happy importing these as options, but that has no effect now.

Rolling back gets me weird stuff, should I patch the code that was removed that made this work back in? No matter what environment variables I set, I can no longer make it go.

store json_emails into 'es://email/email?id=message_id&json=true&size=1000' USING com.infochimps.elasticsearch.pig.ElasticSearchStorage('/me/elasticsearch-0.18.6/config/elasticsearch.yml', '/me/elasticsearch-0.18.6/plugins');

java.lang.RuntimeException: java.lang.NullPointerException at com.infochimps.elasticsearch.ElasticSearchOutputFormat$ElasticSearchRecordWriter.(ElasticSearchOutputFormat.java:133) at com.infochimps.elasticsearch.ElasticSearchOutputFormat.getRecordWriter(ElasticSearchOutputFormat.java:262) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.(MapTask.java:628) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:753) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) Caused by: java.lang.NullPointerException at java.util.Hashtable.put(Hashtable.java:394) at java.util.Properties.setProperty(Properties.java:143) at java.lang.System.setProperty(System.java:746) at com.infochimps.elasticsearch.ElasticSearchOutputFormat$ElasticSearchRecordWriter.(ElasticSearchOutputFormat.java:130) ... 6 more

rjurney commented 12 years ago

Fixed by https://github.com/infochimps-labs/wonderdog/pull/8

alexmc6 commented 11 years ago

I just did a git clone and got a very similar error. Has the above fix made it into the main codebase?

eg

I have elasticsearch unpacked and running under /home/ubuntu and not yet "installed".

It first of all complained about not finding /etc/elasticsearch/elasticsearch.yml I touched that file (after looking at the sample config) and then....

2012-11-02 17:29:20,848 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2999: Unexpected internal error. java.io.FileNotFoundException: File file:/usr/local/share/elasticsearch/plugins does not exist Details at logfile: /home/ubuntu/anorak_cluster/pig_1351877356601.log /home/ubuntu/anorak_cluster$ echo $ES_HOME /home/ubuntu/elasticsearch

However creating that missing directory does help....