Insert into remote host error

[badalb]

_Seems inserting into remote host is not working properly - _ My client code : - String host = "";

    DDataSource.adjustPoolSettings(100, 50, 50, 50);

    List<String> inserts = ImmutableList
            .of("INSERT INTO abc (timestamp , provider , uuid, DOUBLE_SUM(click) AS click) "
                    + "  VALUES ('2014-10-31 00:00:00','sri','fsd-sdf-dfgdf','2') WHERE"
                    + " interval BETWEEN '2014-10-30' AND '2014-11-01'  BREAK BY 'day'",
                    "INSERT INTO abc (timestamp , provider , uuid, DOUBLE_SUM(click) AS click)  "
                            + "VALUES ('2014-11-01 00:00:00','siri','abd-dfgdf-32de1','5') WHERE"
                            + " interval BETWEEN '2014-11-01' AND '2014-11-02'  BREAK BY 'day'",
                    "INSERT INTO abc (timestamp , provider , uuid, DOUBLE_SUM(click) AS click)"
                            + "  VALUES ('2014-11-02 00:00:00','preeti','dfgdf-fsd-sdf','8') WHERE"
                            + " interval BETWEEN '2014-11-02' AND '2014-11-03'  BREAK BY 'day'",
                    "INSERT INTO abc (timestamp , provider , uuid, DOUBLE_SUM(click) AS click)"
                            + "  VALUES ('2014-11-03 00:00:00','aditya','fsd-sdf-dfgdf','1') WHERE"
                            + " interval BETWEEN '2014-11-03' AND '2014-11-04'  BREAK BY 'day'",
                    "INSERT INTO abc (timestamp , provider , uuid, DOUBLE_SUM(click) AS click)"
                            + "  VALUES ('2014-11-04 00:00:00','adi','fsd-sdf-dfgdf','9999') WHERE"
                            + " interval BETWEEN '2014-11-04' AND '2014-11-05'  BREAK BY 'day'",
                    "INSERT INTO abc (timestamp , provider , uuid, DOUBLE_SUM(click) AS click) "
                            + " VALUES ('2014-11-05 00:00:00','antham','fsd-sdf-dfgdf','99999')"
                            + " WHERE interval BETWEEN '2014-11-05' AND '2014-11-06'  BREAK BY 'day'");
    //DDataSource driver = new DDataSource("localhost", 8080, "localhost",
        //  8082, "localhost", 8087, "localhost", 3306, "root", "password",
            //"druid");
    DDataSource driver = new DDataSource(host, 8082, host, 8081, host,
            8190, host, 3306, "root", "password", "druid");

    for (String insert : inserts) {
        driver.query(insert, null, null, true, "sql");
    }

BasicInsertMeta.java

dataPath is always null. So creating the csv file in client local system.

public Map<String, Object> getFirehose() { Map<String, Object> finalMap = new LinkedHashMap<>();
finalMap.put("type", "local"); if (dataPath != null) { int folderEndIndex = dataPath.lastIndexOf("/"); finalMap.put("baseDir", dataPath.substring(0, folderEndIndex + 1)); finalMap.put("filter", (folderEndIndex == dataPath.length() - 1)?"*":dataPath.substring(folderEndIndex + 1)); if (dataPath.endsWith("json")) { dataFormat = "json"; } else if (dataPath.endsWith("csv")) { dataFormat = "csv"; } } else { finalMap.put("baseDir", tmpFolder); String fileName = UUID.randomUUID().toString() + ".csv"; finalMap.put("filter", fileName); dataFormat = "csv"; if (values.isEmpty()) { throw new IllegalStateException("No values to insert !!"); } try { File file = new File(tmpFolder + File.separator + fileName); FileUtils.write(file, Joiner.on(",").join(values)); System.out.println("Written to " + file); Object timestamp = values.get(0); timestampFormat = TimeUtils.detectFormat(timestamp.toString()); } catch (IOException ex) { Logger.getLogger(BasicInsertMeta.class.getName()).log(Level.SEVERE, null, ex); }

    }
    return finalMap;
}

Error log:- Coordinator console error log -> looking for csv in remote server. Although its written in client 2016-01-29T06:15:19,189 INFO [task-runner-0] io.druid.segment.realtime.firehose.LocalFirehoseFactory - Searching for all [6e06a2e0-ae0e-479e-ad38-60769f27802f.csv] in and beneath [/var/folders/_y/hxlk4pz16qj2lsp6gs_0yjxnbt3pf8/T] 2016-01-29T06:15:19,197 ERROR [task-runner-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[IndexTask{id=index_abc_2016-01-29T06:15:11.269Z, type=index, dataSource=abc}] java.lang.IllegalArgumentException: Parameter 'directory' is not a directory at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:358) ~[commons-io-2.0.1.jar:2.0.1] at io.druid.segment.realtime.firehose.LocalFirehoseFactory.connect(LocalFirehoseFactory.java:87) ~[druid-server-0.8.2.jar:0.8.2] at io.druid.segment.realtime.firehose.LocalFirehoseFactory.connect(LocalFirehoseFactory.java:43) ~[druid-server-0.8.2.jar:0.8.2] at io.druid.indexing.common.task.IndexTask.getDataIntervals(IndexTask.java:234) ~[druid-indexing-service-0.8.2.jar:0.8.2] at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:192) ~[druid-indexing-service-0.8.2.jar:0.8.2] at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:221) [druid-indexing-service-0.8.2.jar:0.8.2] at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:200) [druid-indexing-service-0.8.2.jar:0.8.2] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_67] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_67] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_67] at java.lang.Thread.run(Thread.java:745) [?:1.7.0_67] 2016-01-29T06:15:19,202 INFO [task-runner-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: { "id" : "index_abc_2016-01-29T06:15:11.269Z", "status" : "FAILED", "duration" : 65 }

druidGParser.java line 1142 always returning 88 for query like above forcing to switch statement case(1) breaking the loop.

[srikalyc]

Using insert as INSERT INTO table VALUES ('') works only with LOCAL. If you need remote host to work use INSERT_HADOOP . However you use INSERT INTO .. FILE but that file should exist in the remote host (If you have NFS much better).

Thanks kalyan

On Fri, Jan 29, 2016 at 2:04 AM, Badal Baidya notifications@github.com wrote:

druidGParser.java line 1142 always returning 88 for query like above forcing to switch statement case(1) breaking the loop.

— Reply to this email directly or view it on GitHub https://github.com/srikalyc/Sql4D/issues/94#issuecomment-176676941.

Bye & regards C.Srikalyan

[badalb]

Hi, Thanks a lot for your response. Do you have any sample code example to demonstrate INSERT_HADOOP ? Could you please help.

Regards, Badal

[srikalyc]

Badal, If you see https://github.com/srikalyc/Sql4D/ and scroll down a bit, you will find a table with bunch of links to blogs. Hope this helps.

Thanks kalyan

On Sun, Jan 31, 2016 at 10:36 PM, Badal Baidya notifications@github.com wrote:

Hi, Thanks a lot for your response. Do you have any sample code example to demonstrate INSERT_HADOOP ? Could you please help.

Regards, Badal

— Reply to this email directly or view it on GitHub https://github.com/srikalyc/Sql4D/issues/94#issuecomment-177803705.

Bye & regards C.Srikalyan

[badalb]

Hi Kalyan,

Got it. Thanks a lot.

Regards, Badal