jprante / elasticsearch-jdbc

JDBC importer for Elasticsearch
Apache License 2.0
2.84k stars 710 forks source link

incrementally updates using UNIX_TIMESTAMP returns 0 rows? #410

Open ksdpmx opened 9 years ago

ksdpmx commented 9 years ago

According to the manual, I did a small test.

My db.table is:

id last_active_begin unix_timestamp 1 2014-12-11 00:40:15.000 1418029615 2 2014-12-11 01:48:45.000 1418062525 3 2014-12-11 01:49:15.000 1418062555 4 2014-12-11 01:49:45.000 1418062585

$river.state.last_active_begin is 2014-12-11T03:14:10.855Z. last_active_begin and unix_timestamp are nonsense and have no relations. And I imported the db.table data to elasticsearch db.table.

Then I add a row in MySQL manually: 5 2014-12-12 00:00.00.000 1518062585, curl -XPOST localhost:9200/_river/incremental/_meta -d @incremental.json:

{
  "type": "jdbc",
  "jdbc": {
    "url": "jdbc:mysql://localhost:3306/db",
    "user": "",
    "password": "",
    "sql": [
      {
        "statement": "SELECT id AS _id, unix_timestamp FROM table WHERE unix_timestamp > UNIX_TIMESTAMP(?)",
        "parameter": [ "$river.state.last_active_begin" ]
      }
    ],
    "index": "db",
    "type": "table"
  }  
}

elasticsearch.log returns:

[2014-12-10][INFO][river.jdbc.RiverMetrics  ] pipeline org.xbib.elasticsearch.plugin.jdbc.RiverPipeline@7ae3418b complete: river jdbc/incremental metrics: 0 rows, 0.0 mean, (0.0 0.0 0.0), ingest metrics: elapsed 0 seconds, 0.0 bytes bytes, 0.0 bytes avg, 0 MB/s

Also, I've tried to work around by changing unix_timestamp to last_update_active with a completely clean ES:

{
  "type": "jdbc",
  "interval": "30s",
  "jdbc": {
    "url": "jdbc:mysql://localhost:3306/db",
    "user": "",
    "password": "",
    "sql": [
      {
        "statement": "SELECT id AS _id, last_active_begin, unix_timestamp FROM table WHERE last_active_begin > ?",
        "parameter": [ "$river.state.last_active_begin" ]
      }
    ],
    "index": "db",
    "type": "table"
  }
}

Still got nothing.

Why I cannot get any results? Thank you.

ES v1.3.4, elasticsearch-river-jdbc v1.3.4.4, jdbc v5.1.33.

ps: manual "statement" : "select * from \"products\" where \"mytimestamp\" > ?", there's no need to add quotations in mytimestamp.

ksdpmx commented 9 years ago

Changing the statement above to "statement": "SELECT id AS _id, last_active_begin, unix_timestamp FROM table WHERE last_active_begin >= ? solves the problem...

UNIX_TIMESTAMP, FROM_UNIXTIME cannot be used in sql?

ES v1.3.4

ps: FROM_UNIXTIME can be used in ES v1.3.6...