compose / transporter

Sync data between persistence engines, like ETL only not stodgy
https://github.com/compose/transporter/issues/523
BSD 3-Clause "New" or "Revised" License
1.45k stars 213 forks source link

New object added during transformer is saved empty #185

Closed randysofia closed 8 years ago

randysofia commented 8 years ago

When creating / adding a new object to data in a transformer:

module.exports = function(doc) {
     doc.data.hero = {"name":"batman","villian":"joker"};    // NEW OBJECT 'HERO'
     var something = new Object(); 
     something = {
        "input" : [doc.data.name, doc.data.city, doc.data.state],
        "output" : doc.data.name + " - " + doc.data.city + ", " + doc.data.state,
        "payload" : { "ceeb" : doc.data.ceeb }
     };
     doc.data.suggest=something;   // NEW OBJECT 'SUGGEST'

     console.log(JSON.stringify(doc));
     return doc;
}

Console output correctly displays new 'hero' and 'suggest' object data:

{"data":{"_id":{"$oid":"55352e5a2e402e53598b456f"},"ceeb":"10000","name":"Abbeville High School","state":"AL","street_1":"411 Graball Cutoff","suggest":{"input":["Abbeville High School","Abbeville","AL"],"output":"Abbeville High School - Abbeville, AL","payload":{"ceeb":"10000"}},"type":"highschool","updated_at":{"$date":1429548634000},"zipcode":"36310","zipcode4":"2073"},"ns":"luminous-live.luminous_ceebs","op":"insert","ts":1461873170}

However, .save output shows the new object as empty:

{"_id":"55352e5a2e402e53598b456f","ceeb":"10000","hero":{},"name":"Abbeville High School","state":"AL","street_1":"411 Graball Cutoff","suggest":{},"type":"highschool","updated_at":"2015-04-20T16:50:34Z","zipcode":"36310","zipcode4":"2073"}

Config:

nodes:
  mdb:
    type: mongodb
    uri: mongodb://secret
    tail: true
  debug:
    type: file
    uri: stdout://

js pipeline:

var pipeline=Source({name:"mdb",namespace:"luminous-live.luminous_ceebs"})
  .transform({filename: "/home/elasticsearch/ceebs_filter.js", namespace: "luminous-live.luminous_ceebs"})
  .save({name:"debug",namespace:"luminous-test.luminous_ceebs"})
[elasticsearch@ip-172-31-27-6 transporter]$ ./transporter --version
0.1.1
randysofia commented 8 years ago

Note: I've tested adding a string and it works fine. Issue only pertains to added objects.

trinchan commented 8 years ago

Resolved in experimental build 4c8bb7d