V2 mongoDB migration

[cdupont]

Data from MongoDB and MySQL need to be migrated to new format. Old format:

• each domain as a collection
• document format:

  { "_id" : ObjectId("5b5a022ef21f630001b4be6e"), "entityID" : "FL_Sensor32", "entityType" : "SensingDevice", "attributeID" : "SMC", "value" : "327.50", "timestamp" : ISODate("2018-07-26T17:17:30Z") }

New format:

• one collection for all datapoints (called "waziup_history")
• document format:

  { "_id" : ObjectId("5cd308bd41dabe0006000000"), "sensor_id" : "TC1", "device_id" : "MyDevice", "value" : 25, "timestamp" : "2016-06-08T18:20:27.873Z" }

[cdupont]

Actions:

• merge all documents in the same collection
• rename entityID -> device_id
• rename attributeID -> sensor_id
• remove SensingDevice

[cdupont]

Used this script:

use waziup_history;
db.getCollectionNames().forEach(function(collname) {
  db.getCollection(collname).copyTo(db.getSiblingDB('waziup')['waziup_history'])
  print(collname + " done");
});

use waziup;
db.waziup_history.update({}, {$rename:{"entityID":"device_id", "attributeID":"sensor_id"}, $unset: {"entityType":""}}, false, true);

[cdupont]

Problem now is to add the keycloak Id for existing sensors. Or should they be considered public? Script need to:

• read a device
• extract name
• get resource in Keycloak with the same name
• write the keycloak Id