pouchdb-community / pouchdb-replication-stream

Replicate PouchDB/CouchDB databases with Node.js-style streams
Apache License 2.0
189 stars 58 forks source link

[NodeJS] Attachment data are written as object #60

Open chancezeus opened 7 years ago

chancezeus commented 7 years ago

When using this plugin to dump my database to a file, the attachments are written:

{
  content_type: ***,
  data: {
    type: 'Buffer',
    data: [***]
  }
}

Importing this goes wrong in the browser since the browser has no clue how to handle this... Your readme states attachments should be saved as 'base64' encoded data. Is there any way to get this behaviour, since the plugin is broken now as a tool for exporting a prepopulated database (used to debug/develop a cordova app in the browser)...

nolanlawson commented 7 years ago

Looks like it is dumping the Buffer as a Node-style Buffer object rather than a Blob, so it will have to be converted. Yes this is a bug; it seems I didn't consider the case of attachments in the browser.

HarelM commented 5 years ago

@chancezeus how did you solve this? @nolanlawson any pointer to where the fix should be? I'd be happy to add a PR for this issue as I'm experiencing exactly this problem now - also using cordova for loading and NodeJS for dumping.

HarelM commented 5 years ago

I have managed to write the following code in the client side to overcome this issue if anyone is interested (bare in mind that it probably has bugs), I'll be writing a workaround in the server side in order to crate a file the load can read until this issue is resolved:

var textSplit = text.split('\n');
for (let i = 0; i < textSplit.length; i++) {
    if (!textSplit[i].startsWith('{"docs')) {
        continue;
    }
    let jsonText = JSON.parse(textSplit[i]);
    for (let doc of jsonText.docs) {
        let byteArray = doc._attachments[Object.keys(doc._attachments)[0]].data.data;
        let str = btoa(byteArray);
        doc._attachments[Object.keys(doc._attachments)[0]].data = str
        console.log(doc);
    }
    textSplit[i] = JSON.stringify(jsonText);
}
text = textSplit.join('\n');

nodejs code:

var fs = require("fs");

fs.readFile("./tiles.txt", (err, data) => {
    let textSplit = data.toString().split("\n");
    for (let i = 0; i < textSplit.length; i++) {
        console.log("line: " + i);
        if (!textSplit[i].startsWith('{"docs')) {
            continue;
        }
        let jsonText = JSON.parse(textSplit[i]);
        for (let doc of jsonText.docs) {
            let byteArray = doc._attachments[Object.keys(doc._attachments)[0]].data.data;
            let str = new Buffer(byteArray).toString("base64");
            doc._attachments[Object.keys(doc._attachments)[0]].data = str
        }
        textSplit[i] = JSON.stringify(jsonText);
    }
    text = textSplit.join('\n');
    console.log("complete");
    fs.writeFile("./tile-converted.txt", text, () => console.log("complete writing"));
});