Error: size mismatch

[fullstackPrincess]

I'm trying to dump from mongo to tar, scripts work 3-4 minutes and throw error

last log

...
14|backup_task  | remaining bytes to write: 246657495
14|backup_task  | remaining bytes to write: 246657286
14|backup_task  | remaining bytes to write: 246657077
14|backup_task  | remaining bytes to write: 246656868
14|backup_task  | remaining bytes to write: 246656659
14|backup_task  | remaining bytes to write: 246656450
14|backup_task  | remaining bytes to write: 246656241
14|backup_task  | remaining bytes to write: 246656032
14|backup_task  | remaining bytes to write: 246655823
14|backup_task  | remaining bytes to write: 246655614
14|backup_task  | could not write collection data into: type: local, path: ./mongo.tar due to error:
14|backup_task  | Error: size mismatch
14|backup_task  |     at Sink.<anonymous> (/srv/backup/node_modules/tar-stream/pack.js:175:23)
14|backup_task  |     at Sink.f (/srv/backup/node_modules/once/once.js:25:25)
14|backup_task  |     at Sink.onfinish (/srv/backup/node_modules/end-of-stream/index.js:31:27)
14|backup_task  |     at Sink.emit (events.js:315:20)
14|backup_task  |     at finishMaybe (/srv/backup/node_modules/tar-stream/node_modules/readable-stream/lib/_stream_writable.js:624:14)
14|backup_task  |     at endWritable (/srv/backup/node_modules/tar-stream/node_modules/readable-stream/lib/_stream_writable.js:643:3)
14|backup_task  |     at Sink.Writable.end (/srv/backup/node_modules/tar-stream/node_modules/readable-stream/lib/_stream_writable.js:571:22)
14|backup_task  |     at SafeSubscriber._complete (/srv/backup/node_modules/mongodb-snapshot/dist/connectors/filesystem/FileSystemDuplexConnector.js:273:19)
14|backup_task  |     at SafeSubscriber.wrappedComplete (/srv/backup/node_modules/rxjs/internal/Subscriber.js:190:76)
14|backup_task  |     at SafeSubscriber.__tryOrUnsub (/srv/backup/node_modules/rxjs/internal/Subscriber.js:207:16)
14|backup_task  | removing: "type: local, path: ./mongo.tar" because an error was thrown during the transfer

used code

async function dumpMongo2Localfile() {
    console.log(`mongo bk -- start`);
    const mongo_connector = new MongoDBDuplexConnector({
        connection: {
            uri: `mongodb://localhost`,
            dbname: 'geoip',
        },
    });

    const localfile_connector = new LocalFileSystemDuplexConnector({
        connection: {
            path: './mongo.tar',
        },
    });

    const transferer = new MongoTransferer({
        source: mongo_connector,
        targets: [localfile_connector],
    });

    for await (const { total, write } of transferer) {
       console.log(`remaining bytes to write: ${total - write}`);
    }

    console.log(`mongo bk -- done`);
}

npm package version

"mongodb-snapshot": "^1.2.1"

node version

root@cs553573:/srv/backup# node -v
v14.1.0

mongodb version

root@cs553573:/srv/backup# mongod --version
db version v4.2.5
git version: 2261279b51ea13df08ae708ff278f0679c59dc32
OpenSSL version: OpenSSL 1.1.1  11 Sep 2018
allocator: tcmalloc
modules: none
build environment:
    distmod: ubuntu1804
    distarch: x86_64
    target_arch: x86_64

db stats

  {
    "avgObjSize": 253.81038799023667707,
    "collections": 8,
    "dataSize": 332441100,
    "db": "geoip",
    "fsTotalSize": 63345287168,
    "fsUsedSize": 45903093760,
    "indexSize": 135430144,
    "indexes": 13,
    "numExtents": 0,
    "objects": 1309801,
    "ok": 1,
    "scaleFactor": 1,
    "storageSize": 345493504,
    "views": 0
  }

on another project this code work perfectly

[lihaibh]

@RusTorg this error occurs when the size of a collection is not the same as the size of the data itself. The library currently need to know the file size inside the result tar file before writing into it. What happens actually is that you are getting a stream of collection data that is bigger than the size that was declared. can you find what collection is problematic?

you can use the collections property inside the "source" object. Try to run it multiple times to see if it is consistent.

If it is, the question is why the metadata of mongo is not synced with the data itself, we need to think of a solution for this end case, maybe limiting the stream size even if it's bigger.

[lihaibh]

@RusTorg can you give an update?

[paulbraam]

Hi @lihaibh

I'm getting this issue, too

could not write collection data into: type: local, path: ./backup/2021-02-20.tar due to error:
Error: size mismatch
    at Sink.<anonymous> (C:\Users\pavel\Desktop\botdev\bot\node_modules\tar-stream\pack.js:175:23)
    at Sink.f (C:\Users\pavel\Desktop\botdev\bot\node_modules\once\once.js:25:25)
    at Sink.onfinish (C:\Users\pavel\Desktop\botdev\bot\node_modules\end-of-stream\index.js:31:27)
    at Sink.emit (events.js:223:5)
    at finishMaybe (C:\Users\pavel\Desktop\botdev\bot\node_modules\readable-stream\lib\_stream_writable.js:624:14)
    at endWritable (C:\Users\pavel\Desktop\botdev\bot\node_modules\readable-stream\lib\_stream_writable.js:643:3)
    at Sink.Writable.end (C:\Users\pavel\Desktop\botdev\bot\node_modules\readable-stream\lib\_stream_writable.js:571:22)
    at SafeSubscriber._error (C:\Users\pavel\Desktop\botdev\bot\node_modules\mongodb-snapshot\dist\connectors\filesystem\FileSystemDuplexConnector.js:271:19)
    at SafeSubscriber.__tryOrUnsub (C:\Users\pavel\Desktop\botdev\bot\node_modules\mongodb-snapshot\node_modules\rxjs\internal\Subscriber.js:207:16)
    at SafeSubscriber.error (C:\Users\pavel\Desktop\botdev\bot\node_modules\mongodb-snapshot\node_modules\rxjs\internal\Subscriber.js:158:26)
    at Subscriber._error (C:\Users\pavel\Desktop\botdev\bot\node_modules\mongodb-snapshot\node_modules\rxjs\internal\Subscriber.js:92:26)
    at Subscriber.error (C:\Users\pavel\Desktop\botdev\bot\node_modules\mongodb-snapshot\node_modules\rxjs\internal\Subscriber.js:72:18)
    at MergeMapSubscriber.Subscriber._error (C:\Users\pavel\Desktop\botdev\bot\node_modules\mongodb-snapshot\node_modules\rxjs\internal\Subscriber.js:92:26)
    at MergeMapSubscriber.Subscriber.error (C:\Users\pavel\Desktop\botdev\bot\node_modules\mongodb-snapshot\node_modules\rxjs\internal\Subscriber.js:72:18)
    at MergeMapSubscriber.OuterSubscriber.notifyError (C:\Users\pavel\Desktop\botdev\bot\node_modules\mongodb-snapshot\node_modules\rxjs\internal\OuterSubscriber.js:26:26)
    at InnerSubscriber._error (C:\Users\pavel\Desktop\botdev\bot\node_modules\mongodb-snapshot\node_modules\rxjs\internal\InnerSubscriber.js:31:21)
    at InnerSubscriber.Subscriber.error (C:\Users\pavel\Desktop\botdev\bot\node_modules\mongodb-snapshot\node_modules\rxjs\internal\Subscriber.js:72:18)
    at C:\Users\pavel\Desktop\botdev\bot\node_modules\mongodb-snapshot\dist\connectors\MongoDBDuplexConnector.js:472:26
    at C:\Users\pavel\Desktop\botdev\bot\node_modules\mongodb\lib\cursor.js:741:11
    at handleCallback (C:\Users\pavel\Desktop\botdev\bot\node_modules\mongodb\lib\utils.js:102:55)
    at C:\Users\pavel\Desktop\botdev\bot\node_modules\mongodb\lib\operations\cursor_ops.js:97:23
    at C:\Users\pavel\Desktop\botdev\bot\node_modules\mongodb\lib\utils.js:684:9
    at C:\Users\pavel\Desktop\botdev\bot\node_modules\mongodb\lib\cursor.js:250:25
    at C:\Users\pavel\Desktop\botdev\bot\node_modules\mongodb\lib\core\cursor.js:739:9
    at done (C:\Users\pavel\Desktop\botdev\bot\node_modules\mongodb\lib\core\cursor.js:461:7)
    at C:\Users\pavel\Desktop\botdev\bot\node_modules\mongodb\lib\core\cursor.js:536:11
    at executeCallback (C:\Users\pavel\Desktop\botdev\bot\node_modules\mongodb\lib\operations\execute_operation.js:70:5)
    at callbackWithRetry (C:\Users\pavel\Desktop\botdev\bot\node_modules\mongodb\lib\operations\execute_operation.js:122:14)
removing: "type: local, path: ./backup/2021-02-20.tar" because an error was thrown during the transfer

The solution above doesn't work either

const transferer = new MongoTransferer({
      source: {
        collections: ['users']
      },
      targets: [mongo_connector],
  });

Please suggest how to workaround this. Thanks.

[lihaibh]

@paulbraam can you please give me the metadata of this collection? you can do it by simply run:

db.getCollection('users').stats()

The problem is that for some reason the size of the collection as we get from the collection metadata is not the right size. so if we try to write more data (in bytes) than the size of the collection you will have this error.

I can do a workaround fix for that by just cutting the stream of bytes to be exactly the size of the collection, but i dont think it is the right way of solving this issue.

You can take a look at the function packCollectionData$ (on file: src/connectors/filesystem/FileSystemDuplexConnector.ts) which pack a collection to the tar file. as you can see before we are streaming the data directly to the tar file, we first need to know the size of the file that we want to create inside the tar, the size is coming from the collection metadata. so for some reason the actual size of the collection and the size from the metadata is not synced, why is that?

[BradGamez]

I seem to be having the same issue with one of my databases. My other two databases are working perfectly fine for backing up. I've tried the things mentioned above as troubleshooting but I got the same result of mismatched size.

[bakdakonusuruz]

Hi there, I'm having the same issue when I first time tried to use it. I just have a test log db and just one collection in it. But it's in the Mongo Atlas Cluster, is it affecting in any way?

Edit: I just tried in a local docker container with Mongo and it works fine. Is it intentional?

[lihaibh]

@bakdakonusuruz @BradGamez maybe the collection modified during the transfer? is it possible? or the problem is consistent even if you try to perform the backup multiple times? i just want to understand the core of the problem. I managed to reproduce the error only when i added more documents to a collection during its backup.

[bakdakonusuruz]

@lihaibh I think the same thing. I added a document while I was trying to backup my local mongo instance and it gave the same error. So, it confirms your assumption.

But still not working on Atlas clusters. My Atlas Cluster is just a test DB and I'm pretty sure that I'm not modifying any document in Atlas Cluster.

[BradGamez]

@bakdakonusuruz @BradGamez maybe the collection modified during the transfer? is it possible? or the problem is consistent even if you try to perform the backup multiple times? i just want to understand the core of the problem. I managed to reproduce the error only when i added more documents to a collection during its backup.

The backup has tried to run every day since 2/17/2021 at 5am. However, is it very likely the collection is being modified during the transfer now that you mention it because it is a live database serving ~ 450,000 users. Is there a way around that since it is most likely that causing it?

[lihaibh]

@bakdakonusuruz @BradGamez I don't want to lock the database obviously so the only workaround for this problem is cutting out the extra data without backup it.

[BradGamez]

@lihaibh Sounds like a great option for my use case. How could I make that happen?

[lihaibh]

@BradGamez im working on a fix now

[BradGamez]

Any updates on the fix?

[lihaibh]

@BradGamez @bakdakonusuruz @paulbraam i made some changes and local testing, ive released this version: 1.2.2-rc.0 so please update your package.json file and ping me if everything is ok so i can close the branch.

[lihaibh]

@BradGamez any update?

[lihaibh]

@bakdakonusuruz @paulbraam can you please check this i think it solves the problem but i want to verify it before i release official version. Thanks.

[paulbraam]

Hi @lihaibh I will check it and let you know asap.

[arghyac35]

1.2.2-rc.0 so please update your package.json

How do I update it to this version? as it is not released in npm so it gives an error.

Also, I tried to change package.json like this: "mongodb-snapshot": "lihaibh/mongodb-snapshot#v1.2.2-rc.0", yarn cloud build it, but in code, it shows: Cannot find module 'mongodb-snapshot' or its corresponding type declarations.

Can you please say what I am doing wrong?

[lihaibh]

@arghyac35 you right, i didnt notice that the previous publish failed. You can try now

[arghyac35]

@lihaibh I tried it just now, unfortunately the error is still there:

[lihaibh]

@arghyac35 can you make sure in package.lock that you are using the right version? 1.2.2-rc.0

[arghyac35]

@arghyac35 can you make sure in package.lock that you are using the right version? 1.2.2-rc.0

Yes, I can see in noode_modules, that the version in 1.2.2-rc.0:

[paulbraam]

Hi @lihaibh I also confirm that the error is still there.

[lihaibh]

@paulbraam @arghyac35 The dist was not updated with the changes. Please try now with: 1.2.2-rc.2 thx.

[arghyac35]

Still the same error. I can confirm that my package is updated as you can see in yarn.lock:

[manorlh]

also for me. , any news about that?

[lihaibh]

@manorlh @arghyac35 there is a branch opened with the fix, but people report it still doesn't work for them, can someone help with the solution? according to my local testing (i tried to modify collections during the backup) and it fixed the problem, so i dont understand why this solution doesn't work for others, what i miss?

in order to reproduce the issue, just remove add or modify documents in collections during the backup

[naizapp]

I was been using this awesome module for last 3 months in aws instance running node js and the same instance serving mongodb. I've done things like backing up data from that aws instance, restoring that tar file into my laptop localhost and reverse too. While doing such things, sometimes there will be a difference of more than 10,000 records, and added collections, deleted collections etc.. None of those changes were a problem, and I still get exact thing when restored.

Iam writing these so that @lihaibh can understand that the problem is not due to change in the collection.

I started facing this problem when I migrated db to atlas. While migrating, I created a db at atlas, and just added one empty collection. Then after that, I used this same module to restore all data from the file I backed up. The restore was clean and clear and easy. Awesome.. All collections and documents got migrated. But this error appeared when I tried to take a backup from there.

Conclusion.. This error is happening while trying to backup from atlas. Regardless of whether it was restored or manually loaded or just one empty collection

[naizapp]

I'm not good at github. But I've tried pushing changes.

I just changed the tar-stream version to 2.2.0 and it is working good now.

[naizapp]

I was so stuck in my current project because of this issue. So I left with no option other than to fix this for myself. After lots of debugging, finally found a solution. I have sent a pull request.

[lihaibh]

im closing the current pull request as it looks like it works for some cases

[lihaibh]

@naizapp @manorlh @arghyac35 @paulbraam @BradGamez @RusTorg I made more fixes and added more features, you are welcome to use the library at version 1.3.0. Sorry for the delay. The errors (if any) should be more informative. Enjoy and open an issue if you have one. Kindly give a feedback if the solutions work for you.

Thank you for your cooperation.

[lihaibh]

Do you still have an issue?

[naizapp]

@lihaibh The current version from my temp forked branch is working smoothly now. I cannot risk disturbing it because it takes backup once daily. So I will start using the version 1.3.0 after two weeks for my project's beta update. Then I will give feedback for sure. Anyways thanks for this awesome library.

[lihaibh]

We can add a new connector for S3, if you want to backup directly to S3 instead of locally. Let me know if its a needed feature i will work on it.

[naizapp]

My specific use case is downloading the mongo bkp from Atlas and images backup from S3. Having backup within S3 sounds very useful to others 👍🏻

[naizapp]

[naizapp]

[lihaibh]

@naizapp you can change the for await (const ...) {} to return transferer.promise()

[peepo5]

if anyone still has this issue, run npm update or yarn update, whatever you use.

[lihaibh]

As i received no complaints about this error as i fixed it on the last pull request, this issue considered resolved.