Push replication stops with error `400 bad_request`

[pahmed]

I have an issue where the the push replicator stops after receiving this error

Error Domain=CBLHTTP Code=400 "400 bad_request" UserInfo={NSURL=http://xxx:4984/xxx/contact%3AE0294D55-EB14-4C71-8100-98B71E772C01?new_edits=false, NSLocalizedFailureReason=bad_request, NSLocalizedDescription=400 bad_request}

When i tried to get that document, i didn't find it. I noticed also that after terminating and reopening again, the replicator pushes some changes and stops again. I tried disabling ForestDB but that didn't solve the problem.

I have enabled logging SyncVerbose, RemoteRequest and attached the logs sync_logs.txt

---

• Version: CBL 1.2
• Client OS: iOS 9.3.1
• Server: 3.0.1 Community Edition
• Syncgateway: 1.2

[brendand]

Good idea. I intend on writing a simple Mac app to let my customers view and setup user accounts on Sync Gateway so they can run their own instances. I just haven't had a chance to work on that yet. It was pretty easy to login to Cloudant and create databases using their REST API and the infrastructure is already there. It would be nice to have some alternatives I could integrate with as well. But initially it will be Cloudant, then Sync Gateway. And of course, I have peer-to-peer sync because of the awesome work you guys do there. I'm happy to send you a beta of my app if you're interested. You'd at least be able to see how I've integrated with Couchbase in my Mac and iOS apps.

[willholley]

I just tested this against CouchDB 2.0 and get a similar 'expected more data' error. I think this is worth raising as a CouchDB bug and the fix will then filter through to Cloudant.

[willholley]

actually, I spoke to soon - user error. Do you have a complete reproduction of the issue using curl?

[brendand]

I just attempted to test it using the CouchDB 2.0 service (Porter) on Cloudant's servers and it failed with a different error:

curl -X PUT -H 'Content-Type: multipart/related; boundary="2D0DAB20-BE70-4161-8A3E-CBF31CB525E6"' --data-binary @mime.txt 'http://tapforms.cloudant.com/attachment-test/facbfe2292a55958dae426f732021ac1?new_edits=false'

{"error":"missing_stub","reason":"Invalid attachment stub in facbfe2292a55958dae426f732021ac1 for attach1"}

Not sure if I did something wrong. But you can try it yourself as I've made that database public for now.

[snej]

@willholley: Yes, see my earlier comment with a reproduction.

[brendand]

I've actually seen this error coming from my app as well. Not sure what causes it.

I just tried @snej example above again to his database and it failed again with the attachment longer than expected error. But to my CouchDB 2.0 account I get the invalid attachment stub error.

[brendand]

FYI, the document I created for the test is pretty much empty except for the id and rev:

{
  "_id": "facbfe2292a55958dae426f732021ac1",
  "_rev": "1-967a00dff5e02add41819138abb3284d"
}

[willholley]

Just to confirm, are the complete steps simply:

$ curl 'http://localhost:5984/attachment-test' -XPUT 

$ curl -X PUT -H 'Content-Type: multipart/related; boundary="2D0DAB20-BE70-4161-8A3E-CBF31CB525E6"' --data-binary @mime.txt 'http://localhost:5984/attachment-test/mydoc-485135379?new_edits=false' 

That fails for me in CouchDB 1.6 with {"error":"missing_stub","reason":"id:mydoc-485135379, name:attach1"}

[willholley]

just another test - it works against CouchDB 1.6, CouchDB 2.0, Cloudant (current) and Cloudant (CouchDB 2.0) if I remove the attach1 entry from the "attachments" object. So it looks as though the stub is invalid. presumably because it references an attachment which doesn't exist on the server? Perhaps that's just an indication that the test is incomplete.

[brendand]

Oh right, I never noticed that attach1 had nothing in it in the mime.txt file. I guess I'll add an attach1 attachment in the Cloudant dashboard to see what happens.

[brendand]

I uploaded a file called Braveheart.jpg to my account and then modified the mime.txt file as follows:

{"_attachments":{"Braveheart.jpg":{"revpos":6,"stub":true},"attach2":{"content_type":"text/plain; charset=utf-8","digest":"sha1-B+wEdNQykhYbu8TdjWNY48pB4ZY=","follows":true,"length":3072,"revpos":3},"attach3":{"content_type":"text/plain; charset=utf-8","digest":"sha1-0sH/15NY0Tn+60Yhujv5dkWpLi8=","follows":true,"length":9216,"revpos":4}},"_id":"mydoc-485135379","_rev":"4-34a83e2f0f66471623d1e736ae27b8cf","_revisions":{"ids":["34a83e2f0f66471623d1e736ae27b8cf","2a4060d488884f608771f637e73b4278","ea73ed3f7d71d204e04f119b4cae842d"],"start":4},"foo":"bar"}

The current document looks like this:

{
  "_id": "facbfe2292a55958dae426f732021ac1",
  "_rev": "6-66169fcb7529c9611c46adaec61e6412",
  "_attachments": {
    "Braveheart.jpg": {
      "content_type": "image/jpeg",
      "revpos": 6,
      "digest": "md5-fgj35FyJnJFnsrX/O0UZNg==",
      "length": 292472,
      "stub": true
    }
  }
}

So I think the way I modified the mime.txt file was ok. It matches the document's revpos and attachment name.

But I got that invalid stub error anyway:

{"error":"missing_stub","reason":"Invalid attachment stub in facbfe2292a55958dae426f732021ac1 for Braveheart.jpg"}

This was against the Cloudant CouchDB 2.0 server.

[snej]

@willholley no, the complete steps are simply

1. Download the mime.txt file from my earlier comment to your disk
2. Enter the exact curl command I gave in that comment.

[willholley]

ok - I replicated the attachment-test database to my own accounts on CouchDB 1.X and CouchDB 2.X based Cloudant clusters. The good news is that it works on the CouchDB 2.X based builds.

$ curl -X PUT -H 'Content-Type: multipart/related; boundary="2D0DAB20-BE70-4161-8A3E-CBF31CB525E6"' --data-binary @mime.txt 'https://willholley-couch20.cloudant.com/attachment-test/mydoc-485135379?new_edits=false'
{"ok":true,"id":"mydoc-485135379","rev":"4-34a83e2f0f66471623d1e736ae27b8cf"}

vs

$ curl -X PUT -H 'Content-Type: multipart/related; boundary="2D0DAB20-BE70-4161-8A3E-CBF31CB525E6"' --data-binary @mime.txt 'https://willholley.cloudant.com/attachment-test/mydoc-485135379?new_edits=false'
{"error":"bad_request","reason":"attachment longer than expected"}

[brendand]

Did you still have to remove attach1 to get past the missing_stub error?

[willholley]

@brendand no - this was using exactly the database, mime.txt and curl command that @snej reproduced the error with.

[brendand]

hmm... I'm not sure what I'm doing wrong then. I didn't use exactly the same database as Jens but I thought I had modified the mime.txt file appropriately to match my database. How did you use his database exactly? Were you able to setup a replication on the Cloudant dashboard to copy it?

[willholley]

Jens' database is public - I replicated it to my account (you can use the Cloudant dashboard to do this). On 27 May 2016 20:54, "Brendan Duddridge" notifications@github.com wrote:

hmm... I'm not sure what I'm doing wrong then. I didn't use exactly the same database as Jens but I thought I had modified the mime.txt file appropriately to match my database. How did you use his database exactly? Were you able to setup a replication on the Cloudant dashboard to copy it?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/couchbase/couchbase-lite-ios/issues/1207#issuecomment-222239058, or mute the thread https://github.com/notifications/unsubscribe/AAF9-xAYT8kJ4br2x9SoeK6YFeYb9MJOks5qF0wPgaJpZM4IBSMa .

[sellingsolutions]

I just wanted to let you guys know that I get this bug for all my users every single day and I'm on: Couchbase Server 4.1.0-5005 Enterprise Edition (build-5005 Couchbase Sync Gateway/1.2.1(4;26c202a) CouchbaseLite 1.2.1

Right now I'm monitoring their CBL logs and when I see something like:

2016-06-02 10:58:13.443 MyApp[358:169878] WARNING: CBLRestPusher[https://....:*****@....]: _bulk_docs got an error: {
    error = "bad_request";
    id = "photo_brf_ursvikskulle_5DC3DEDD-98C6-42F6-9AD4-F93621F08E58";
    reason = "Missing digest in stub attachment \"signatur\"";
    status = 400;
}
2016-06-02 10:58:13.444 MyApp[358:169878] Sync: CBLRestPusher[https://https://....:*****@....] Progress: set error = 400 bad_request
2016-06-02 10:58:13.444 MyApp[358:169878] Sync: CBLRestPusher[https://....:*****@....] STOPPING...
2016-06-02 10:58:13.445 MyApp[358:169878] Sync: Stopping 36 remote requests

I simply extract the documentID and purge that document, the loss of one document is something I just have to accept in order to get the other 10,000 documents that the replicator might stop from syncing only because one attachment isn't "OK".

Who do I need sleep with to get a 'stopReplicatorOnError' : false option in the CBLReplicator ? That would solve almost all of my problems with users not being able to sync anything at all past that one broken attachment.

[snej]

@sellingsolutions: We've already got two different bugs being discussed in this issue (one reported by @pahmed, the other by @brendand.) A Bad Request error from the remote server is a very generic condition that can have multiple causes — the most recent discussion here has involved what looks like a bug in Cloudant's servers, which is clearly unrelated to your problem.

The best step you can take is to file your own issue, and provide as much data as you can about what's going on, especially about how those attachments were created.

[snej]

@willholley @brendand: Any update on this?

[willholley]

from the Cloudant side I think this is fixed as we roll out CouchDB 2.0-based builds. We're doing this with appropriate caution but customers can ask to be placed on a cluster with that build by raising a ticket with Cloudant support. I don't think there's anything to do on the CBL side.

[brendand]

I have been using the workaround on CBLRestPusher to avoid this problem:

    _dontSendMultipart = YES;

I can't really ask my customers to use any particular Cloudant server. I'm not sure exactly which ones are CouchDB 2.0 just yet or how long it will take to migrate them all over. So to be on the safe side I've been using this workaround.

The biggest issue right now is that Cloudant can't handle requests that are more than 10 MB big. I don't know if that's because I'm not using multi-part at the moment or if that's just a setting they have configured to reject large attachments.

@willholley , is there any definitive answer on how big an attachment and/or request can be with Cloudant? Is it safe to re-enable multi-part requests?

[willholley]

@brendand Cloudant's maximum request size is 64MB. We don't have a restriction on attachment size but we do intend to implement a maximum document (JSON) size in future - likely 1MB.

[snej]

It sounds like there's no action for CBL to take, so I'm going to close this issue. Thanks for your help, Will!

[brendand]

Hmm... 1 MB seems kind of small for a JSON document. What's the purpose for that? That would definitely prevent me from using the _dontSendMultipart workaround for the Cloudant multi-part parsing bug I'm using right now.

[willholley]

@brendand we wouldn't introduce any limit until after the CouchDB 2.0 rollout§. 1MB is actually pretty large for a JSON doc in my experience (remember 1MB doesn't include attachment data). What's your use case?

[brendand]

@willholley I've sent you an email so we can take this discussion off this issue report.

[rizwanmcs]

Hi i am using IBM watson api of visual recognition here is my code

`if ( isset($_FILES['uploadedfile']) && $_POST!="" ) { $fileData = $_FILES['uploadedfile']['name']; $post_data = array( 'image_file' => $fileData ); $url = 'https://gateway-a.watsonplatform.net/visual-recognition/api/v3/collections/searchItems_c5c517/images?api_key=655e41188e967ce58edfasde0b67be3ebfaadsfsdd22e38a34e&version=2016-05-20'; $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, $post_data); $headers = array(); $headers[] = "Content-Type: multipart/form-data"; $headers[] = "Accept: application/json"; curl_setopt($ch, CURLOPT_HTTPHEADER, $headers); $result = curl_exec($ch); if (curl_errno($ch)) { echo 'Error:' . curl_error($ch);die; }

` when i run this file and upload image i always receive the following error `string(59) "{ "error": "Missing multipart/form-data", "code": 400 }" bool(true)` so if there is any solution then please let me know Thank you

[mikemliu]

1.4 references this issue fixed. What exactly was modified?

I am also getting a similar error involved with multiple photo attachments stopping push replication with a "400 MIME part #3 doesn't match any attachment". Once this happens subsequent push replications all fail, presumably stopping at this document.