Error migrating tombstoned documents: `Did not update document "DOC_ID" w/ xattr: invalid arguments`

[pradipvaghasiya]

Hello, After updating to Sync Gateway 1.5 with convergence enabled in our dev environment. I am seeing so many errors mentioned in subject.

After checking client calls I could see this error by simply reading the doc from Sync Gateway which could not be imported earlier due to some reason. I have tried restarting Sync Gateway few times but no luck. Cluster has just one SyncGateway and Couchbase node

I am also able to create the error in Sync Gateway by firing below curl command curl http://{{host}}:4984/dbName/DOC_ID

I also tried to retrieve the Couchbase Doc directly using below N1QL Query select * from dbName where meta().id = 'DOC_ID';

I could find the DOC_ID in database and it was in deleted state.

I am attaching Sync Gateway Error log and full document for DOC_ID here.

I could not find this exact issue being covered elsewhere, I could find one issue like this (#2663) but that was related to delete using SDK. As of now We don't use any SDK.

Couchbase Lite iOS: latest

Sync Gateway version: 1.5.0 Community

Operating system: Ubuntu 14 LTS

sync_gateway_log.txt

docN1QLQuery.txt

[adamcfraser]

Looking into this. Can you provide the following information:

• What build/version of Couchbase Server are you using?
• Your Sync Gateway config (sync function can be removed for privacy reasons if necessary)

[adamcfraser]

This looks like a legitimate bug during migration of existing tombstones to using extended attributes. Looking into the fix now.

[adamcfraser]

Migration is bypassing the standard tombstone xattr handling that's in https://github.com/couchbase/sync_gateway/blob/master/base/bucket_gocb.go#L1561 because it's using WriteCasWithXattr instead of WriteUpdateWithXattr (due to differences in CAS retry behaviour).

[adamcfraser]

Fixed on master by #3099 Fixed on 1.5.1 by #3100

[pradipvaghasiya]

Couchbase Server: 5.0 community Config: config.txt

[adamcfraser]

Thanks @pradipvaghasiya. I'll update this ticket when I get details on release scheduling for 1.5.1 - the goal is to push it out asap to include this fix.

[pradipvaghasiya]

Thanks to you @adamcfraser! Just to let you know I have also received many errors which says network error along with these as mentioned below. Network error docs are also in deleted state.

I am using same machine for sync gateway and couchbase node, not sure why network error.

2017-11-22T15:51:44.683Z Import: Error importing doc "doc_id": network error 2017-11-22T15:51:44.688Z Import: Error importing doc "doc_id": invalid arguments 2017-11-22T15:51:44.693Z Import: Error importing doc "doc_id": invalid arguments 2017-11-22T15:51:44.703Z Import: Error importing doc "doc_id": invalid arguments

[adamcfraser]

@pradipvaghasiya It would be useful to know whether you're still getting the 'network error' messages when using the 1.5.1 build. I'll see if I can get a developer preview build made available.

In the meantime, if you're able to provide a sanitized copy of the document that was associated with the 'network error' message, I can try to validate locally.

[pradipvaghasiya]

Attaching 2 docs for which network error was found in logs.

doc1.txt doc2.txt

1 More question around this. In Sync Gateway Config when doc gets deleted, I check the oldDoc and assign the deleted doc to correct channel so that mobile clients can sync deletes. I noticed that even if I don't assign deleted doc to any channel in config, Delete still gets synced properly. So my question is, is it necessary for us to assign channel to deleted docs?

[pradipvaghasiya]

Hi @adamcfraser In 1.5.1, There are still error messages appearing like below.

Import: Error importing doc "do-id-1": 404 missing Import: Error importing doc "do-id-2": 404 missing

These doc ids are deleted in database.

[pradipvaghasiya]

Hello @adamcfraser Is above comment useful? Should I raise new issue for this?