Deleted docs at cloudant.com are not replicated to device(deleted on device too)

[defaultbr]

my pull:

 Map<String, String> parameters = new HashMap<String, String>();
        parameters.put("doc_type", doc_type);
        PullFilter filter = new PullFilter("filters/getByDocType", parameters);
        Replicator replicator = ReplicatorBuilder.pull()
                .from(uri)
                .to(documentStore)
                .filter(filter)
                .withId(id).build();

my filter at cloudant:

function (doc, req) { 
     if(doc.doc_type === req.query.doc_type) {
     return true;
  }

  return false;
}

i tried a workaround:

    if(doc.deleted) {
        return true;
    }

but no success, document at device are not deleted:

I/System.out: REV:{ id: aba05d7513d30f8c31d4af175ea657ad, rev: 1-82435da0a023e2e218549fb37dfdbce3, seq: 38, parent: -1, current: true, deleted false }

thru url: /mydb/aba05d7513d30f8c31d4af175ea657ad

{"error":"not_found","reason":"deleted"}

the only thing i can guess, is that the latest revision (deleted one) has no attr doc_type, so its not returned by the return( when doc_type = req.query.doc_type), thats why i tried with no success the workaround above

how can i handle this deleted docs and replicate even the deletion to the device?

[tomblench]

@defaultbr it looks like you made a small mistake in the "workaround" for your filter, you need to use _deleted instead of deleted:

(using the animaldb database as an example)

{
  "_id": "_design/animal",
  "_rev": "3-bf2aa5336257f9e99f3dab823129392d",
  "filters": {
    "by_class": "function(doc, req) { if (doc._deleted == true) {return true;} if(doc.class && doc.class == req.query.class) { return true; } return false; }",
  }
}

Now I can set up a filter:

        PullFilter filter = new PullFilter("animal/by_class",
                CollectionFactory.MAP.of("class", "mammal"));

If I pull replicate, then delete a document, I can see the delete reflected in my database.

I agree that this is a work-around because you will end up replicating all deleted documents and not just the ones for the document type you want. But the problem is that once the document has been deleted, it has no body so you can't tell what the type is.

[tomblench]

@defaultbr a colleague suggested another approach which is to write a "tombstone" document. You can perform a PUT on your couch/cloudant instance with the _deleted flag set:

{"_id": "foo", "_rev": "123-abc", "_deleted": true, "class": "mammal"}

In your case the tombstone would have "doc_type". The key is to include the field which your filter depends on.

[defaultbr]

I got it, i had another light about this:

1) Pull all docs by doc_type and not deleted (or by_class you created above)

2) Pull all deleted docs by ids that i pass to the filter -> query for non deleted docs inside the device -> get all IDS: -> pull from getDeletedFromIdsList(pass the list as parameter to the filter)

so filter will update the docs that i have and i will not take the unecessary files, ex: something deleted that i dont have, what do you think?

@tomblench is there any way to REMOVE the doc from device(not _delete flag), ex: take all tombstone and delete it to free some disk space? cuz i have a incremental database and i replicate the docs from today (i pass the date parameter to filter), but i must really remove the yesterday docs so i can have free disk space

EDIT

Something like this:

at cloudant:

function (doc, req) { 
    if(req.query.doc_ids.indexOf(doc.id) > -1) {
        return true;
    }
  return false;
}

at the device:

String[] ids = getDocsIds....//['foo','bar'...]
        Map<String, String> parameters = new HashMap<String, String>();
        parameters.put("doc_ids", Arrays.toString(ids));
        PullFilter filter = new PullFilter("filters/getDeletedDocsByIds", parameters);
        Replicator replicator = ReplicatorBuilder.pull()
                .from(uri)
                .to(documentStore)
                .filter(filter)
                .withId(id).build();
        return replicator;

im just thinking about the query size and limit of how long this 'doc_ids' can get ( i can get ids filtering by doc_type and do another replicator for other doc_types.. etc..)

[tomblench]

Firstly, you are correct that you will run into problems using the document IDs as parameters to a filter. They are passed in the query part of the URL and you will soon run into the path length limit (I can't remember what is is off-hand for Cloudant but we have definitely seen other users run into similar issues).

Secondly, what you refer to as "REMOVE" is called purge in CouchDB parlance. You can see from the link that there lots of caveats associated with purging documents. Currently purging is not supported in sync-android. However, if you do a compact() after deleting a document then only minimal metadata about the revision history is retained - on the order of tens of bytes per revision. Whether this suffices for you will depend on your data model and the number of documents/revisions you expect to create and delete.

[defaultbr]

Solved this splitting the replicator in parts to avoid to explode get path limit