After upgrade version from 2.2.0 to 4.0.0 Search stopped working

[TatyanaBol]

Please read these guidelines before opening an issue.

Bug Description

When set cloudant package version to 4.0.0, search stopped working

1. Steps to reproduce and the simplest code sample possible to demonstrate the issue

const params = {
  "q": "((name_x:test* OR name_x:test)) AND instance_id:\"crn:v1:staging:public:service:us-south:a/1234567897894561230123456:3871aeec-1d11-4369-bcb0-33784e1009fe::\"",
  "sort": [ "expires_on<number>", "_id<string>" ],
  "limit": 100,
  "include_fields": [  "name",  "domains",  "status" ]
}
       db.search("designName", "indexName", params) 

2. What you expected to happen

This code worked with all previous version, expected to work

3. What actually happened

throws error {"name": "Error", "error": "bad_request", "reason": "invalid UTF-8 JSON", "scope": "couch", "statusCode": 400, }

I tried to run the query in Postman. It fails with GET request with the same error, but succeeds with POST request (params in body).

Environment details

• Version(s) that are affected by this issue. v4.0.0
• Node.js version v8.13.0

[TatyanaBol]

I checked - our code works with 3.0.1 (and earlier version) and doesn't work with 3.0.2 (and later)

[ricellis]

That seems very strange, the only change between 3.0.1 and 3.0.2 was the removal of an un-needed import @types/nano. Are you using typescript?

I am not able to reproduce this problem using the query parameters you provided. Is the failing query always this same query? The 400 invalid JSON seems like perhaps there is an encoding issue with query parameters, but as I said, I can't get a 400 from encoding the query parameters you provided above.

[TatyanaBol]

@ricellis I checked it again, I was wrong, in 3.0.2 it works. Sorry. It stopped working from 4.0.0. We changed only cloudant package version and started to get the error above. The error reproduced with our param in v 4.0.0

[TatyanaBol]

@ricellis did you succeed to reproduce the error?

[ricellis]

Our support team reproduced the issue, it appears to be a regression in the Nano dependency.

[TatyanaBol]

@ricellis Thank you for your answer Do you have any estimation when it will be fixed?

[ricellis]

No, I don't, sorry. The issue is around repeated query string parameters, which it appears are acceptable (and indeed required) in CouchDB for some parameters, but not others. The change in Nano fixed issues with multiple drilldown parameters, but at the expense of breaking other parameters. We can't currently issue a request with both types of parameters handled correctly using the request library so it's going to take some time to figure something.

[ricellis]

We think making an upstream change in Nano to support POSTing to search is the right way to go here to get everything working again and address the regression that was introduced into Nano. We'll update this issue again soon with a link to a PR in Nano where you can follow the progress.

[ricellis]

See https://github.com/apache/couchdb-nano/pull/157

[brianewilkins]

The error can be simply reproduced by running the program repro.js below. Before running the program the following environment variables must be set: ACCOUNTNAME - the name of a Cloudant account PASSWORD - the password of the Cloudant account DATABASE - the name of a database in the Cloudant account that contains a search index _design/designName/_search/indexName

Here is the program repro.js which reproduces the error:

var Cloudant = require('@cloudant/cloudant');

var me = process.env.ACCOUNTNAME;
var password = process.env.PASSWORD;
var dbname = process.env.DATABASE;

// Initialize the library with my account.
var cloudant = Cloudant({ account: me, password: password });

db = cloudant.db.use(dbname, (err, data) => {
    if (err) {
        console.log(err);
    } else {
        console.log(data);
    }
});

const params = {
  "q": "anyfield:anyterm",     
  "include_fields": ["name"]
};

db.search("designName", "indexName", params, (err, data) => {
    if (err) {
        console.log(err);
    } else {
        console.log(data);
    }
});

Here is what appeared when I ran the program (except with the name of the Cloudant account replaced by $ACCOUNTNAME):

$ node repro.js
{ Error: invalid UTF-8 JSON
    at Object.clientCallback (/Users/brianwilkins/Documents/cloudant/cases/fb118796/node_modules/nano/lib/nano.js:154:15)
    at Request._callback (/Users/brianwilkins/Documents/cloudant/cases/fb118796/node_modules/@cloudant/cloudant/lib/clientutils.js:164:11)
    at Request.self.callback (/Users/brianwilkins/Documents/cloudant/cases/fb118796/node_modules/request/request.js:185:22)
    at Request.emit (events.js:197:13)
    at Request.self._source.emit (/Users/brianwilkins/Documents/cloudant/cases/fb118796/node_modules/@cloudant/cloudant/lib/eventrelay.js:78:21)
    at Request.<anonymous> (/Users/brianwilkins/Documents/cloudant/cases/fb118796/node_modules/request/request.js:1161:10)
    at Request.emit (events.js:197:13)
    at Request.self._source.emit (/Users/brianwilkins/Documents/cloudant/cases/fb118796/node_modules/@cloudant/cloudant/lib/eventrelay.js:78:21)
    at IncomingMessage.<anonymous> (/Users/brianwilkins/Documents/cloudant/cases/fb118796/node_modules/request/request.js:1083:12)
name: 'Error',
error: 'bad_request',
reason: 'invalid UTF-8 JSON',
scope: 'couch',
statusCode: 400,
request:
{ method: 'GET',
    headers:
    { 'content-type': 'application/json',
        accept: 'application/json' },
    uri:
    'https://XXXXXX:XXXXXX@$ACCOUNTNAME.cloudant.com/fb118796/_design/designName/_search/indexName',
    qs: { q: 'anyfield:anyterm', include_fields: [Array] },
    qsStringifyOptions: { arrayFormat: 'repeat' } },
headers:
{ uri:
    'https://XXXXXX:XXXXXX@$ACCOUNTNAME.cloudant.com/fb118796/_design/designName/_search/indexName',
    statusCode: 400,
    'cache-control': 'must-revalidate',
    'content-type': 'application/json',
    date: 'Thu, 25 Apr 2019 09:41:52 GMT',
    'x-couch-request-id': '412ce40562',
    'x-frame-options': 'DENY',
    'strict-transport-security': 'max-age=31536000',
    'x-content-type-options': 'nosniff',
    'x-cloudant-request-class': 'query',
    'x-cloudant-backend': 'bm-cc-dal-02',
    via: '1.1 lb1.bm-cc-dal-02 (Glum/1.77.7)' },
errid: 'non_200',
description: 'couch returned 400' }

[ricellis]

Thanks @brianewilkins we used your reproduce a week ago to find the problem. The fix is in PR state upstream already.

[ricellis]

@TatyanaBol Apache CouchDB Nano 8.1.0 has been released with a fix for this. The nodejs-cloudant package.json has "nano": "^8.0.0" so the fix should get picked up automatically if you reinstall or upgrade dependencies.