Error in Search feature while installing SW360 in Native method without docker & vagrant

[dineshr93]

Commit: f369528

Installation type: Native https://github.com/eclipse/sw360/wiki/Deploy-Natively

Have did the installation CouchDB lucene https://github.com/eclipse/sw360/wiki/Deploy-Natively#deploy-couchdb-lucene

1. when I click the top search box

url changes to http://ourserver/search/?p_p_lifecycle=0&p_p_mode=view&_sw360_portlet_components_javax.portlet.action=applyFilters&p_auth=nedYlp3u&q=boost

Displays below html page

2. When I click the components page search box

I get below page where search result is empty

I am not sure where I am making the mistake.

_design/lucene components
2020-08-05 17:02:01 ERROR LuceneAwareDatabaseConnector:120 - Error querying database.
 org.ektorp.DbAccessException: 400:Bad Request
URI: /_fti/local/sw360db/_design/lucene/components?include_docs=false&limit=150&q=%28+name%3A%28%22boost*%22+boost*%29+%29+
Response Body:
{
  "error" : "illegal_database_name",
  "reason" : "Name: '_fti'. Only lowercase characters (a-z), digits (0-9), and any of the characters _, $, (, ), +, -, and / are allowed. Must begin with a letter."
}
        at org.ektorp.http.StdResponseHandler.createDbAccessException(StdResponseHandler.java:50)
        at org.ektorp.http.StdResponseHandler.error(StdResponseHandler.java:91)
        at org.ektorp.http.RestTemplate.handleResponse(RestTemplate.java:126)

[dineshr93]

@mcjaeger Hi Michael, we separately tried to create a database names _fti

curl -X PUT localhost:5984/_fti But it returned below error

{"error":"illegal_database_name","reason":"Name: 'fti'. Only lowercase characters (a-z), digits (0-9), and any of the characters , $, (, ), +, -, and / are allowed. Must begin with a letter."}

But during the search, it is trying to search the component name under database name "_fti"

We listed all the database using

curl localhost:5984/_all_dbs

["_replicator","_users","fti","sw360_test_attachments","sw360_test_db","sw360_test_users","sw360_test_vm","sw360attachments","sw360changelogs","sw360config","sw360db","sw360oauthclients","sw360users","sw360vm"]

We created the "fti" for our testing purpose. It is succeeding in creating it but not _fti

we use CouchDB 1.6.1 as mentioned <2 here

[mcjaeger]

please use

• current master
• use couchdb 2.1
• use couchdb lucene
• apply patch with this script before deploying: https://github.com/sw360/sw360vagrant/blob/79a829c3b514d6a9f9a6bdff5ee74065b951442e/shared/scripts/install-lucene.sh

[mcjaeger]

for old couchdb please add the fti configuration as here:

https://github.com/sw360/sw360vagrant/blob/master/puppet/modules/sw360/templates/couchdb_local.ini.erb

[dineshr93]

for old couchdb please add the fti configuration as here:

https://github.com/sw360/sw360vagrant/blob/master/puppet/modules/sw360/templates/couchdb_local.ini.erb

As of now I tried only this step. After I restarted when I searched for component named "boost" & It threw below error

2020-08-19 13:58:00 ERROR LuceneAwareDatabaseConnector:120 - Error querying database.
 org.ektorp.DbAccessException: 500:Internal Server Error
URI: /_fti/local/sw360db/_design/lucene/components?include_docs=false&limit=150&q=%28+name%3A%28%22boost*%22+boost*%29+%29+
Response Body:
null
        at org.ektorp.http.StdResponseHandler.createDbAccessException(StdResponseHandler.java:50)
        at org.ektorp.http.StdResponseHandler.error(StdResponseHandler.java:91)
        at org.ektorp.http.RestTemplate.handleResponse(RestTemplate.java:126)
        at org.ektorp.http.RestTemplate.get(RestTemplate.java:22)
        at com.github.ldriscoll.ektorplucene.LuceneAwareCouchDbConnector.queryLucene(LuceneAwareCouchDbConnector.java:101)
        at org.eclipse.sw360.datahandler.couchdb.lucene.LuceneAwareDatabaseConnector.searchView(LuceneAwareDatabaseConnector.java:118)
2020-08-19 13:58:13.528 ERROR [liferay/search_writer/SYSTEM_ENGINE-3][ElasticsearchIndexWriter:364] Update failed
2020-08-19 13:58:13.533 ERROR [liferay/search_writer/SYSTEM_ENGINE-4][ElasticsearchIndexWriter:409] Bulk update failed

[mcjaeger]

did you install a patched version of couchdb-lucene?

[dineshr93]

@mcjaeger HI michael we patched it already during our initial setup

[SiddarthHS]

We installed latest SW360-11.0.0-M1(c27a46075912c1fd82bc7b0cbf3ef84ffd676960) using the following link https://github.com/eclipse/sw360/wiki/Deploy-Natively-11 with latest couchdb and couchdb-lucene, however we are getting same error.

2020-11-06 09:52:10 ERROR LuceneAwareDatabaseConnector:128 - Error querying database using _fti hook.nothing found on db path: /_fti/local/sw360db/_design/lucene/components?include_docs=false&limit=150&q=%28+name%3A%28%22Comp1*%22+Comp1*%29+%29+, Response body: {"error":"not_found","reason":"Database does not exist."}
2020-11-06 09:52:10 INFO LuceneAwareDatabaseConnector:129 - Trying to call lucene directly
2020-11-06 09:52:20 ERROR LuceneAwareDatabaseConnector:163 - Getting error with reponse code = 500.Retrying with stale parameter
2020-11-06 09:52:20 ERROR LuceneAwareDatabaseConnector:171 - Retried with stale parameter.Getting error with reponse code=400

[dineshr93]

@mcjaeger I guess it has to do something with either couchdb.properties or local.ini. we have listed both below

/etc/sw360/couchdb.properties

server:/etc/sw360$ cat couchdb.properties
#
# Copyright Siemens AG, 2013-2019. Part of the SW360 Portal Project.
#
# This program and the accompanying materials are made
# available under the terms of the Eclipse Public License 2.0
# which is available at https://www.eclipse.org/legal/epl-2.0/
#
# SPDX-License-Identifier: EPL-2.0
#

# N.B this is the default build property file, defined in module build-configuration

couchdb.url = http://localhost:5984
couchdb.user =
couchdb.password =
couchdb.database = sw360db
couchdb.usersdb = sw360users
couchdb.attachments = sw360attachments
couchdb.change_logs = sw360changelogs
couchdb.config = sw360config
couchdb.vulnerability_management = sw360vm
lucenesearch.limit = 150
couchdb.lucene.url = http://localhost:8080/couchdb-lucene

# Warning: If you enable lucene leading wildcards you have to enable this configuration also in couchdb-lucene.ini
# leading wildcard search is disabled as default because its a expensive operation.
# couchdb-lucene.ini (is part of the couchdb-lucene .war package)
# [lucene]
# allowLeadingWildcard=true
# see more: https://wiki.apache.org/lucene-java/LuceneFAQ#What_wildcard_search_support_is_available_from_Lucene.3F
lucenesearch.leading.wildcard = false

/etc/sw360/local.ini

server:/etc/sw360$ cat local.ini
; Modifications applied by Siemens AG 2015-2016.
; Modifications for supporting own infrastructure.
; Original file from couchdb project

; CouchDB Configuration Settings

; Custom settings should be made in this file. They will override settings
; in default.ini, but unlike changes made to default.ini, this file won't be
; overwritten on server upgrade.

[couchdb]
;max_document_size = 4294967296 ; bytes
uuid = 8123d83e39b4cb59c8b8928636842252
os_process_timeout=60000

[external]
fti=/usr/bin/java -jar /opt/couchdb-lucene-*-jar-with-dependencies.jar -search

[httpd]
port = 5984
bind_address = 127.0.0.1
; Options for the MochiWeb HTTP server.
;server_options = [{backlog, 128}, {acceptor_pool_size, 16}]
; For more socket options, consult Erlang's module 'inet' man page.
;socket_options = [{recbuf, 262144}, {sndbuf, 262144}, {nodelay, true}]

; Uncomment next line to trigger basic-auth popup on unauthorized requests.
;WWW-Authenticate = Basic realm="administrator"

; Uncomment next line to set the configuration modification whitelist. Only
; whitelisted values may be changed via the /_config URLs. To allow the admin
; to change this value over HTTP, remember to include {httpd,config_whitelist}
; itself. Excluding it from the list would require editing this file to update
; the whitelist.
;config_whitelist = [{httpd,config_whitelist}, {log,level}, {etc,etc}]

[query_servers]
;nodejs = /usr/local/bin/couchjs-node /path/to/couchdb/share/server/main.js

[httpd_global_handlers]
;_google = {couch_httpd_proxy, handle_proxy_req, <<"http://www.google.com">>}
_fti = {couch_httpd_proxy, handle_proxy_req, <<"http://127.0.0.1:8080/couchdb-lucene">>}

;[httpd_db_handlers]
;_fti = {couch_httpd_external, handle_external_req, <<"fti">>}

[couch_httpd_auth]
; If you set this to true, you should also uncomment the WWW-Authenticate line
; above. If you don't configure a WWW-Authenticate header, CouchDB will send
; Basic realm="server" in order to prevent you getting logged out.
; require_valid_user = false

[log]
;level = debug

[log_level_by_module]
; In this section you can specify any of the four log levels 'none', 'info',
; 'error' or 'debug' on a per-module basis. See src/*/*.erl for various
; modules.
;couch_httpd = error

[os_daemons]
; For any commands listed here, CouchDB will attempt to ensure that
; the process remains alive. Daemons should monitor their environment
; to know when to exit. This can most easily be accomplished by exiting
; when stdin is closed.
;foo = /path/to/command -with args

[mcjaeger]

I am not sure, one reason could be that the wildcard.allow.property is not set, pleachse check if that is in /etc/sw360/couchdb.properties as listed in the file below:

https://github.com/eclipse/sw360/blob/master/build-configuration/resources/couchdb.properties

[mcjaeger]

unfortunately I seeing the same error with search on the current master with current vagrant (did the testgn for tagging):

2020-11-10 21:43:39 ERROR LuceneAwareDatabaseConnector:128 - Error querying database using _fti hook.nothing found on db path: /_fti/local/sw360db/_design/lucene/all?include_docs=true&limit=150&q=%28%22testp*%22+testp*%29, Response body: {"error":"not_found","reason":"Database does not exist."}
2020-11-10 21:43:39 INFO  LuceneAwareDatabaseConnector:129 - Trying to call lucene directly
2020-11-10 21:43:49 ERROR LuceneAwareDatabaseConnector:163 - Getting error with reponse code = 500.Retrying with stale parameter
2020-11-10 21:43:49 ERROR LuceneAwareDatabaseConnector:171 - Retried with stale parameter.Getting error with reponse code=400

-> search is indeed not working at the moment

[mcjaeger]

@mcjaeger I guess it has to do something with either couchdb.properties or local.ini. we have listed both below

/etc/sw360/couchdb.properties

server:/etc/sw360$ cat couchdb.properties
#
# Copyright Siemens AG, 2013-2019. Part of the SW360 Portal Project.
#
# This program and the accompanying materials are made
# available under the terms of the Eclipse Public License 2.0
# which is available at https://www.eclipse.org/legal/epl-2.0/
#
# SPDX-License-Identifier: EPL-2.0
#

# N.B this is the default build property file, defined in module build-configuration

couchdb.url = http://localhost:5984
couchdb.user =
couchdb.password =
couchdb.database = sw360db
couchdb.usersdb = sw360users
couchdb.attachments = sw360attachments
couchdb.change_logs = sw360changelogs
couchdb.config = sw360config
couchdb.vulnerability_management = sw360vm
lucenesearch.limit = 150
couchdb.lucene.url = http://localhost:8080/couchdb-lucene

# Warning: If you enable lucene leading wildcards you have to enable this configuration also in couchdb-lucene.ini
# leading wildcard search is disabled as default because its a expensive operation.
# couchdb-lucene.ini (is part of the couchdb-lucene .war package)
# [lucene]
# allowLeadingWildcard=true
# see more: https://wiki.apache.org/lucene-java/LuceneFAQ#What_wildcard_search_support_is_available_from_Lucene.3F
lucenesearch.leading.wildcard = false

/etc/sw360/local.ini

server:/etc/sw360$ cat local.ini
; Modifications applied by Siemens AG 2015-2016.
; Modifications for supporting own infrastructure.
; Original file from couchdb project

; CouchDB Configuration Settings

; Custom settings should be made in this file. They will override settings
; in default.ini, but unlike changes made to default.ini, this file won't be
; overwritten on server upgrade.

[couchdb]
;max_document_size = 4294967296 ; bytes
uuid = 8123d83e39b4cb59c8b8928636842252
os_process_timeout=60000

[external]
fti=/usr/bin/java -jar /opt/couchdb-lucene-*-jar-with-dependencies.jar -search

[httpd]
port = 5984
bind_address = 127.0.0.1
; Options for the MochiWeb HTTP server.
;server_options = [{backlog, 128}, {acceptor_pool_size, 16}]
; For more socket options, consult Erlang's module 'inet' man page.
;socket_options = [{recbuf, 262144}, {sndbuf, 262144}, {nodelay, true}]

; Uncomment next line to trigger basic-auth popup on unauthorized requests.
;WWW-Authenticate = Basic realm="administrator"

; Uncomment next line to set the configuration modification whitelist. Only
; whitelisted values may be changed via the /_config URLs. To allow the admin
; to change this value over HTTP, remember to include {httpd,config_whitelist}
; itself. Excluding it from the list would require editing this file to update
; the whitelist.
;config_whitelist = [{httpd,config_whitelist}, {log,level}, {etc,etc}]

[query_servers]
;nodejs = /usr/local/bin/couchjs-node /path/to/couchdb/share/server/main.js

[httpd_global_handlers]
;_google = {couch_httpd_proxy, handle_proxy_req, <<"http://www.google.com">>}
_fti = {couch_httpd_proxy, handle_proxy_req, <<"http://127.0.0.1:8080/couchdb-lucene">>}

;[httpd_db_handlers]
;_fti = {couch_httpd_external, handle_external_req, <<"fti">>}

[couch_httpd_auth]
; If you set this to true, you should also uncomment the WWW-Authenticate line
; above. If you don't configure a WWW-Authenticate header, CouchDB will send
; Basic realm="server" in order to prevent you getting logged out.
; require_valid_user = false

[log]
;level = debug

[log_level_by_module]
; In this section you can specify any of the four log levels 'none', 'info',
; 'error' or 'debug' on a per-module basis. See src/*/*.erl for various
; modules.
;couch_httpd = error

[os_daemons]
; For any commands listed here, CouchDB will attempt to ensure that
; the process remains alive. Daemons should monitor their environment
; to know when to exit. This can most easily be accomplished by exiting
; when stdin is closed.
;foo = /path/to/command -with args

This appears to be the ini for couchdb 1.X I am not sure if that is mixing couchdb-lucene 2.1 with couchdb 1.X which causes a problem, it should work, but I never tested it.

[mcjaeger]

search seem to work for me now. I think it is the issue of constructing the index right after first search (which is not ready when submitting a few searches right after). Same instance works this morning. @dineshr93 could you pls try again on your instance?

[dineshr93]

I am not sure, one reason could be that the wildcard.allow.property is not set, pleachse check if that is in /etc/sw360/couchdb.properties as listed in the file below:

https://github.com/eclipse/sw360/blob/master/build-configuration/resources/couchdb.properties

Hi @mcjaeger We set wild card property to true in both /etc/sw360/couchdb.properties & in /opt/liferay-ce-portalxxx-ga2/tomcat-9.0.17/webapps/couchdb-lucene/WEB-INF/classes/couchdb-lucene.ini

Search feature worked successfully Thanks for your info.

However the log is spitting some error message

2020-11-11 10:12:03 INFO  SearchPortlet:70 - typeMask set to emptyList
_design/lucene all
2020-11-11 10:12:03 ERROR LuceneAwareDatabaseConnector:128 - Error querying database using _fti hook.nothing found on db path: /_fti/local/sw360users/_design/lucene/all?include_docs=true&limit=150&q=%28+type%3Auser+%29+AND+%28%22*test*%22+*test*%29, Response body: {"error":"not_found","reason":"Database does not exist."}
2020-11-11 10:12:03 INFO  LuceneAwareDatabaseConnector:129 - Trying to call lucene directly
2020-11-11 10:12:13 ERROR LuceneAwareDatabaseConnector:163 - Getting error with reponse code = 500.Retrying with stale parameter
_design/lucene all
2020-11-11 10:12:13 ERROR LuceneAwareDatabaseConnector:128 - Error querying database using _fti hook.nothing found on db path: /_fti/local/sw360db/_design/lucene/all?include_docs=true&limit=150&q=%28%22*test*%22+*test*%29, Response body: {"error":"not_found","reason":"Database does not exist."}
2020-11-11 10:12:13 INFO  LuceneAwareDatabaseConnector:129 - Trying to call lucene directly
2020-11-11 10:12:23 ERROR LuceneAwareDatabaseConnector:163 - Getting error with reponse code = 500.Retrying with stale parameter

Is this point of concern ?

[mcjaeger]

the error message is normal, because during all the migration from couchdb 1 to 2, we decided to have the search support both versions at the same time. So it uses the couchdb 1 fti hook mechanism first and fails and tries the couchdb 2 way.

No concern, maybe at some time, couchdb 1 support shall be dropped, maybe the log level (error) is too aggressive.

[mcjaeger]

@dineshr93 OK to close this?

[dineshr93]

@mcjaeger yes Michael..Thanks..