search

eclipse-sw360 / sw360

SW360 project
https://www.eclipse.org/sw360/
Other
123 stars 98 forks source link

Backend /thrift endpoints do not exist #2289

Open riamaria opened 10 months ago

riamaria commented 10 months ago

Description

Because going to http://localhost:8080/users/thrift resolves into a 404, I cannot save users into CouchDB.

How to reproduce

  1. Fresh clone
  2. In docker_build.sh, change all instances of docker-image://${DOCKER_IMAGE_ROOT} to docker-image://ghcr.io/${DOCKER_IMAGE_ROOT}
  3. Run ./docker_build.sh
  4. Run docker-compose up -d
  5. Go to http://localhost:8080
  6. Follow steps in https://eclipse.dev/sw360/docs/deployment/legacy/deploy-liferay7.4/ including "Import User Accounts for Testing" steps
  7. Notice that the imported user accounts could not be saved onto CouchDB
  8. Go to http://localhost:8080/users/thrift. Notice the 404 error page.

Screenshots

http://localhost:8080/users/thrift image

Versions

Logs

Front end logs

/app/sw360/tomcat/logs/catalina.out

10:30:16,839 ERROR [UserPortlet:775] Could not fetch users from backend!
org.apache.thrift.transport.TTransportException: HTTP Response code: 404
        at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:382) ~[bundleFile:?]
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73) ~[bundleFile:?]
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) ~[bundleFile:?]
        at org.eclipse.sw360.datahandler.thrift.users.UserService$Client.send_getUsersWithPagination(UserService.java:527) ~[bundleFile:?]
        at org.eclipse.sw360.datahandler.thrift.users.UserService$Client.getUsersWithPagination(UserService.java:518) ~[bundleFile:?]
        at org.eclipse.sw360.portal.portlets.admin.UserPortlet.getFilteredUsersList(UserPortlet.java:766) ~[bundleFile:?]
        at org.eclipse.sw360.portal.portlets.admin.UserPortlet.serveUsersListPresentInCouchDB(UserPortlet.java:696) ~[bundleFile:?]
        at org.eclipse.sw360.portal.portlets.admin.UserPortlet.serveResource(UserPortlet.java:574) ~[bundleFile:?]
        at com.liferay.portlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:146) ~[portal-impl.jar:?]
        at com.liferay.portlet.ScriptDataPortletFilter.doFilter(ScriptDataPortletFilter.java:88) ~[portal-impl.jar:?]
        at com.liferay.portlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:140) ~[portal-impl.jar:?]
        at com.liferay.portal.kernel.portlet.PortletFilterUtil.doFilter(PortletFilterUtil.java:78) ~[portal-kernel.jar:?]
        at com.liferay.portal.kernel.servlet.PortletServlet.service(PortletServlet.java:115) ~[portal-kernel.jar:?]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) ~[servlet-api.jar:4.0.FR]

Backend logs

/app/sw360/tomcat/logs/localhost.2024-01-25.log

25-Jan-2024 10:29:20.479 INFO [main] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath
25-Jan-2024 10:29:20.664 INFO [main] org.apache.catalina.core.ApplicationContext.log Marking servlet [UserService] as unavailable
25-Jan-2024 10:29:20.664 SEVERE [main] org.apache.catalina.core.StandardContext.loadOnStartup Servlet [UserService] in web application [/users] threw load() exception
        java.lang.NullPointerException
riamaria commented 10 months ago

UPDATE: Bundles loaded when I cleared cache and started from scratch. Original problem still persists.

image

Another log during startup. This might be closer to the root cause:

2024-01-25 21:05:19.753 ERROR [fileinstall-directory-watcher][DirectoryWatcher:1173] Unable to start bundle: file:/app/sw360/osgi/modules/org.eclipse.sw360.importers-18.0.0-SNAPSHOT.jar
com.liferay.portal.kernel.log.LogSanitizerException: org.osgi.framework.BundleException: Could not resolve module: sw360-importers [1503]_  Unresolved requirement: Import-Package: org.eclipse.sw360.datahandler.common; version="[1.0.0,2.0.0)"_  Unresolved requirement: Import-Package: org.eclipse.sw360.commonIO; version="[1.0.0,2.0.0)"_    -> Export-Package: org.eclipse.sw360.commonIO; bundle-symbolic-name="sw360-common-io"; bundle-version="1.0.0"; version="1.0.0"; uses:="com.google.common.base,org.apache.thrift,org.eclipse.sw360.datahandler.couchdb,org.eclipse.sw360.datahandler.thrift.attachments,org.eclipse.sw360.datahandler.thrift.users"_       sw360-common-io [1501]_         Unresolved requirement: Import-Package: org.apache.thrift; version="[0.16.0,1.0.0)"_ [Sanitized]
        at org.eclipse.osgi.container.Module.start(Module.java:444) ~[org.eclipse.osgi.jar:?]
        at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:428) ~[org.eclipse.osgi.jar:?]
        at com.liferay.portal.file.install.internal.DirectoryWatcher._startBundle(DirectoryWatcher.java:1156) [bundleFile:?]
        at com.liferay.portal.file.install.internal.DirectoryWatcher._startBundles(DirectoryWatcher.java:1189) [bundleFile:?]
        at com.liferay.portal.file.install.internal.DirectoryWatcher._process(DirectoryWatcher.java:1046) [bundleFile:?]
        at com.liferay.portal.file.install.internal.DirectoryWatcher.run(DirectoryWatcher.java:221) [bundleFile:?]
JSurf commented 9 months ago

Same issue here, not able to run the software following the instructions

All backend servlets seem to fail, for example:

07-Feb-2024 14:31:01.602 SEVERE [main] org.apache.catalina.core.StandardContext.loadOnStartup Servlet [VendorService] in web application [/vendors] threw load() exception java.lang.NullPointerException at java.base/java.io.FilterInputStream.read(FilterInputStream.java:133) at com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.ensureLoaded(ByteSourceJsonBootstrapper.java:539) at com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.detectEncoding(ByteSourceJsonBootstrapper.java:133) at com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.constructParser(ByteSourceJsonBootstrapper.java:256) at com.fasterxml.jackson.core.JsonFactory._createParser(JsonFactory.java:1685) at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:1084) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3714) at org.ektorp.impl.StdCouchDbConnector$5.success(StdCouchDbConnector.java:244) at org.ektorp.http.RestTemplate.handleResponse(RestTemplate.java:126) at org.ektorp.http.RestTemplate.get(RestTemplate.java:22) at org.ektorp.impl.StdCouchDbConnector.get(StdCouchDbConnector.java:240) at org.ektorp.impl.StdCouchDbConnector.get(StdCouchDbConnector.java:231) at com.github.ldriscoll.ektorplucene.util.IndexUploader.updateSearchFunctionIfNecessary(IndexUploader.java:55) at org.eclipse.sw360.datahandler.couchdb.lucene.LuceneAwareDatabaseConnector.addView(LuceneAwareDatabaseConnector.java:93) at org.eclipse.sw360.datahandler.db.VendorSearchHandler.(VendorSearchHandler.java:50)

JSurf commented 9 months ago

Is this related ? https://github.com/helun/Ektorp/issues/222

JSurf commented 9 months ago

We can confirm, downgrading all apache http client libraries to 4.3.x fixes the issue. See https://github.com/eclipse-sw360/sw360/compare/main...vitecde:sw360:fix-npe

So the root cause is https://github.com/helun/Ektorp/issues/222

heliocastro commented 9 months ago

@JSurf Can you verify with this PR: https://github.com/eclipse-sw360/sw360/pull/2305

heliocastro commented 9 months ago

Actually, hold on to that yet. Regular users can be added normally using the interface. There's new user rest endpoint now that users can be added normally as well.

Maybe because that is really old instructions, the csv is not well formed, and we need fix the code. Are you able to try using the rest api ?

The reason I'm beating on it is that httpcomponents 4.3.3 have vulnerabilities and the fact we are still in need of ektorp code make decision on updated complicated. A second option is add the property of couchdb.cache set to false on config.

JSurf commented 9 months ago

I'm confused how a rest api or change in csv formatting would help in a case where all of the backend services won't start because of the ektorp/couchdb issue ? It is not only the user service but all services which use couchdb/ektorp

The issue is also pretty easy to reproduce. Just fire up a completely new instance sw360 using the docker compose and the default configuration and watch all backend services fail to start with "loadOnStartup" errors in the localhost log of tomcat.

heliocastro commented 9 months ago

We already reproduced, we need verify what is happening because is pretty unusual. Minor question, you tested in some cloud environment ?

JSurf commented 9 months ago

We use a Ubuntu-VM with docker on a on-premise ESXi

AlbertHjy commented 9 months ago

I have tried "couchdb.cache = false", but it does not work.

heliocastro commented 9 months ago

I was investigating this issue last night as I was able to reproduce it, but suddenly the local build started to work.

So, I think there are a mismatch on the docker images build on GitHub, and I will do a different experiment with the image generation there to guarantee the proper combination

Thanks for your patience

On Thu, Feb 22, 2024, 06:47 He, Albert @.***> wrote:

I have tried "couchdb.cache = false", but it does not work.

— Reply to this email directly, view it on GitHub https://github.com/eclipse-sw360/sw360/issues/2289#issuecomment-1958747083, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADGDAFE6IJDYOOJNXSFPRDYU3LYZAVCNFSM6AAAAABCKJLJWCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJYG42DOMBYGM . You are receiving this because you commented.Message ID: @.***>

heliocastro commented 9 months ago

@AlbertHjy If possible, try a clean install from the current docker image ?

AlbertHjy commented 9 months ago

Hi @heliocastro , I tried ghcr.io/eclipse-sw360/sw360:sw360-18.0.1-M1, but it seems this image does not have the apps.

image