oracle / opengrok

OpenGrok is a fast and usable source code search and cross reference engine, written in Java
http://oracle.github.io/opengrok/
Other
4.36k stars 748 forks source link

OpenGrok home page doesn't show projects in group during configuration upload #3122

Open danielviorreta opened 4 years ago

danielviorreta commented 4 years ago

Describe the bug The home page is not updated with the last index timestamp using the -U param in the indexer or after calling the configuration api method using the webapp context (http://localhost:8080/source).

To Reproduce Calling the api method:

curl -X PUT -H "Content-Type: application/xml"  --data "@/var/opengrok/etc/configuration.xml" http://localhost:8080/source/api/v1/configuration?reindex=true

produces the next output in tomcat:

20-Apr-2020 09:58:13.240 INFO [http-nio-127.0.0.1-8080-exec-5] org.opengrok.indexer.util.Statistics.report done invalidating 4119 repositories (took 52.275 seconds)
20-Apr-2020 10:00:30.336 INFO [http-nio-127.0.0.1-8080-exec-5] org.opengrok.indexer.configuration.RuntimeEnvironment.applyConfig Configuration updated
20-Apr-2020 10:00:30.336 INFO [http-nio-127.0.0.1-8080-exec-5] org.opengrok.indexer.configuration.WatchDogService.stop Watchdog stoped
20-Apr-2020 10:00:30.336 WARNING [http-nio-127.0.0.1-8080-exec-5] org.opengrok.indexer.framework.PluginFramework.reload Plugin directory not found or not readable: /var/opengrok/data/../plugins. All requests allowed.
20-Apr-2020 10:00:30.336 INFO [http-nio-127.0.0.1-8080-exec-5] org.opengrok.web.api.v1.suggester.provider.service.impl.SuggesterServiceImpl.initSuggester Suggester disabled

But the home page is not refreshed with the date of the timestamp file. The date of the timestamp file in the DATA directory is properly updated. Same effect adding the -U param in the indexer.

The home page is properly updated after calling the same api method but without the context (source). But the page doesn't show any project during the api call: curl -X PUT -H "Content-Type: application/xml" --data "@/var/opengrok/etc/configuration.xml" http://localhost:8080//api/v1/configuration?reindex=true

vladak commented 4 years ago

Do you use per project reindex by a chance ?

vladak commented 4 years ago

I am asking because my workflow involves running indexer for each project in parallel which avoids to set the configuration at the end of the reindex (see https://github.com/oracle/opengrok/wiki/Per-project-management-and-workflow). Instead the timestamp file is touched once the project indexing is over and that works (#1670).

danielviorreta commented 4 years ago

I index all the data at once. Is there any way to force the timestamp to be reloaded as you suggested in https://github.com/oracle/opengrok/issues/1670#issuecomment-393512093? It looks like the /opengrok/dist/bin/sync.py doesn't accept the -p param anymore.

vladak commented 4 years ago

The timestamp file's modified time stamp is only updated during reindex in IndexDatabase#update() via env.setIndexTimestamp(). The configuration controller calls just env.applyConfig() that refreshes the timestamp in memory via refreshDateForLastIndexRun() by stat'ing the file. Updating the configuration via API cannot change the timestamp of the file.

As for sending the API call to the app server without context I don't see how this could work without also deploying the app to ROOT.

vladak commented 4 years ago

Returning the the original problem (the timestamp file is touched however the changes do not manifest in the UI): when running the indexer with -U, the configuration is sent to the webapp at the end of the indexing using the configuration API end point. This in turn refreshes the time stamp. Then, foot.jspf calls cfg.getEnv().getDateForLastIndexRun() to retrieve the refreshed timestamp when any page is reloaded.

I tried the basic indexer incantation and cannot reproduce the reported problem.

For completeness, the timestamp file is touched only if there were any changes to the index (there is dirty flag in IndexDatabase).

danielviorreta commented 4 years ago

I have my app deployed to ROOT so adding the source context didn't fix my problem as it was expected. Sorry for the confusion, I misinterpreted the logs.

So, my original problem is that adding -U http://localhost:8080 to the indexer grok UI doesn't show any project during projects invalidations. Indexer logs:

2020-04-21 05:26:16.122-0400 CONFIG t1 ExtendedLogger.config: Selected ExecutorServiceProvider implementation [org.glassfish.jersey.client.DefaultClientAsyncExecutorProvider] to be used for injection of executor qualified by [org.glassfish.jersey.client.ClientAsyncExecutor] annotation.
2020-04-21 05:26:16.122-0400 CONFIG t1 ExtendedLogger.config: Selected ScheduledExecutorServiceProvider implementation [org.glassfish.jersey.client.DefaultClientBackgroundSchedulerProvider] to be used for injection of scheduler qualified by [org.glassfish.jersey.client.ClientBackgroundScheduler] annotation.
2020-04-21 05:26:16.123-0400 CONFIG t1 HttpUrlConnector.<init>: Restricted headers are not enabled using [sun.net.http.allowRestrictedHeaders] system property (setting only takes effect on connections created after the property has been set/changed).
2020-04-21 05:26:16.125-0400 FINEST t1 HttpURLConnection.plainConnect0: ProxySelector Request for http://localhost:8080/api/v1/configuration?reindex=true
2020-04-21 05:26:16.125-0400 FINEST t1 HttpClient.logFinest: KeepAlive stream retrieved from the cache, sun.net.www.http.HttpClient(http://localhost:8080/api/v1/projects/jdk7u60/indexed)
2020-04-21 05:26:16.125-0400 FINEST t1 HttpURLConnection.plainConnect0: Proxy used: DIRECT
2020-04-21 05:26:16.127-0400 FINE t1 HttpURLConnection.writeRequests: sun.net.www.MessageHeader@54bc3bcf7 pairs: {PUT /api/v1/configuration?reindex=true HTTP/1.1: null}{Content-Type: application/xml}{User-Agent: Jersey/2.29 (HttpUrlConnection 11.0.3)}{Host: localhost:8080}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}{Content-Length: 2103065}
2020-04-21 05:27:25.994-0400 FINE t1 HttpURLConnection.getInputStream0: sun.net.www.MessageHeader@d3d50946 pairs: {null: HTTP/1.1 204}{Cache-Control: private}{Expires: Thu, 01 Jan 1970 00:00:00 GMT}{Date: Tue, 21 Apr 2020 09:27:25 GMT}{Keep-Alive: timeout=20}{Connection: keep-alive}
2020-04-21 05:27:25.994-0400 INFO t1 Indexer.sendToConfigHost: Configuration update routine done, check log output for errors.`

In tomcat logs I have this:

`21-Apr-2020 05:27:14.581 INFO [http-nio-127.0.0.1-8080-exec-603] org.opengrok.indexer.util.Statistics.report done invalidating 1487 repositories (took 58.290 seconds)`

During that minute, grok UI doesn't show any project. Is this expected or do you think I have any configuration problem? I tried to deploy my webapp using the source context path but I get the same problem.

Thanks!

vladak commented 4 years ago

The old repositories should show during invalidation. How does the index page look like without repositories ?

What is the version you are running ?

danielviorreta commented 4 years ago

I am running version 1.3.12. I reproduce the problem every time that I call: /bin/curl -X PUT -H "Content-Type: application/xml" --data "@/var/opengrok/etc/configuration.xml" http://localhost:8080/source/api/v1/configuration?reindex=true

Screenshot 2020-04-21 at 16 19 19
danielviorreta commented 4 years ago

@vladak did you manage to reproduce the problem? Do you have any suggestion about that? Thanks!

danielviorreta commented 4 years ago

@vladak I think the problem is in https://github.com/oracle/opengrok/blob/3b65bef1246c8f826edd489e43f865c7bafc512d/opengrok-indexer/src/main/java/org/opengrok/indexer/configuration/RuntimeEnvironment.java#L1587

All my projects are included in a group. Uploading a new configuration clears all the repositories from all the groups. Projects that are not part of group are still listed during the upload.

Does it make sense?

tarzanek commented 4 years ago

@tulinkry ^ any obvious prob that you can see when groups are merged?

tulinkry commented 4 years ago

It does not merge them, it replaces the old ones but it shouldn't take a minute even for thousand of projects.

danielviorreta commented 4 years ago

it takes ~ 5 minutes for 5000 projects and 500 groups. The repositories in groups disappeared from the UI when I see this log in tomcat: https://github.com/oracle/opengrok/blob/2173ed7b20b4d6ae704b761403b2af7f52426e38/opengrok-indexer/src/main/java/org/opengrok/indexer/history/HistoryGuru.java#L862 (maybe because we are clearing the repos in https://github.com/oracle/opengrok/blob/2173ed7b20b4d6ae704b761403b2af7f52426e38/opengrok-indexer/src/main/java/org/opengrok/indexer/history/HistoryGuru.java#L854)

And they only come back after https://github.com/oracle/opengrok/blob/3b65bef1246c8f826edd489e43f865c7bafc512d/opengrok-indexer/src/main/java/org/opengrok/indexer/configuration/RuntimeEnvironment.java#L1760

danielviorreta commented 4 years ago

Repositories are removed from UI here: https://github.com/oracle/opengrok/blob/3b65bef1246c8f826edd489e43f865c7bafc512d/opengrok-indexer/src/main/java/org/opengrok/indexer/configuration/RuntimeEnvironment.java#L1642

And they come back after calling the populateGroups method: https://github.com/oracle/opengrok/blob/3b65bef1246c8f826edd489e43f865c7bafc512d/opengrok-indexer/src/main/java/org/opengrok/indexer/configuration/RuntimeEnvironment.java#L1671

I just added an extra call to populateGroups in https://github.com/oracle/opengrok/blob/3b65bef1246c8f826edd489e43f865c7bafc512d/opengrok-indexer/src/main/java/org/opengrok/indexer/configuration/RuntimeEnvironment.java#L1646 and it fixes my issue. I am not saying that this is the correct solution, only trying to help you to debug the problem

danielviorreta commented 4 years ago

Hey @tulinkry , did you manage to reproduce this problem?

Thanks, Daniel