can't select project after indexing

[chkp-baselz]

Hey, after indexing using java -Djava.util.logging.config.file=/opengrok/etc/logging.properties -jar /opengrok/dist/lib/opengrok.jar -c /usr/local/bin/ctags -s /opengrok/src -d /opengrok/data -H -P -S -G -W /opengrok/etc/configuration.xml -U http://localhost:8080/source I can't select projects to search on

also, I have attached my configration.xml configuration.zip

Thank you for your help

[vladak]

This seems like some glitch in the UI. Looking at the configuration it has 3 important items:

• the projectsEnabled property is true
• there are projects defined
• each of the projects has indexed property set to true

so nothing should prevent the projects from being displayed. You can confirm this is the case also in the web app by doing these queries:

• http://localhost:8080/source/api/v1/projects/indexed (should return list of indexed projects)
• http://localhost:8080/source/api/v1/configuration/projectsEnabled

Next, look into the source of the index page. It should contain something like:

<select tabindex="8" class="q" id="project"
        name="project" multiple="multiple" size="15"><option value="almson-refcount">almson-refcount</option><option value="bar">bar</option> ... </select>

If it does, then this is probably some issue with JavaScript loading - check e.g. the Web developer menu in Firefox for any sign of trouble.

[vladak]

Also, the usual question: has anything changed recently ?

[chkp-baselz]

Hey @vladak and thnx for your answer

the queries return 404

About the source of the index page, I have looked under /$tomcat/webapps/source and found only index.jsp and index_body.html, and both do not have what you mentioned above, to which one should I add it?

I have changed owner and permissions of /opengrok folder, permissions:755, owner user:root

Thnx again

[vladak]

Where did you deploy the web app ? What exactly is in /$tomcat/webapps/source/index.jsp ?

[chkp-baselz]

@vladak all as wiki https://github.com/oracle/opengrok/wiki/How-to-setup-OpenGrok deployed under /usr/share/tomcat index is attached index.zip

[vladak]

Okay, that's a normal index.jsp (in case of a Docker container it could contain a redirect to the real web app location).

Which OpenGrok version are you running ?

[vladak]

The API queries should really return something unless you are running really old OpenGrok version or the web app is not deployed properly. Could you check Tomcat logs ? Also, what happens if you load the index page (http://localhost:8080/source/) from a command line utility such as curl ?

[chkp-baselz]

@vladak version 1.5.8 suddenly, all API works great, and it returns what it should, like the list or the webpage, etc.. But still, can't see projects

Attached is /usr/share/tomcat/logs/catalina.2021-02-11.log , I didn't find anything weird log.zip

many many thnx

[vladak]

The log looks fine. Time to check the index page contents and JavaScript loading.

[chkp-baselz]

@vladak I'm sorry Vlad I'm not sure I have understood what you mean

[vladak]

It's quite simple: display the HTML code of the index page in your browser (Ctrl + U in Firefox) and look for the projects related select HTML element like described in https://github.com/oracle/opengrok/issues/3420#issuecomment-777568071

[vladak]

Also, does this happen from another browser/computer ? What if you try Private window (Firefox) ?

[vladak]

To shed a bit more light on how the project picker works:

1. menu.jspf that is included from index.jsp generates the select HTML element and its contents to the index HTML page: https://github.com/oracle/opengrok/blob/876391fae90ef7239a29b98c01205b3ab9a89fcc/opengrok-web/src/main/webapp/menu.jspf#L72
2. JavaScript code that is inserted into the HTML in https://github.com/oracle/opengrok/blob/876391fae90ef7239a29b98c01205b3ab9a89fcc/opengrok-web/src/main/webapp/httpheader.jspf#L74-L85 rewrites the DOM via https://github.com/oracle/opengrok/blob/876391fae90ef7239a29b98c01205b3ab9a89fcc/opengrok-web/src/main/webapp/js/utils-0.0.39.js#L1584-L1587 to override the default project picker with JavaScript based code in Searchable Options List based on the contents of the select element: https://github.com/oracle/opengrok/blob/876391fae90ef7239a29b98c01205b3ab9a89fcc/opengrok-web/src/main/webapp/js/searchable-option-list-2.0.14.js#L664-L669

This means either there is something wrong with the select form generation or JavaScript execution. Given that the projects are listed in the initial screenshot below the search (in the blue mishmash) form I'd guess the latter.

[chkp-baselz]

@vladak thnx for your answers and explain

first of all as you said, blue mishmash is the projects list this issue happens in all browsers mainly I use Chrome(including private)

The attachment is index.html, all seems to be okay index.zip

There was no JavaScript error and all files as you described above

[vladak]

Is this really the whole index.html ? Seems like there is missing stuff at the end.

[chkp-baselz]

@vladak yes by google Chrome this is the whole index.html, may this have become after wrong indexing or something?

[vladak]

Could you try getting the index page with a CLI utility to make sure this is not a browser issue ? E.g. curl -o index.html http://localhost:8080/source/

[vladak]

To me, this looks like as if the foot.jspf was not processed. The index.jsp ends with:

include file="repos.jspf"

            %><% } %>
        </div>

<%
/* ---------------------- index.jsp end --------------------- */
%><%@

include file="foot.jspf"

%>

and repos.jspf ends with:

                        </table>
                    </div>
                </div>
            </div><%
        }
    }
}
%>
<br/>
<br/>

This matches the end of your index.html:

</table>                                                                        
                    </div>                                                      
                </div>                                                          
            </div><br/>                                                         
<br/>                                                                           
</div>

Normally, the foot.jspf generates the logo at the bottom of the page. The JavaScript includes are at the very bottom of the page:

<%
    /**
     * Print out all collected javascript files (or inline script) by the calls
     * to {@link PageConfig#addScript} methods.
     */
%>
    <%= PageConfig.get(request).getScripts() %>
</body>
</html>
<%
}
/* ---------------------- foot.jspf end --------------------- */
%>

Could you check that the foot.jspf file is present and readable under the webapps/source directory inside the Tomcat directory structure ?

[chkp-baselz]

@vladak curl return same html

I think you're correct, I don't have foot.jsp under source folder

[vladak]

The file should be foot.jspf, assuming you are running recent OpenGrok version. Anyhow, if foot.jsp* is missing under the webapps/source directory something went wrong with the web application deploy.

[vladak]

Also, it is surprising to me that Tomcat did not report any problem w.r.t. the missing file.

[chkp-baselz]

@vladak where could I get the missing files from? and where should I include them?

[vladak]

Normally the file is contained in the WAR file distributed with OpenGrok, i.e. source.war:

$ jar tvf source.war foot.jspf
  2346 Thu Feb 11 16:07:50 CET 2021 foot.jspf

Rather than extracting just a single file, I'd recommend deploying the web application as whole.

There is no need to include the file anywhere, the web application WAR file is self contained.

I wonder if there is something fishy with the deploy procedure on your side also given the 404 errors above.

[chkp-baselz]

@vladak my index.jsp does not include foot at the end( attached above) so if I extract the foor.jspf It'll will include automatically in index.jsp for example?

Deplying the web again will take time and I prefer not

[vladak]

Note that index.jsp is not index.html. The JSP file is basically a template for producing the HTML file. The way it works is that the container server (Tomcat) compiles the JSP file into Java byte code and executes that. The execution will generate index.html dynamically and the server sends it to the client (e.g. a browser). You can certainly try extracting just the foot.jspf from the WAR archive however as we say in other places of the company I work for: "this is unsupported" :-)

[vladak]

A more generic question in this case is how to prevent problems like this from happening, i.e. if a include cannot be satisfied in a JSP, throw an exception.

[vladak]

Actually, if I remove foot.jspf from the extracted webapps/source directory, then start Tomcat and then access the index page I get:

HTTP Status 500 – Internal Server Error

Type Exception Report

Message /index.jsp (line: [70], column: [3]) File [foot.jspf] not found

Description The server encountered an unexpected condition that prevented it from fulfilling the request.

Exception

org.apache.jasper.JasperException: /index.jsp (line: [70], column: [3]) File [foot.jspf] not found
    org.apache.jasper.compiler.DefaultErrorHandler.jspError(DefaultErrorHandler.java:41)
    org.apache.jasper.compiler.ErrorDispatcher.dispatch(ErrorDispatcher.java:291)
    org.apache.jasper.compiler.ErrorDispatcher.jspError(ErrorDispatcher.java:97)
    org.apache.jasper.compiler.Parser.processIncludeDirective(Parser.java:345)
    org.apache.jasper.compiler.Parser.parseIncludeDirective(Parser.java:380)
    org.apache.jasper.compiler.Parser.parseDirective(Parser.java:481)
    org.apache.jasper.compiler.Parser.parseFileDirectives(Parser.java:1797)
    org.apache.jasper.compiler.Parser.parse(Parser.java:141)
    org.apache.jasper.compiler.ParserController.doParse(ParserController.java:244)
    org.apache.jasper.compiler.ParserController.parseDirectives(ParserController.java:127)
    org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:197)
    org.apache.jasper.compiler.Compiler.compile(Compiler.java:373)
    org.apache.jasper.compiler.Compiler.compile(Compiler.java:350)
    org.apache.jasper.compiler.Compiler.compile(Compiler.java:334)
    org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:595)
    org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:399)
    org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:386)
    org.apache.jasper.servlet.JspServlet.service(JspServlet.java:330)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
    org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    org.opengrok.web.CookieFilter.doFilter(CookieFilter.java:49)
    org.opengrok.web.AuthorizationFilter.doFilter(AuthorizationFilter.java:95)
    org.opengrok.web.StatisticsFilter.doFilter(StatisticsFilter.java:71)

so this works as expected.

[vladak]

Even if I remove foot.jspf while the Tomcat is running the server will produce the above exception so I am not sure what gives.

[chkp-baselz]

@vladak maybe because I don't have

<%@

include file="foot.jspf"

%>

in index.jsp, menu.jsp, etc...? I'm not sure if this came as this or someone changed it

tho, do you think that this is related to Projects issue?

[vladak]

Um, yes. What is the OpenGrok version you are running ? The foot.jspf include was added to index.jsp long time ago (2011) so unless you are running really old OpenGrok version the contents should be there. One should never edit the JSP/JSPF files manually.

[chkp-baselz]

@vladak version 1.5.8

[vladak]

That's recent enough. I don't have any other explanation that someone edited the JSP file(s) and removed this section.

[chkp-baselz]

@vladak I have created the foot.jspf again, and include it in this whole files error.jsp help.jsp more.jsp search.jsp status.jsp diff.jsp enoent.jsp history.jsp index.jsp list.jsp

after restarting the server I get a blank screen( white and nothing there) I've run index, will update you once it's done Thank you

[vladak]

I think it would be safer to copy the files from source.war extraction.

[vladak]

Also, reindex should not be necessary. According to my experience once the files are copied to the web app the effect is immediate.

[chkp-baselz]

@vladak you're right, exactly indexing finished and nothing changed, do you suggest copying files again instead of editing them?

[vladak]

Yep. Try with index.jsp first.

[chkp-baselz]

@vladak it didn't help, still blank screen, I've attached tomcat log, it says something but didn't truly understand log (2).zip

[vladak]

The logs are clouded by #3384 (fixed in 1.5.10), I don't see anything of interest there. Check the HTTP result codes in the 'Web developer -> Network' in Firefox.

[chkp-baselz]

@vladak 304

[chkp-baselz]

@vladak Do you suggest copying the whole files I've mentioned above instead of only index.jsp?

[vladak]

HTTP code 304 is to use cached version, try reloading the page with Ctrl+Shift+R.

[vladak]

My advice would be to deploy the web app anew.

[chkp-baselz]

@vladak on a different topic, I haven't found on wiki how to easily update OpenGrok version, is there's an easy way?

[vladak]

Good idea, there should be a "how to update" wiki. In general it is easy - just extract the OpenGrok and deploy the web app. However, in case the index format changes, it is necessary to reindex from scratch. Unless the release notes say it is necessary to reindex from scratch, it is the former procedure.

[chkp-baselz]

@vladak I see, okay I could deploy again but what about files I have changed, such as web.xml? should I change it again?

[chkp-baselz]

@vladak also should I change anything under /Opengrok/* ? no right?

[chkp-baselz]

just step 2 https://github.com/oracle/opengrok/wiki/How-to-setup-OpenGrok

[vladak]

@vladak I see, okay I could deploy again but what about files I have changed, such as web.xml? should I change it again?

Yes, web.xml is the only file from the web app that can be changed. Ideally, one should use the opengrok-deploy tool to do that safely.

[vladak]

I've written the Updating wiki. It likely has some gaps however it is a start.