oxygenxml / oxygen-git-client-addon

A Git client that can be installed as an add-on in Oxygen XML Editor.
Other
51 stars 21 forks source link

Error retrieving files from Gitlab repo with large number of files #178

Closed lukastheblack closed 6 months ago

lukastheblack commented 6 months ago

I am trying to connect Oxygen with my gitlab repo. It is self host in Openshift(Kubernetes). I am able to setup the connection in plugins, I am able to authenticate. I am able to browse most of my git repositories except for the one I need Oxygen for. It is a second level folder (Root/xml_sf/) , and has 38336 files.

My Oxygen Web Author is also running in Openshift, with 3 CPU and 4GB of memory allocated to it.

In the error logs it returns "com.oxygenxml.examples.gitlab.GitlabApi - ro.sync.net.protocol.http.HttpExceptionWithDetails: 404 Not Found for: https://gitlab.REDACTED.com/api/v4/projects/72/repository/tree?id=72&page=90&pagination=legacy&path=xml_sf&per_page=20&recursive=false&ref=master ro.sync.net.protocol.http.HttpExceptionWithDetails: 404 Not Found for: https://gitlab.REDACTED.com/api/v4/projects/72/repository/tree?id=72&page=90&pagination=legacy&path=xml_sf&per_page=20&recursive=false&ref=master"

I am able to browse that URI via any web browser, and other git repositories work without issue, so I assume this must be related to the number of files in there directory. Is there any settings I can tweak, even if they're code related that could help with this ? Any help is appreciated, thank you.

] com.oxygenxml.examples.gitlab.GitlabApi - ro.sync.net.protocol.http.HttpExceptionWithDetails: 404 Not Found for: https://gitlab.REDACTED.com/api/v4/projects/72/repository/tree?id=72&page=90&pagination=legacy&path=xml_sf&per_page=20&recursive=false&ref=master ro.sync.net.protocol.http.HttpExceptionWithDetails: 404 Not Found for: https://gitlab.REDACTED.com/api/v4/projects/72/repository/tree?id=72&page=90&pagination=legacy&path=xml_sf&per_page=20&recursive=false&ref=master at ro.sync.net.protocol.http.b.r.x(Unknown Source) at ro.sync.net.protocol.http.b.r.c(Unknown Source) at ro.sync.net.protocol.http.abstraction.o.r(Unknown Source) at ro.sync.net.protocol.http.abstraction.o.e(Unknown Source) at ro.sync.net.protocol.http.WebdavHttpURLConnection.obtainResponse(Unknown Source) at ro.sync.net.protocol.http.WebdavHttpURLConnection.getInputStream(Unknown Source) at ro.sync.net.protocol.https.WebdavHttpsURLConnection.getInputStream(Unknown Source) at com.oxygenxml.examples.gitlab.GitlabApi.listFilesInternal(GitlabApi.java:735) at com.oxygenxml.examples.gitlab.GitlabApi.lambda$listFiles$6(GitlabApi.java:683) at com.oxygenxml.examples.gitlab.GitlabOAuthService.tryAccess(GitlabOAuthService.java:97) at com.oxygenxml.examples.gitlab.GitlabApi.listFiles(GitlabApi.java:682) at com.oxygenxml.examples.gitlab.GitlabUrlConnection.listFolder(GitlabUrlConnection.java:71) at ro.sync.ecss.extensions.api.webapp.plugin.FilterURLConnection.listFolder(Unknown Source) at ro.sync.servlet.RESTFileBrowser.retrieveFiles(RESTFileBrowser.java:471) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:170) at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:130) at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:660) at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:524) at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:474) at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364) at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:476) at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:434) at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:408) at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:69) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:492) at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:261) at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:161) at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364) at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:164) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:247) at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:249) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:60) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:200) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at ro.sync.servlet.config.UnlockDocumentsFilter.doFilter(UnlockDocumentsFilter.java:65) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:169) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at ro.sync.servlet.config.ExceptionHandlingFilter.doFilter(ExceptionHandlingFilter.java:33) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:169) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at ro.sync.ecss.webapp.d.i.doFilter(Unknown Source) at ro.sync.ecss.extensions.api.webapp.license.LicenseEnforcerFilter.doFilter(Unknown Source) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:169) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at ro.sync.servlet.config.OxygenLanguageFilter.doFilter(OxygenLanguageFilter.java:44) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:169) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at ro.sync.servlet.config.OxygenContextFilter.doFilter(OxygenContextFilter.java:61) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:169) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at ro.sync.servlet.config.UTF8CharsetEnforcer.doFilter(UTF8CharsetEnforcer.java:28) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:169) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:169) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at ro.sync.auth.HstsHeaderFilter.doFilter(HstsHeaderFilter.java:30) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:169) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at ro.sync.auth.CsrfFilter.doFilter(CsrfFilter.java:73) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:169) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at ro.sync.auth.CspFilter.doFilter(CspFilter.java:83) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:169) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at ro.sync.auth.CrossDomainFilter.doFilter(CrossDomainFilter.java:29) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:169) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at com.oxygenxml.examples.git.SynchronizedReposTrackerFilter.doFilter(SynchronizedReposTrackerFilter.java:27) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:169) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at ro.sync.servlet.ssl.SSLRedirectFilter.doFilter(SSLRedirectFilter.java:69) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:169) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at ro.sync.servlet.config.ApplicationAuthenticationFilter.doFilter(ApplicationAuthenticationFilter.java:112) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:169) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at ro.sync.servlet.config.SessionFilter.doFilter(SessionFilter.java:59) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:169) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:458) at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:373) at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387) at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:370) at ro.sync.auth.WebappShiroFilter.doFilterInternal(WebappShiroFilter.java:72) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:169) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:670) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1786) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) at java.base/java.lang.Thread.run(Unknown Source)

lukastheblack commented 6 months ago

Further information from testing. I used multiple sources, gitlab.com hosted repo, Sharepoint, and Owncloud(webDav), and while the latter 2 would show files when connected, it was limited. I could not even open files by name, it would return error saying "file does not exist".

It seems that the application is unable to enumerate above a certain number of files in a given folder. Based on the files I was able to see in Sharepoint and Owncloud, that number is 593. I would love any ideas/tips/pointers, thank you so much!

raducoravu commented 6 months ago

This particular project is for the Git client add-on for the Oxygen desktop application, not for WebAuthor. I will try to get a colleague from the WebAuthor team to reply here, ideally you should have written us an email instead or used the forum.

lukastheblack commented 6 months ago

@raducoravu my mistake, I would be happy to close this and open an issue in the appropriate channel. Thank you for your reply.

cosmin-eftenie commented 6 months ago

Hello,

Thank you for reporting this issue. We have registered an internal issue (WA-7311) regarding this behavior. As soon as we have a solution, we will notify you here. Alternatively, you can provide us via email with an email address where we can keep in touch with you.

Best, Cosmin

lukastheblack commented 6 months ago

@cosmin-eftenie Thank you! Keeping in touch here should be sufficient.

I have been digging a little deeper on the issue as I noticed something regarding Gitlab behavior that may be relevant. When I change the "per_page" parameter to 40,000 (from 20 in the link from the error log) , it still only returns 100 Items. I have linked a Gitlab Issue below where this has been in discussion for 7 years, so it seems there is also a limitation on the Gitlab side, and it doesn't seem likely to be resolved in the near future.

I think my use case is pretty specific, and the fact that all these files live in the same gitlab directory is rather stupid besides, but that is the way the end users who edit the files want it. I don't want to make this into a huge priority issue for your team, Thank you for your time either way.

https://gitlab.com/gitlab-org/gitlab/-/issues/17329

cosmin-eftenie commented 6 months ago

Hello,

We thank you for your involvement. We have noted this additional information.

Best, Cosmin