Composum does not work with the FileVault FSPackageRegistry

[rombert]

FileVault has support for non-JCR package managers, which support storing the content packages in other locations.

In my case, I've configured Sling to work with a FSPackageRegistry so that all content packages are stored in the filesystem and the OSGi installer is not active anymore.

However, these content packages, although installed, are not visible in the Packages tab. I am not sure what the root cause is, but please ask for more information if needed.

[stoerr]

Right, this would require extending / rewriting the Composum package manager - right now it just works with the PackageManager, not with the newer PackageRegistry. I guess stuff from the PackageRegistry doesn't show up in the PackageManager.

[ist-rw]

I have not found a way to determine the set of available PackageRegistries. That would be the entry point to generally use PackageRegistries instead of PackageManager. Trying to implement a custom PackageRegistry collector service fails because the JcrPackageRegistry does not implement a service. (org.apache.jackrabbit.vault-3.2.4)

[rombert]

@DominikSuess @kwin - maybe you have an idea about how this can be achieved?

[kwin]

Indeed that is not easily possible yet. I created https://issues.apache.org/jira/browse/JCRVLT-502 and https://issues.apache.org/jira/browse/JCRVLT-503 to ease working with it.

You could try to act on JcrPackageManager.getRegistry() and then implement such a composite service to merge this with the FS based registry. Alternatively you have to act on both PackageRegistry (for the FS based registry) and the PackageManager (for the JCR based registry).

[kwin]

I released FileVault 3.4.10 which exposes a Composite Package Registry. Hopefully that eases working with both registries...

[stoerr]

@rombert : how would you configure a Sling Starter 12 so that you get packages installed into a FSPackageRegistry? I enabled the FSPackageRegistry, but have trouble finding an easy way to put packages into it until the Composum Package Manager support is fully done. Neither filesystem install, nor installing the packages as features put's it there.

[rombert]

@stoerr - I need to remember exactly what I did :-) I think this was centered around using https://github.com/apache/sling-org-apache-sling-feature-extension-content as a feature launcher extension. I'll try to come up with a concrete example next week.

[stoerr]

@rombert Thank you! Ah, OK, just what I did in https://issues.apache.org/jira/browse/JCRVLT-525 . Is that still used in Sling Starter 12? I didn't find the ExecutionPlanRepoInitializer there, but didn't try hard. Well, never mind - I'll just put in something programmatically for testing purposes. If you don't think that'd be educational for me and others, there is no need to spend time on that.

[stoerr]

@kwin / @rombert : I'm wondering: the JcrPackageManager has methods create, update, assemble that are used in the workflow of the Composum package manager. I do not see anything comparable in PackageRegistry. There are some things in JcrPackageRegistry, but even there I don't see something matching JcrPackageManager.assemble. How would you envision a new interface? I'm wondering whether we should actually move from JcrPackageManager to other, newer, interfaces (then I'm probably missing something), or just keep the JcrPackageManager stuff as it is for creating / updating / building JcrPackages , and add some new stuff for handling PackageRegistries , where we just support inspecting the packages, and upload/download/install/remove/extract, but no build=assemble and no modification.

[rombert]

@stoerr - I planned to create a sample using the Content Extension but ran out of time, sorry. About the JcrPackageManager vs PackageRegistry API ... I think @kwin knows best.

[stoerr]

@rombert / @kwin : we've just merged a first version for this into develop (and the Composum Nodes 4.x branch sling12). It's testable with two registries by enabling the FS Package Registry Service in the felix console. The basic functionality is provided (upload / download / install / uninstall etc., inspect the package), though creating and editing packages can only be done with the old interface, as the package registry doesn't seem to provide that functionality. (The new interface is accessible by switching the tree by it's tabs.)

I'm currently working on extending that a little. So, feedback is welcome, though it's not in a release yet.

[rombert]

@stoerr - I just had the chance to play with the 4.1.3 version a bit and it looks good, thanks! I can see my 'FS Registry' package and download it. I guess coverage and filters would be nice to have in the future.

I don't have a strong need to edit/install packages backed by the fs registry, as those are typically installed using the feature model directly. Are you saying that the 'registry' functionality is limited because of the APIs and that the 'manager' view is the one that has all the bells and whistles? If so, @kwin can probably tell you if there is a way around it, I am not that familiar with FileVault.

[kwin]

Known limitation: https://issues.apache.org/jira/browse/JCRVLT-504

[stoerr]

@rombert Thank you! I ported the coverage tab to the registry part, though it'll take a bit until that's in a release. What were you missing about the filters? They are displayed at least, though you cannot modify them.

Right: with 'limited' I meant the registry stuff currently seems to provide just the functionality to handle packages as a whole (add/upload/download/install/uninstall packages) and to inspect them, but not to modify individual packages: create, modify and build. That seems what the JCRVLT-504 @kwin mentioned will implement. But until that's present for a while in the API I guess you'll have to go the old JCR package manager part of the Composum package manager to tweak the contents of a package. No such luck for packages in the FS Registry, though.

[rombert]

@stoerr - I don't actually see the filters, see below screenshot

I downloaded the file using the link exposed by Composum and I verified that the META-INF/vault/filter.xml file contains a single filter, e.g.

<workspaceFilter version="1.0">
<filter root="/content/FOO"/>
</workspaceFilter>

Regarding packages in the FS Registry, my current thinking is that they are immutable and provisioned when the instance starts. For development work one can always ues the JCR registry together with the package installer.

[stoerr]

@rombert Strange. Do you see any exception in the error log or a Javascript problem in the browser inspector? I tried several packages with the release, and it always shows the filters nicely. I cannot reproduce that problem.

[rombert]

@stoerr - sorry, for the late turnaround. Yes, I did find something in the logs after switching to the 'filter' tab

java.lang.NullPointerException: Cannot invoke "org.apache.jackrabbit.vault.fs.api.WorkspaceFilter.getFilterSets()" because the return value of "org.apache.jackrabbit.vault.packaging.registry.RegisteredPackage.getWorkspaceFilter()" is null
    at com.composum.sling.core.pckgmgr.jcrpckg.PackageServlet$ListFiltersOperation.doIt(PackageServlet.java:1294) [com.composum.nodes.pckgmgr:4.1.3]
    at com.composum.sling.core.servlet.ServletOperationSet.doGet(ServletOperationSet.java:134) [com.composum.nodes.commons:4.1.3]
    at com.composum.sling.core.servlet.AbstractServiceServlet.doGet(AbstractServiceServlet.java:134) [com.composum.nodes.commons:4.1.3]
    at org.apache.sling.api.servlets.SlingSafeMethodsServlet.mayService(SlingSafeMethodsServlet.java:266) [org.apache.sling.api:2.27.0]
    at org.apache.sling.api.servlets.SlingAllMethodsServlet.mayService(SlingAllMethodsServlet.java:137) [org.apache.sling.api:2.27.0]
    at org.apache.sling.api.servlets.SlingSafeMethodsServlet.service(SlingSafeMethodsServlet.java:342) [org.apache.sling.api:2.27.0]
    at org.apache.sling.api.servlets.SlingSafeMethodsServlet.service(SlingSafeMethodsServlet.java:374) [org.apache.sling.api:2.27.0]
    at org.apache.sling.engine.impl.request.RequestData.service(RequestData.java:518) [org.apache.sling.engine:2.12.0]
    at org.apache.sling.engine.impl.filter.SlingComponentFilterChain.render(SlingComponentFilterChain.java:45) [org.apache.sling.engine:2.12.0]
    at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:88) [org.apache.sling.engine:2.12.0]
    at org.apache.sling.engine.impl.SlingRequestProcessorImpl.processComponent(SlingRequestProcessorImpl.java:325) [org.apache.sling.engine:2.12.0]
    at org.apache.sling.engine.impl.filter.RequestSlingFilterChain.render(RequestSlingFilterChain.java:49) [org.apache.sling.engine:2.12.0]
    at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:88) [org.apache.sling.engine:2.12.0]
    at org.apache.sling.engine.impl.debug.RequestProgressTrackerLogFilter.doFilter(RequestProgressTrackerLogFilter.java:109) [org.apache.sling.engine:2.12.0]
    at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:78) [org.apache.sling.engine:2.12.0]
    at org.apache.sling.i18n.impl.I18NFilter.doFilter(I18NFilter.java:131) [org.apache.sling.i18n:2.5.18]
    at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:78) [org.apache.sling.engine:2.12.0]
    at com.composum.sling.nodes.mount.remote.RemoteRequestFilter.doFilter(RemoteRequestFilter.java:76) [com.composum.nodes.console:4.1.3]
    at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:78) [org.apache.sling.engine:2.12.0]
    at org.apache.sling.engine.impl.SlingRequestProcessorImpl.doProcessRequest(SlingRequestProcessorImpl.java:211) [org.apache.sling.engine:2.12.0]
    at org.apache.sling.engine.impl.SlingMainServlet.service(SlingMainServlet.java:128) [org.apache.sling.engine:2.12.0]
    at org.apache.felix.http.base.internal.handler.ServletHandler.handle(ServletHandler.java:126) [org.apache.felix.http.jetty:4.1.14]
    at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:86) [org.apache.felix.http.jetty:4.1.14]
    at org.apache.sling.i18n.impl.I18NFilter.doFilter(I18NFilter.java:131) [org.apache.sling.i18n:2.5.18]
    at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:142) [org.apache.felix.http.jetty:4.1.14]
    at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:81) [org.apache.felix.http.jetty:4.1.14]
    at org.apache.sling.engine.impl.log.RequestLoggerFilter.doFilter(RequestLoggerFilter.java:73) [org.apache.sling.engine:2.12.0]
    at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:142) [org.apache.felix.http.jetty:4.1.14]
    at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:81) [org.apache.felix.http.jetty:4.1.14]
    at org.apache.sling.engine.impl.parameters.RequestParameterSupportConfigurer.doFilter(RequestParameterSupportConfigurer.java:67) [org.apache.sling.engine:2.12.0]
    at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:142) [org.apache.felix.http.jetty:4.1.14]
    at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:81) [org.apache.felix.http.jetty:4.1.14]
    at org.apache.felix.http.base.internal.dispatch.Dispatcher$1.doFilter(Dispatcher.java:152) [org.apache.felix.http.jetty:4.1.14]
    at org.apache.felix.http.base.internal.whiteboard.WhiteboardManager$2.doFilter(WhiteboardManager.java:1002) [org.apache.felix.http.jetty:4.1.14]
    at org.apache.felix.http.sslfilter.internal.SslFilter.doFilter(SslFilter.java:97) [org.apache.felix.http.sslfilter:1.2.6]
    at org.apache.felix.http.base.internal.handler.PreprocessorHandler.handle(PreprocessorHandler.java:137) [org.apache.felix.http.jetty:4.1.14]
    at org.apache.felix.http.base.internal.whiteboard.WhiteboardManager$2.doFilter(WhiteboardManager.java:1008) [org.apache.felix.http.jetty:4.1.14]
    at org.apache.felix.http.base.internal.whiteboard.WhiteboardManager.invokePreprocessors(WhiteboardManager.java:1012) [org.apache.felix.http.jetty:4.1.14]
    at org.apache.felix.http.base.internal.dispatch.Dispatcher.dispatch(Dispatcher.java:94) [org.apache.felix.http.jetty:4.1.14]
    at org.apache.felix.http.base.internal.dispatch.DispatcherServlet.service(DispatcherServlet.java:49) [org.apache.felix.http.jetty:4.1.14]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) [org.apache.felix.http.servlet-api:1.1.4]
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.server.Server.handle(Server.java:516) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [org.apache.felix.http.jetty:4.1.14]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [org.apache.felix.http.jetty:4.1.14]
    at java.base/java.lang.Thread.run(Thread.java:833)

I can privately supply the package if needed.

[stoerr]

@rombert Weird. At least for vault 3.2.4 RegisteredPackage.getWorkspaceFilter() is declared as @ Nonnull , so that NPE shouldn't be possible. I'll throw in a null check but that won't give you your filters, either, if getWorkspaceFilter() just doesn't return them. I don't see how that can be the Composum package managers fault.

So that rather looks like a bug in vault to me. FSRegisteredPackage gets that from FSInstallState, which might have no filterset if something went wrong before that. If you can't have a look at that you might give me details about your setup and probably that package, but my gut feeling says it's not very likely that I can replicate that. I think that stuff is cached in xml files in the configured filesystem package registry - you might have a look whether that xml file is broken for your package. (I'm assuming this is from a filesystem package registry - I don't see how a JcrRegisteredPackage.getWorkspaceFilter() could return null.)

[kwin]

Indeed the NPE should not happen here, @rombert can you share a test case for FileVault with which I can reproduce it, then I can look into improving the exception handling here.

[stoerr]

The new release 4.2.1 https://github.com/ist-dresden/composum-nodes/releases/tag/composum-nodes-4.2.1 contains quite a couple of improvements in this area. You are welcome to try that out. I currently don't have further plans on that - please see how it serves in practice and let us know! Especially I'm interested in the cases where that is actually used - when is there a filesystem registry?

[rombert]

Thanks for the heads-up. I didn't have any time to look into this recently, will post back the results here when I have them.

[rombert]

@stoerr - I've updated to version 4.2.1 of Composum and 3.6.8 of FileVault, and I still don't see the filters and get the exception in the 'coverage' tab.

@kwin - I won't be able to provide a test case for FileVault, but the package registry file looks OK to me.

<?xml version="1.0" encoding="UTF-8"?>
<registryMetadata packageid="$GROUPID:$ARTIFACTID:0.1-SNAPSHOT" size="74734" installtime="1680122049777" filepath="/opt/sling/artifacts/PATH/TO/ARTIFACT/0.1-SNAPSHOT/$ARTIFACTID-0.1-SNAPSHOT.zip" external="true" packagestatus="extracted">
    <packageproperties allowIndexDefinitions="false" created="2023-03-29T20:27:03.430Z" groupId="$ARTIFACTID" name="$ARTIFACTID" description="XXXX :: Frontend Package" artifactId="$ARTIFACTID" packageType="content" version="0.1-SNAPSHOT" requiresRoot="false" group="$GROUPID"/>
</registryMetadata>

The filepath is correct (and also downloading the package works ). I can share a zip file of the launcher directory and launching instructions or a docker image (somewhere) if you think that helps.

If not, I'm happy with at least being able to inspect the registry contents so thanks for providing that :-)

[stoerr]

@rombert @kwin Sorry, I know it's ages ago, but I got detracted by something rather urgent.

I just tried @rombert 's zip of the launcher directory, and it is as I suspected: there is something missing in the registry metadata xml file. As in the previous message it just contains the packageproperties subnode, but no workspacefilter subnode, which is normally present. Interestingly, this only happens when it's installed via the org.apache.sling.jcr.packageinit.impl.ExecutionPlanRepoInitializer during startup. I don't see any error message during the installation, though. If I delete the package and upload it via the browser, the workspacefilter node is there, and thus the filters are shown in the package browser. That also happens if I take a Composum platform package and add it to the feature-oak_tar.json - the workspacefilter node is missing, too.

So it seems that happens generally when you install a package as a sling feature into a FS package registry - at least with the particular versions in that launcher, and it's not a problem with the package or the Composum package manager. I'll see whether that happens with our launchers too when I'm changing something there again.

[rombert]

Thanks for coming back to this @stoerr - don't worry about the long pauses :-)

If the way the FSPackageRegistry is set up seems problematic, perhaps @DominikSuess has an idea.