search

metafacture / metafacture-fix

Work in progress towards an implementation of the Fix language for Metafacture
Apache License 2.0
6 stars 2 forks source link

Integrate FIX into Metafacture CLI / Use FIX as additional library #90

Closed TobiasNx closed 7 months ago

TobiasNx commented 2 years ago

At the moment Metafacture Fix is only available as library for Metafacture in Java.

I would suggest that we also find a solution to integrate it as library into Metafacture CLI.

If you could install FIX as additional library to the command line programm then this would allow users of metafacture CLI to use it before metafacture-fix is part of metafacture-core. It would allow Metafacture CLI users to test FIX.

@fsteeg says that Metafacture does not support adding new libraries to the CLI-programm in general.

Catmandu has the option to install additional packages/libraries to add new featrues to the default features of Catmandu. https://librecat.org/Catmandu/#installation

For me personally it would make it easier to review new features in Metafacture-Fix since I am not depending on a full java project to test things. I also could create a repo with scenarios that are not working that could be tried. Then I could review as I do in metafacture-core tickets.

This would need an additional feature to allow install of libraries from branches similar to: https://github.com/metafacture/metafacture-core#building-metafacture-core-from-source

TL;DR Support installing FIX as library to metafacture-core CLI. Allpw installing from different branches.

blackwinter commented 2 years ago

@fsteeg says that Metafacture does not support adding new libraries to the CLI-programm in general.

But it does? You can place the (locally built) metafacture-fix jar in the plugins/ directory of your metafacture-runner distribution.

fsteeg commented 2 years ago

You can place the (locally built) metafacture-fix jar in the plugins/ directory

Oh right, I completely forgot about that. So that would be ./gradlew shadowJar → creates a jar in metafix/build/libs/. So e.g. being on the master branch in metafacture-fix, this should update a local metafacture distribution at ~/metafacture/: ./gradlew shadowJar ; cp metafix/build/libs/metafix-0.2.0-SNAPSHOT.jar ~/metafacture/plugins.

TobiasNx commented 2 years ago

Cool, Thanks @blackwinter So this works on a branch in metafacture-fix too?

blackwinter commented 2 years ago

Yes, you can build the jar from any branch: just check out the branch and build (see @fsteeg's comment).

TobiasNx commented 2 years ago

Thanks, I think the possibility to integrate plugins should be documented in the wiki of metafacture-core and also here in the metafacture-fix ReadMe.

I updated the "installation"-command so that swichting between version and updating to other branch brings no conflicts:

First installation: $ ./gradlew shadowJar ; cp metafix/build/libs/metafix-*-SNAPSHOT.jar ~/metafacture-core/plugins

Following installations: $ rm ~/metafacture-core/plugins/metafix-*-SNAPSHOT.jar ; ./gradlew shadowJar ; cp metafix/build/libs/metafix-*-SNAPSHOT.jar ~/metafacture-core/plugins

TobiasNx commented 2 years ago

okay, I tried to set it up. using a CLI version from a source build runner: metafacture-core/metafacture-runner/build/install/metafacture-core/plugins (fluxDev is my alias for flux.sh)

i get following error when running a flux:

$ fluxDev 'test1_externalMapInline.flux'
Exception in thread "main" org.metafacture.commons.reflection.ReflectionException: class could not be instantiated: class org.metafacture.metafix.Metafix
        at org.metafacture.commons.reflection.ConfigurableClass.newInstance(ConfigurableClass.java:140)
        at org.metafacture.commons.reflection.ObjectFactory.newInstance(ObjectFactory.java:105)
        at org.metafacture.flux.parser.FluxProgramm.createElement(FluxProgramm.java:78)
        at org.metafacture.flux.parser.FluxProgramm.addElement(FluxProgramm.java:90)
        at org.metafacture.flux.parser.FlowBuilder.pipe(FlowBuilder.java:736)
        at org.metafacture.flux.parser.FlowBuilder.flowtail(FlowBuilder.java:514)
        at org.metafacture.flux.parser.FlowBuilder.flow(FlowBuilder.java:226)
        at org.metafacture.flux.parser.FlowBuilder.flux(FlowBuilder.java:122)
        at org.metafacture.flux.FluxCompiler.compileFlow(FluxCompiler.java:66)
        at org.metafacture.flux.FluxCompiler.compile(FluxCompiler.java:54)
        at org.metafacture.runner.Flux.main(Flux.java:85)
Caused by: java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at org.metafacture.commons.reflection.ConfigurableClass.newInstance(ConfigurableClass.java:135)
        ... 10 more
Caused by: java.lang.NoClassDefFoundError: org/eclipse/xtext/ISetup
        at java.base/java.lang.ClassLoader.defineClass1(Native Method)
        at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
        at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
        at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:550)
        at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:458)
        at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:452)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:451)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        at java.base/java.lang.ClassLoader.defineClass1(Native Method)
        at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
        at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
        at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:550)
        at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:458)
        at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:452)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:451)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        at org.metafacture.metafix.Metafix.buildPipeline(Metafix.java:118)
        at org.metafacture.metafix.Metafix.<init>(Metafix.java:99)
        at org.metafacture.metafix.Metafix.<init>(Metafix.java:91)
        at org.metafacture.metafix.Metafix.<init>(Metafix.java:87)
        ... 15 more
Caused by: java.lang.ClassNotFoundException: org.eclipse.xtext.ISetup
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        ... 39 more

Any idea why this is not working?

fsteeg commented 2 years ago

Unassigning since we never made this ready. If it's not actually working, I suggest you use a different way of testing for now @TobiasNx and we discuss when to address this in the future.

TobiasNx commented 2 years ago

Okay agree. I will use another way and wait for it being ready. (I assigned since it was said that there is a easy way to integrate plugins.)

When making it ready we should keep in mind if this a general problem of metafacture or a problem of the fix module.

blackwinter commented 2 years ago

The task :metafix:jar doesn't create a fat jar, so it doesn't include the dependencies.

We can add a new task for that:

diff --git metafix/build.gradle metafix/build.gradle
index db7e978..8f884f2 100644
--- metafix/build.gradle
+++ metafix/build.gradle
@@ -1,5 +1,6 @@
 plugins {
   id 'com.adarshr.test-logger' version '1.7.0'
+  id 'com.github.johnrengelman.shadow' version '4.0.1'
   id 'maven-publish'
 }

@@ -28,6 +29,8 @@ dependencies {
   testImplementation "org.mockito:mockito-junit-jupiter:${versions.mockito}"
 }

+apply plugin: 'com.github.johnrengelman.shadow'
+
 configurations {
   mwe2 {
     extendsFrom implementation

The invocation would then be:

./gradlew -p metafix clean shadowJar

NOTE: Those changes alone are not sufficient long-term, because they interact badly with :metafix:jar and :metafix-ide:shadowJar.

UPDATE: Doesn't work either :disappointed: 

java.lang.NoSuchFieldError: EOF_TOKEN
        at org.eclipse.xtext.parser.antlr.Lexer.nextToken(Lexer.java:59)
        at org.antlr.runtime.BufferedTokenStream.fetch(BufferedTokenStream.java:143)
        at org.antlr.runtime.BufferedTokenStream.sync(BufferedTokenStream.java:137)
        at org.antlr.runtime.BufferedTokenStream.fill(BufferedTokenStream.java:286)
        at org.antlr.runtime.BufferedTokenStream.toString(BufferedTokenStream.java:251)
        at org.eclipse.xtext.parser.antlr.AbstractInternalAntlrParser.parse(AbstractInternalAntlrParser.java:553)
        at org.eclipse.xtext.parser.antlr.AbstractAntlrParser.doParse(AbstractAntlrParser.java:102)
        at org.eclipse.xtext.parser.antlr.AbstractAntlrParser.parse(AbstractAntlrParser.java:84)
        at org.eclipse.xtext.parser.antlr.AbstractAntlrParser.doParse(AbstractAntlrParser.java:62)
        at org.eclipse.xtext.parser.AbstractParser.parse(AbstractParser.java:33)
        at org.eclipse.xtext.resource.XtextResource.doLoad(XtextResource.java:176)
        at org.eclipse.xtext.linking.lazy.LazyLinkingResource.doLoad(LazyLinkingResource.java:114)
        at org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(ResourceImpl.java:1518)
        at org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(ResourceImpl.java:1297)
        at org.eclipse.xtext.resource.persistence.StorageAwareResource.load(StorageAwareResource.java:79)
        at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLoad(ResourceSetImpl.java:259)
        at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLoadHelper(ResourceSetImpl.java:274)
        at org.eclipse.xtext.resource.XtextResourceSet.getResource(XtextResourceSet.java:265)
        at org.eclipse.xtext.resource.SynchronizedXtextResourceSet.getResource(SynchronizedXtextResourceSet.java:25)
        at org.metafacture.metafix.validation.XtextValidator.getResource(XtextValidator.java:61)
        at org.metafacture.metafix.validation.XtextValidator.getValidatedResource(XtextValidator.java:65)
        at org.metafacture.metafix.FixStandaloneSetup.parseFix(FixStandaloneSetup.java:40)
        at org.metafacture.metafix.Metafix.buildPipeline(Metafix.java:108)
        at org.metafacture.metafix.Metafix.<init>(Metafix.java:89)
        at org.metafacture.metafix.Metafix.<init>(Metafix.java:81)
        at org.metafacture.metafix.Metafix.<init>(Metafix.java:77)
        ... 15 more
TobiasNx commented 2 years ago

I think we should make the additional integration of FIX as library for the CLI possible before the Bibliothekskongress. What do you think @fsteeg and @blackwinter ?

If we introduce FIX to the world it should be usable beyond JAVA and the playground.

blackwinter commented 2 years ago

I definitely support the effort, but I don't think I'll be able to do it. There are more pressing issues w.r.t. our pipeline that I have to address before Go Live (performance still being one of them). So I'm unassigning myself.

fsteeg commented 2 years ago

If we introduce FIX to the world it should be usable beyond JAVA and the playground.

That's why I made the metafix-runner part of the 0.2.0 release, it allows users to run fix workflows on the command line, see https://github.com/metafacture/metafacture-fix/releases/

Making it available in the core distribution remains our longer-term goal (see also https://github.com/metafacture/metafacture-fix/commit/659a5518b2477e300d0e4eb57823e3031a3fa248#r74105804).

dr0i commented 7 months ago

Is this not solved? Here is how it works for me, testing some new things not yet released, version in build.gradle is set to 1.0.1-SNAPSHOT:

./gradlew distZip
unzip ./metafix-runner/build/distributions/metafix-runner-1.0.1-SNAPSHOT.zip
# test it on a flux:
metafix-runner-1.0.1-SNAPSHOT/bin/metafix-runner ./metafix/src/test/resources/org/metafacture/metafix/integration/lookup/fromXml/toJson/lookupInDeeplyNestedArrayOfObjects/test.flux
cat ./metafix/src/test/resources/org/metafacture/metafix/integration/lookup/fromXml/toJson/lookupInDeeplyNestedArrayOfObjects/output-metafix.json
blackwinter commented 7 months ago

FYI: You don't need to create the distribution, you can execute the runner directly.

./gradlew :metafix-runner:run --args="$PWD/path/to.flux"
TobiasNx commented 7 months ago

This is a "problem" of fix as a separate repo. I haven't checked if it is still a problem, but as a workaround @fsteeg created a separate runner for fix.

I do not know if it needs to be fixed until fix will be integrated into core.

TobiasNx commented 7 months ago

I see this as a wont fix. The workaround with the fix runner and the CLI version that you described in your PR is okay for me!