search

AtesComp / rdf-transform

RDF Transform is an extension for OpenRefine to transform data into RDF formats.
Other
27 stars 7 forks source link

RDF-Transform causing Java exceptions when faceting or merging clusters in OpenRefine? #20

Open smurphy13IIT opened 2 days ago

smurphy13IIT commented 2 days ago

I submitted this issue to the OpenRefine repo, and it was recommended to me to submit an issue here as well after another user inspected the OpenRefine server console output. Here's a link to my reply with the server printout.

Has anyone else seen a similar thing happen? The colleague that I'm working with isn't familiar with RDF-Transform; is it an extension that is included in fresh OpenRefine installations, or must it be manually added?

Thanks so much!

AtesComp commented 2 days ago

Extremely odd!

RDF Transform is installed by the user. I don't think it's being packaged by the OpenRefine installer...no one told me if it is.

In any case, the RDF Transform code should not be called by the functions you are using. Some kind of cross contamination is occurring. This could be a side effect of some process in the OpenRefine code confusing some function call with an RDF Transform function.

The thread pool / worker looks like it's getting confused and aligning on RDF Transform functions. Possibly loading the extensions in a different order or removing RDF Transform would push the issue elsewhere. I'm as baffled as you are!

juandql commented 2 days ago

Hi,

I have been facing the same problem. I have uploaded a csv file and could activate all facets and filters that I needed. Then I installed RDF Transform v2.2.2 and suddenly facets give me the exception:

java.lang.IllegalArgumentException: cannot change field "project" from index options=DOCS to inconsistent index options=NONE

I'm happy to see that I'm not the only one, but it would be good to find a solution to that :) Thanks in advance for the support.

AtesComp commented 1 day ago

I'm looking into the current issue and documented at the other submitted issue for extra visibility but am repeating here...

@smurphy13IIT Thanks! That helps a bit.

It appears that someone has loaded RDF Transform AND the RDF Extension. Loading both of these "might" cause issues since RDF Transform is a rewrite of RDF Extension (I doubt it, but who knows at this point). I have not tested the effect that may have on OpenRefine. Note that the "RDFT:..." messages are from RDF Transform and the "rdf_extension" messages are from the older RDF Extension.

There also appears to be many issues loading project metadata in general.

There is some problem with the Lucene store RDF Transform uses to manage ontology information. The Lucene library itself is complaining about it at org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:216). In the past, I've just deleted the extension's Lucene store and let it recreate itself.

Notice:

10:28:52.318 [             RDFT:InitCmd] Preferences: { bDebugJSON:false, iSampleLimit:20, iVerbosity:0, iExportLimit:10737418, bPreviewStream:false, bDebugMode:false } (6ms)

You can set the "Verbosity" higher and the "DebugMode" true as per the documentation in Features -> Preferences at: https://github.com/AtesComp/rdf-transform/wiki/Features This will help with narrowing down the issue. They are set via the standard OpenRefine preferences. Rerun and post another log.

RDF Transform attempts the register the current project with itself just after

10:30:04.199 [                   refine] POST /command/core/mass-edit (6ms)

The missing metadata for the projects may or may not be a related issue as RDF Transform needs certain metadata from the project for registration. Clearing up the other issues may help scope the problem down to RDF Transform or at least indicate some checks it can perform if related to the other issues.

AtesComp commented 1 day ago

@juandql I'd like to see the logs so we can narrow it down. There may be similar project metadata issues. Also, use the preferences as mentioned above. Thanks in advance!

juandql commented 14 hours ago

Hi @AtesComp thanks for your messages. For more information, I'm just trying to add the extension RDF Transform. I have not downloaded the one called RDF Extension.

I have set the preferences as you recommended and have here added the logs. I have opened a project that I have, and with RDF Transform already installed, I try to use the facet for one of the fields. And then I get the Execption error message.

I suspect that I have my error can be related to all the JRE, JDK versioning and installation, since I get the same error message described in this post here when I try to launch RDF Transform: ERROR: Could not retrieve default namespaces. I personally find this Java versions too complicated, maybe that's a big part of my problem :)

Thank you in advance for any help!

RDF_Transform_Facet_Execption_log.txt

Additional Information:

AtesComp commented 9 hours ago

@juandql Reviewing the log, you don't seem to have any Java version issue (Current Java VM Version: 11.0.25) and RDF Transform initializes without any issues.

The issue:

java.lang.IllegalArgumentException: cannot change field "project" from index options=DOCS to inconsistent index options=NONE
        at org.apache.lucene.index.FieldInfo.verifySameIndexOptions(FieldInfo.java:273)
...
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1485)
        at org.openrefine.rdf.model.vocab.VocabularySearcher.addGlobalDocumentsToProject(VocabularySearcher.java:519)
        at org.openrefine.rdf.model.vocab.VocabularySearcher.addPredefinedVocabulariesToProject(VocabularySearcher.java:204)
        at org.openrefine.rdf.RDFTransform.<init>(RDFTransform.java:391)
        at org.openrefine.rdf.RDFTransform.getRDFTransform(RDFTransform.java:116)
        at org.openrefine.rdf.model.expr.RDFTransformBinder.initializeBindings(RDFTransformBinder.java:67)
        at com.google.refine.expr.ExpressionUtils.createBindings(ExpressionUtils.java:69)
...

is a similar Lucene error. It looks like it errors twice.

The Lucene error has been reported in other software systems as well.

The current Lucene versions packaged with RDF Transform:

Lucene is used to store default and configured ontology elements for selection when creating an RDF transform for the data.

As a temporary fix, deleteing the Lucene store may help as it will regenerate when RDF Transform initializes. See: https://github.com/AtesComp/rdf-transform/wiki/Lucene

I'm working on getting a new version to help mitigate this issue as soon as I can.

juandql commented 9 hours ago

Hi @AtesComp I appreciate the new information. But deleting the Lucene folder within cache hasn't changed the output. I keep receiving the exeption error while trying to apply facets and when I click on Edit RDF Transform I get again the error message "Error while retrieving default namespaces!".

I've created a custom JRE from the OpenJDK as detailed in the prerequisites to installation and have changed the JAVA_HOME environment variable to that custom JRE. But unfortunately I keep getting the same errors.

I appreciate any further support. I would love to use OpenRefine and RDF Transform to work on my Master Thesis, so that's a big issue for me in the moment.

Thanks!

AtesComp commented 8 hours ago

I don't think the JRE has anything to do with the issue at this point. You are using an approved Java version. Kudos to you for creating a custom JRE.

If you are using the OpenRefine Win installer, it should have an approved Java version installer. So, you shouldn't need to do any other Java install process at all.

The "Error while retrieving default namespaces!" is a different error. The log you gave shows:

11:31:43.790 [       RDFT:VocabSearcher] Creating vocabulary searcher... (120ms)
11:31:44.562 [       RDFT:VocabSearcher] ...created vocabulary searcher (772ms)
11:31:44.562 [      RDFT:PredefVocabMgr] Attempting vocabulary reconstruct... (0ms)
11:31:44.570 [          RDFT:Vocabulary] DEBUG: Prefix:[rdf] Namespace:[http://www.w3.org/1999/02/22-rdf-syntax-ns#] (7ms)
11:31:44.570 [          RDFT:Vocabulary] DEBUG: Prefix:[rdfs] Namespace:[http://www.w3.org/2000/01/rdf-schema#] (0ms)
11:31:44.574 [          RDFT:Vocabulary] DEBUG: Prefix:[owl] Namespace:[http://www.w3.org/2002/07/owl#] (4ms)
11:31:44.577 [          RDFT:Vocabulary] DEBUG: Prefix:[xsd] Namespace:[http://www.w3.org/2001/XMLSchema#] (3ms)
11:31:44.577 [          RDFT:Vocabulary] DEBUG: Prefix:[vcard] Namespace:[http://www.w3.org/2006/vcard/ns#] (0ms)
11:31:44.577 [          RDFT:Vocabulary] DEBUG: Prefix:[foaf] Namespace:[http://xmlns.com/foaf/0.1/] (0ms)
11:31:44.577 [      RDFT:PredefVocabMgr] ...reconstructed (0ms)
11:31:44.620 [          RDFT:AppContext] ...Context initialized. (43ms)

so all was good with that. The "Error while retrieving default namespaces!" usually means it couldn't get some ontology from the Internet for some reason. You should be able to read the log to see the offending ontology. You could manually download any ontologies you need and add them yourself. You can also remove them from the default by editing the PredefinedVocabs file.

If you really want to make sure it isn't a Java issue...

Install the full JDK since a custom JRE may be missing needed libs, etc, depending on how it was built. It's a bit of an overkill to do this, but could eliminate other issues. Remember to delete the Lucene data files as well so they are created fresh.

AtesComp commented 8 hours ago

I just saw an issue with the FOAF ontology: http://xmlns.com/foaf/0.1/ All the other ontologies were reachable.

Remove FOAF from the defaults. If you want it, download from FOAF Latest and manually add it.

juandql commented 7 hours ago

I have removed FOAF ontology from the predefined namespaces, but I didn't changed the output. I'm still getting this result:

image

Preloading the namespaces wouldn't be for me any problem, but in the same time, I cannot upload any single ontology from my files. If I try for example to add the OWL ontology that as saved as a rdf file, I get also an error message, as shown below:

image

I'm out of things I could try, I have erased the cache files before every try. I have installed full OpenJDK.

At one point, when I executed OpenRefine, I have received a different error message:

Nov 22, 2024 5:42:13 PM org.apache.lucene.util.HotspotVMOptions <clinit> WARNING: Lucene cannot optimize algorithms or calculate object sizes for JVMs that are not based on Hotspot or a compatible implementation. Nov 22, 2024 5:42:13 PM org.apache.lucene.store.MappedByteBufferIndexInputProvider <init> WARNING: Unmapping is not supported on this platform, because internal Java APIs are not compatible with this Lucene version: java.lang.ClassNotFoundException: sun.misc.Unsafe

Does this help you understanding where my problem could be?

AtesComp commented 4 hours ago

All good info. I'm looking into it.

AtesComp commented 57 minutes ago

Interesting. I just ran OpenRefine under Java 21 JDK. I get the "Error while retrieving default namespaces" as well.

It exposed some additional Java Config Parameters. I also found a Very Hacky Fix for the above error.

Java Config Parameters

Java 21 is reporting some "warnings" about some language issue, Lucene, MMapDirectory, and vectors. To "fix" them, it also reported some Java config parameters to add.

In the refine.ini file add or modify the JAVA_OPTIONS variable:

JAVA_OPTIONS=--enable-native-access=ALL-UNNAMED -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false --add-modules jdk.incubator.vector

The MMapDirectory issue is interesting as some of the errors seem related. I researched this I found many other Lucene reports with similar errors. Will this help? IDK! I didn't see this specific error. Java 21 JDK may have something to do with that since it reported only a "warning". Retry with these options.

It's also good to review and set your REFINE_MEMORY and REFINE_MIN_MEMORY vars here as needed.

Very Hacky Fix

This was rather humorous to me.

  1. Shut down OpenRefine
  2. Edit the .../openrefine/extensions/rdf-transform/module/MOD-INF/classes/files/PredefinedVocabs file
    • Or backup the original and edit
  3. Remove all the entries and save the file--a single empty line is OK
  4. Delete the .../openrefine/cache/rdf-transform directory
  5. Start OpenRefine
  6. Open the affected project
  7. Select the RDF Transform extension -> edit RDF Transform...
  8. NO ERROR!?!
  9. Shut down OpenRefine
  10. Edit the .../openrefine/extensions/rdf-transform/module/MOD-INF/classes/files/PredefinedVocabs file
  11. Re-add the removed entries and save the file (or restore a backup copy): 
    rdf    http://www.w3.org/1999/02/22-rdf-syntax-ns#  http://www.w3.org/1999/02/22-rdf-syntax-ns#
rdfs   http://www.w3.org/2000/01/rdf-schema#        http://www.w3.org/2000/01/rdf-schema#
owl    http://www.w3.org/2002/07/owl#               http://www.w3.org/2002/07/owl#
xsd    http://www.w3.org/2001/XMLSchema#
vcard  http://www.w3.org/2006/vcard/ns#             http://www.w3.org/2006/vcard/ns#
foaf   http://xmlns.com/foaf/0.1/                   http://xmlns.com/foaf/spec/index.rdf
  12. Repeat from step 4 to 8

This apparently initializes the namespaces in the project enough to overcome the error. It suggests that the project namespaces need to be initialized to an empty set before I load the actual namespaces. I'm reviewing the affected code.

NOTE: This only solves this one "Error while retrieving default namespaces" issue. Also, I've only tested using a Linux machine.

NOTE: FOAF does not appear to be an issue--false alarm!