metafacture / metafacture-core

Core package of the Metafacture tool suite for metadata processing.
https://metafacture.org
Apache License 2.0
69 stars 34 forks source link

`FileMap` with missing file(s) should optionally behave as if empty. #500

Open blackwinter opened 9 months ago

blackwinter commented 9 months ago

Whenever FileMap encounters a missing file (or the file cannot be read for other reasons) during access to the map (get() or keySet()), an exception is thrown and the isUninitialized flag remains unset (so it happens repeatedly on multiple accesses). For some use cases (e.g., lookup() in metafacture-fix) it might be preferable to ignore this error condition (maybe just log a warning) and continue as if the missing/unreadable file(s) were empty.


Caused by missing file map: lookup() calls map.get() which tries FileMap.init() leading to an exception every time. This then leaves the looked up value untouched even though delete: "true" was specified.

Originally posted by @blackwinter in https://github.com/hbz/lobid-resources/issues/1906#issuecomment-1742783320

blackwinter commented 9 months ago

Minimum viable solution (quietly ignoring errors):

diff --git metamorph/src/main/java/org/metafacture/metamorph/maps/FileMap.java metamorph/src/main/java/org/metafacture/metamorph/maps/FileMap.java
index 8462007e..621d0bbf 100644
--- metamorph/src/main/java/org/metafacture/metamorph/maps/FileMap.java
+++ metamorph/src/main/java/org/metafacture/metamorph/maps/FileMap.java
@@ -69,6 +69,7 @@ public final class FileMap extends AbstractReadOnlyMap<String, String> {
     private Matcher ignoreMatcher;
     private Pattern split = Pattern.compile("\t", Pattern.LITERAL);
     private boolean allowEmptyValues;
+    private boolean ignoreUnreadableFiles;
     private boolean isUninitialized = true;
     private int expectedColumns;
     private int keyColumn;
@@ -120,6 +121,18 @@ public void setIgnorePattern(final String ignorePattern) {
         this.ignoreMatcher = Pattern.compile(ignorePattern).matcher("");
     }

+    /**
+     * Sets whether to throw an exception upon encountering an unreadable file
+     * or ignore these files.
+     *
+     * <strong>Default value: false</strong>
+     *
+     * @param ignoreUnreadableFiles true if unreadable files should be ignored
+     */
+    public void setIgnoreUnreadableFiles(final boolean ignoreUnreadableFiles) {
+        this.ignoreUnreadableFiles = ignoreUnreadableFiles;
+    }
+
     /**
      * Sets a comma separated list of files which provides the {@link Map}.
      *
@@ -192,7 +205,9 @@ private void loadFile(final String file) {
             }
         }
         catch (final IOException | UncheckedIOException e) {
-            throw new MorphExecutionException("filemap: cannot read map file", e);
+            if (!ignoreUnreadableFiles) {
+                throw new MorphExecutionException("filemap: cannot read map file", e);
+            }
         }
     }