NullpointerException with SiardFromDb

bylexus commented 2 years ago

Hi,

We try to create a SIARD from an existing PostgreSQL database using SiardFromDb, which ends in a Java Nullpointer Exception. Here is what we tried:

System:

SIARD Suite: V 2.2.11
Database: PostgreSQL 12, with a very small test DB (about 5 tables, almost no data in it)
Client (that runs siardcmd): MacOS 12.6, arm64, with Azul zulu jre-fx: zulu11.58.23-ca-fx-jre11.0.16.1-macosx_aarch64

Command ran

We tried the following command:

zulu11.58.23-ca-fx-jre11.0.16.1-macosx_aarch64/bin/java -cp lib/siardcmd.jar ch.admin.bar.siard2.cmd.SiardFromDb \
    -o \
    -x=/path/to/lob/folder \
    -j=jdbc:postgresql://dbhost:5432/testdb \
    -u=testdb -p=testdb \
    -s=/path/to/output.siard \
    2>&1 | tee /path/to/siard-output.log

The command starts, and after some output, it stops with a NullPointerException:

SiardFromDb 2.1.204 - Program to store database content in a .siard file
SIARD Suite 2.1: (c) Swiss Federal Archives, Berne, Switzerland, 2008-2016
Specified by : Hartwig Thomas, Enter AG, Rüti ZH, Switzerland
               Andreas Voss, Swiss Federal Archives, Berne, Switzerland
               Anders Bo Nielsen, Danish National Archives, Denmark
               Claire Röthlisberger-Jourdan, KOST, Berne, Switzerland
Developed by : Hartwig Thomas, Enter AG, Rüti ZH, Switzerland
               Simon Jutz, Cytex GmbH, Zurich, Switzerland
Tested by    : Claudia Matthys, POOL Computer AG, Zurich, Switzerland
               Marcel Büchler, Swiss Federal Archives, Berne, Switzerland
               Yvan Dutoit, Swiss Federal Archives, Berne, Switzerland
Managed by   : Hartwig Thomas, Enter AG, Rüti ZH, Switzerland
               Marcel Büchler, Swiss Federal Archives, Berne, Switzerland
               Alain Mast, Swiss Federal Archives, Berne, Switzerland
               Krystyna Ohnesorge, Swiss Federal Archives, Berne, Switzerland
JAVA         : Version 11.0.16.1 (64)
Mac OS X     : Version 12.6 (aarch64)
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION: SiardFromDb 2.1.204 - Program to store database content in a .siard file
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION: SIARD Suite 2.1: Swiss Federal Archives, Berne, Switzerland, 2008-2016
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION: Specified by : Hartwig Thomas, Enter AG, Rüti ZH, Switzerland
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:                Andreas Voss, Swiss Federal Archives, Berne, Switzerland
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:                Anders Bo Nielsen, Danish National Archives, Denmark
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:                Claire Röthlisberger-Jourdan, KOST, Berne, Switzerland
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION: Developed by : Hartwig Thomas, Enter AG, Rüti ZH, Switzerland
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:                Simon Jutz, Cytex GmbH, Zurich, Switzerland
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION: Tested by    : Claudia Matthys, POOL Computer AG, Zurich, Switzerland
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:                Marcel Büchler, Swiss Federal Archives, Berne, Switzerland
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:                Yvan Dutoit, Swiss Federal Archives, Berne, Switzerland
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION: Managed by   : Hartwig Thomas, Enter AG, Rüti ZH, Switzerland
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:                Marcel Büchler, Swiss Federal Archives, Berne, Switzerland
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:                Alain Mast, Swiss Federal Archives, Berne, Switzerland
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:                Krystyna Ohnesorge, Swiss Federal Archives, Berne, Switzerland
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION: JAVA         : Version 11.0.16.1 (64)
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION: Mac OS X     : Version 12.6 (aarch64)
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION: free memory: 529570536
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION: total memory: 536870912
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION: maximum memory: 8589934592
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.runtime.name: OpenJDK Runtime Environment
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.vm.version: 11.0.16.1+1-LTS
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   sun.boot.library.path: /Blacked/Out/none/of/your/business/Downloads/zulu11.58.23-ca-fx-jre11.0.16.1-macosx_aarch64/zulu-11.jre/Contents/Home/lib
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   gopherProxySet: false
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   jdk.vendor.version: Zulu11.58+23-CA
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.vm.vendor: Azul Systems, Inc.
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.vendor.url: http://www.azul.com/
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   path.separator: :
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.vm.name: OpenJDK 64-Bit Server VM
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   sun.os.patch.level: unknown
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   user.country: CH
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   sun.java.launcher: SUN_STANDARD
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.vm.specification.name: Java Virtual Machine Specification
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   user.dir: /Blacked/Out/none/of/your/business/siardgui
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.vm.compressedOopsMode: Zero based
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.runtime.version: 11.0.16.1+1-LTS
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.awt.graphicsenv: sun.awt.CGraphicsEnvironment
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   os.arch: aarch64
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.io.tmpdir: /var/folders/8d/4c9w06qn75j6fc92vm_gy8dr0000gn/T/
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   line.separator: 

Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.vm.specification.vendor: Oracle Corporation
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   os.name: Mac OS X
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   sun.jnu.encoding: UTF-8
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.library.path: /Blacked/Out/none/of/your/business/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   jdk.debug: release
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.class.version: 55.0
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.specification.name: Java Platform API Specification
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   sun.management.compiler: HotSpot 64-Bit Tiered Compilers
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   os.version: 12.6
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   http.nonProxyHosts: local|*.local|169.254/16|*.169.254/16
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   user.home: /Blacked/Out/none/of/your/business
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   user.timezone: Europe/Zurich
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.awt.printerjob: sun.lwawt.macosx.CPrinterJob
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   file.encoding: UTF-8
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.specification.version: 11
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   user.name: anonymous
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.class.path: lib/siardcmd.jar
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.vm.specification.version: 11
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   sun.arch.data.model: 64
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   sun.java.command: ch.admin.bar.siard2.cmd.SiardFromDb -o -x=xxxxxxxxxxxx -j=jdbc:postgresql://xxxxxxxxx:5432/testdb -u=testdb -p=testdb -s=output.siard
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.home: /Blacked/Out/none/of/your/business/Downloads/zulu11.58.23-ca-fx-jre11.0.16.1-macosx_aarch64/zulu-11.jre/Contents/Home
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   user.language: de
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.specification.vendor: Oracle Corporation
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   awt.toolkit: sun.lwawt.macosx.LWCToolkit
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.vm.info: mixed mode
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.version: 11.0.16.1
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.vendor: Azul Systems, Inc.
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   file.separator: /
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.version.date: 2022-07-19
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.vendor.url.bug: http://www.azul.com/support/
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   sun.io.unicode.encoding: UnicodeBig
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   sun.cpu.endian: little
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   java.vendor.version: Zulu11.58+23-CA
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   socksNonProxyHosts: local|*.local|169.254/16|*.169.254/16
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   ftp.nonProxyHosts: local|*.local|169.254/16|*.169.254/16
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   sun.cpu.isalist: 
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION: 

Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION: Parameters
Parameters
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   JDBC URL               : jdbc:postgresql://xxxxxxxxxxxx:5432/testdb
  JDBC URL               : jdbc:postgresql://xxxxxxxxxxxx:5432/testdb
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   Database user          : testdb
  Database user          : testdb
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   Database password      : ***
  Database password      : ***
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   SIARD file             : /Blacked/Out/none/of/your/business/temp/gaia.siard
  SIARD file             : /Blacked/Out/none/of/your/business/temp/gaia.siard
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION:   External LOB folder    : ../lob/
  External LOB folder    : ../lob/
Okt. 06, 2022 2:38:57 NACHM. ch.enterag.utils.logging.IndentLogger log
INFORMATION: 

Connected to jdbc:postgresql://xxxxxxxxxxxx:5432/testdb
Meta Data
  Table: "public"."identity_test"
  Table: "public"."migtest"
  Table: "public"."schema_migrations"
  Table: "public"."store_test"
  Table: "public"."test"
  Table: "public"."testtable1"
  View: "public"."v_mytest"
  View: "public"."v_test"
  View: "public"."v_test2"
  View: "public"."v_test3"
  View: "public"."v_test4"
No LOB column found to be externalized!

Primary Data
  Table: "public"."identity_test"
    Record 0 (0.00 kB/s)
    Total: 0 records (0 bytes in 0 ms)
  Table: "public"."migtest"
    Record 0 (0.00 kB/s)
    Total: 0 records (0 bytes in 0 ms)
  Table: "public"."schema_migrations"
    Record 31 (432.00 kB/s)
    Total: 31 records (15'984 bytes in 37 ms)
  Table: "public"."store_test"
    Record 1 (0.00 kB/s)
    Total: 1 records (0 bytes in 1 ms)
  Table: "public"."test"
    Record 48 (470.12 kB/s)
    Total: 48 records (7'992 bytes in 17 ms)
  Table: "public"."testtable1"
  java.lang.NullPointerException:

... and that's about it.

We also tried other databases, with the same effect.

Any hints?

Thanks alex

walkerjan commented 2 years ago

Hello @bylexus

Thanks for reporting this issue. Does the execution terminate successfully if you remove -x=/path/to/lob/folder from your command? If yes, can you successfully execute the command after re-adding -x=/path/to/lob/folder to it and making sure that -s=/path/to/output.siard includes a relative reference to the parent directory of the output SIARD file (for instance, by replacing -s=./output.siard for -s=output.siard in the command)?

bylexus commented 2 years ago

Hi @walkerjan,

Unfortunately, no:

removing the -x parameter still causes a null pointer exception
also using a full/relative/no-prefix path for -s does not change anything

For me it seems more like a general non-known data type used? In our test database, it stops working after that line:

...
  Routine: "public"."gen_random_bytes"
  Routine: "public"."gen_salt"
  Routine: "public"."gen_salt"
  Routine: "public"."gin_extract_query_trgm"
  java.lang.NullPointerException:

The gin_extract_query_trgm is a Postgresql function from the Trigram extension (pg_trgm), maybe this could cause the exception?

I also tried to increase the log (in etc/logging.properties), with no luck (not more output, even if I set all the log levels to "ALL").

What kind of data would help you investigate this error further?

walkerjan commented 2 years ago

Hello @bylexus

Thanks for providing more information on the issue. Yes, I agree that the log is not particularly informative in your case. However, it seems to me that you have correctly located the problem on your own: As far as I can see, SIARD Suite currently doesn't support data types that stem from PostgreSQL extensions/modules. Can you avoid the exception by removing all extension/module-specific data types?

bylexus commented 2 years ago

Hi @walkerjan,

Thanks for your answer. Indeed if I skip / drop the postgresql specific extensions (in our case: pg_trgm, unaccent), the export works as expected. This would cause a bit of additional work in automating the siard archive generation, but is a feasible work-around for us.

thanks for your help, regards alex

bylexus commented 2 years ago

OK, my euphorism was too early :-) next problem arrives with own-defined data types (like enums): also here, we get a NullPointerException.

We extensively use own-defined types in the DB like for example like this:

CREATE TYPE public.brief_vorlage_typ AS ENUM ('XML', 'Word', 'Email');

... and use it in regular table definitions:

CREATE TABLE IF NOT EXISTS public.brief_vorlage
(
    -- .....
    brief_vorlage_typ brief_vorlage_typ,
    -- ......
)

It seems that SIARD cannot handle such types?

That makes it very hard / impossible to dump a DB with own-defined types. Or do you have any hints here?

walkerjan commented 2 years ago

Hello @bylexus

SIARD Suite should be capable of handling such custom-defined types. (Using the GUI, I just verified this by successfully creating a SIARD archive from a PostgreSQL database containing a table with a column of your custom-defined type brief_vorlage_typ.) Do some of your custom type definitions still rely on extension-/module-specific features?

bylexus commented 2 years ago

No, I removed all custom extensions. It seems that something else is bothering the tool. I try to figure out what it may be, and let you know. Perhaps I can also provide a table dump, but I have to prepare that first.

bylexus commented 2 years ago

It seems that in my particular case, my example table issues some kind of troubles, but only in the data extraction phase: The table analyzation (schema gathering) seems to be ok, but as soon as the actual data export is startet, SIARD crashes.

I created a sample database with a single table in it (brief_vorlage), including its data. I attach an SQL dump (PostgreSQL 12) for further analysis.

siardtest.sql.zip

You can restore the database with the following postgres commands:

Create an empty database directly in PostgreSQL:

CREATE DATABASE siardtest
    WITH
    OWNER = yourdbuser
    ENCODING = 'UTF8'
    LC_COLLATE = 'de_CH.UTF-8'
    LC_CTYPE = 'de_CH.UTF-8'
    TABLESPACE = pg_default
    CONNECTION LIMIT = -1
    IS_TEMPLATE = False;

Then, restore the data:

psql -1 -U yourdbuser -h yourdbhost -d siardtest -f siardtest.sql

Run the siard cmd:

java -cp lib/siardcmd.jar ch.admin.bar.siard2.cmd.SiardFromDb \
        -o \
        -j=jdbc:postgresql://yourdbhost:5432/siardtest \
        -u=yourdbuser -p=xxxxxpasswordxxxxx \
        -s=siardtest.siard

It crashes then after starting the export:

...

Connected to jdbc:postgresql://127.0.0.1:5432/siardtest
Meta Data
  Table: "public"."brief_vorlage"
  Type: "public"."anrede_form_typ"
  Type: "public"."brief_vorlage_typ"
  Type: "public"."ausgabe_typ"

Primary Data
  Table: "public"."brief_vorlage"
  java.lang.NullPointerException:

Maybe you can investigate into that matter? I would really appreciate it. If you need any more information, please let me know.

walkerjan commented 2 years ago

Hello @bylexus

Thanks for providing this well-documented testcase. As far as I can see, the problem is with column brief_beilagen_ids of array type integer[]. Please let me know whether your execution terminates successfully after deleting this column. If so, we will have to further investigate why SIARD Suite crashes by handling columns of array type integer[].

bylexus commented 2 years ago

Hello @walkerjan

Indeed, removing the integer[] column resolves the problem.

As I remember during some previous tests, SIARD is able to extract array value types. So I investigated a bit further, and found the following:

SIARD cannot handle NULL values within an array column:

create table testtable
(
    id serial primary key not null,
    numbers integer[]
);

-- Works:
INSERT INTO testtable (numbers)
VALUES
('{1,2,3}'),
('{4}'),
;

-- Inserting a NULL value (which is absolutely fine and correct for postgresql) causes SIARD to crash:
INSERT INTO testtable (numbers)
VALUES
(NULL)
;

So maybe this could be fixed in SIARD, as NULL values on non-array-types are working fine? Or do we have to clean our data to convert null array values to empty array values first?

thanks alex

walkerjan commented 2 years ago

Hello @bylexus

Thanks a lot for figuring this out. I have provided a fix for this so that NULL-values will be allowed in columns of array type. Unfortunately, you have to convert NULL-values to empty arrays until the fix will be released.

bylexus commented 2 years ago

Hi @walkerjan Great, thank you, I can live with that. We are in the early beginning of organizing archival records, so this surely has time to fix.

mburri commented 1 year ago

A fix to this issue will be included in the soon to be released next version of the siard suite application. Will be fixed with https://github.com/sfa-siard/JdbcPostgres/pull/4

bylexus commented 1 year ago

Hello @mburri

Thanks for fixing this issue. It may be that this is a problem in the GUI version, too: We created a SIARD V2 archive with another tool, containing (empty) integer array columns (null values). When opening this archive in the SIARD GUI tool (File -> Open), I get the following message:

java.io.ioException: Invalid SIARD meta data!

Console log:

SCHWERWIEGEND: XML validation for null failed at line 9621 and column 49 with message cvc-pattern-valid: Wert '<UNSUPPORTED DATATYPE>' ist nicht Facet-gxFCltig in Bezug auf Muster 'INTEGER|INT|SMALLINT|BIGINT|(NUMERIC|DECIMAL|DEC)(\s*\(\s*[1-9]\d*\s*(,\s*\d+\s*)?\))?|REAL|DOUBLE PRECISION|FLOAT(\s*\(\s*[1-9]\d*\s*\))?|(CHARACTER|CHAR)(\s*\(\s*[1-9]\d*\s*\))?|(CHARACTER\s+VARYING|CHAR\s+VARYING|VARCHAR)(\s*\(\s*[1-9]\d*\s*\))?|(CHARACTER\s+LARGE\s+OBJECT|CLOB)(\s*\(\s*[1-9]\d*(\s*(K|M|G))?\s*\))?|(NATIONAL\s+CHARACTER|NATIONAL\s+CHAR|NCHAR)(\s*\(\s*[1-9]\d*\s*\))?|(NATIONAL\s+CHARACTER\s+VARYING|NATIONAL\s+CHAR\s+VARYING|NCHAR VARYING)(\s*\(\s*[1-9]\d*\s*\))?|(NATIONAL\s+CHARACTER\s+LARGE\s+OBJECT|NCHAR\s+LARGE\s+OBJECT|NCLOB)(\s*\(\s*[1-9]\d*(\s*(K|M|G))?\s*\))?|XML|BINARY(\s*\(\s*[1-9]\d*\s*\))?|(BINARY\s+VARYING|VARBINARY)(\s*\(\s*[1-9]\d*\s*\))?|(BINARY\s+LARGE\s+OBJECT|BLOB)(\s*\(\s*[1-9]\d*(\s*(K|M|G))?\s*\))?|DATE|(TIME|TIME\s+WITH\s+TIME\s+ZONE)(\s*\(\s*[1-9]\d*\s*\))?|(TIMESTAMP|TIMESTAMP\s+WITH\s+TIME\s+ZONE)(\s*\(\s*(0|([1-9]\d*))\s*\))?|INTERVAL\s+(((YEAR|MONTH|DAY|HOUR|MINUTE)(\s*\(\s*[1-9]\d*\s*\))?(\s+TO\s+(MONTH|DAY|HOUR|MINUTE|SECOND)(\s*\(\s*[1-9]\d*\s*\))?)?)|(SECOND(\s*\(\s*[1-9]\d*\s*(,\s*\d+\s*)?\))?))|BOOLEAN' fxFCr Typ 'predefinedTypeType'.
... more content omitted ...

Maybe this is caused by the same root cause?

sfa-siard / SiardGui

NullpointerException with SiardFromDb #48

System:

Command ran