ingest of long(ish) WKT-strings fails

[csae8092]

• Ingest of working_sample.rdf works
• Ingest of failing_sample.rdf fails

run against arche-dev

acdh-oeaw/arche-assets v3.1 Set of static assets used (mainly) for ARCHE data preprocessing acdh-oeaw/arche-core 1.11.1 ARCHE repository backend acdh-oeaw/arche-doorkeeper 1.9.1 Doorkeeping handlers for the ARCHE acdh-oeaw/arche-fcs 1.0.2 CLARIN Federated Content Search plugin for ARCHE acdh-oeaw/arche-lib 1.8.0 A client library for the ACDH digital objects repository acdh-oeaw/arche-lib-disserv 2.4.0 A library providing dissemination services support on top of the acdh-oeaw/arche-lib acdh-oeaw/arche-lib-ingest 1.7.1 A library for ingesting data into the ACDH digital objects repository acdh-oeaw/arche-lib-schema 4.0.6 An API providing commonly used views on the ARCHE ontology acdh-oeaw/arche-oaipmh 1.4.0 OAI-PMH service for the ACDH repo solution acdh-oeaw/arche-resolver 2.2.0 A resolver for the ACDH digital objects repository acdh-oeaw/arche-schema v2.0.6 ACDH repository ontology acdh-oeaw/arche-schema-ingest 1.10.1 Tools for ingesting ARCHE schema into the repository acdh-oeaw/easyrdf 0.14.6 ACDH fork of the EasyRdf library acdh-oeaw/epic-handle 0.2.2 Library for managing EPIC handles (PIDs) acdh-oeaw/uri-normalizer 1.1.1 A simple class for normalizing external entity reference sources' URIs (Geonames, GND, etc. URIs). guzzlehttp/guzzle 6.5.5 Guzzle is a PHP HTTP client library guzzlehttp/promises 1.4.1 Guzzle promises library guzzlehttp/psr7 1.8.2 PSR-7 message implementation that also provides common utility methods ml/iri 1.1.4 IRI handling for PHP ml/json-ld 1.2.0 JSON-LD Processor for PHP paragonie/constant_time_encoding v2.4.0 Constant-time Implementations of RFC 4648 Encoding (Base-64, Base-32, Base-16) paragonie/random_compat v9.99.100 PHP 5.x polyfill for random_bytes() and random_int() from PHP 7 php-amqplib/php-amqplib v2.12.3 Formerly videlalvaro/php-amqplib. This library is a pure PHP implementation of the AMQP protocol. It's been tes... phpseclib/phpseclib 3.0.8 PHP Secure Communications Library - Pure-PHP implementations of RSA, AES, SSH2, SFTP, X.509 etc. pietercolpaert/hardf 0.2.0 A fast parser for RDF serializations such as turtle, n-triples, n-quads, trig and N3 psr/http-message 1.0.1 Common interface for HTTP messages psr/log 1.1.3 Common interface for logging libraries ralouphie/getallheaders 3.0.3 A polyfill for getallheaders. renanbr/bibtex-parser 2.1.1 BibTex Parser provides an API to read .bib files programmatically symfony/polyfill-intl-idn v1.22.1 Symfony polyfill for intl's idn_to_ascii and idn_to_utf8 functions symfony/polyfill-intl-normalizer v1.22.1 Symfony polyfill for intl's Normalizer class and related functions symfony/polyfill-php72 v1.22.1 Symfony polyfill backporting some PHP 7.2+ features to lower PHP versions zozlak/auth 0.7.7 Very simple yet flexible authentication framework zozlak/http-accept 0.1.0 Set of methods to deal with the HTTP Accept header zozlak/logging 0.1.1 Set of methods to deal with the HTTP Accept header zozlak/rdf-constants 1.1.2 A set of commonly used RDF and XSD constants zozlak/yaml-merge 1.0.4 A simple tool for merging & batch-editing yaml files

[zozlak]

Replicated. It's something with the EasyRdf n-triples serializer/parser.

[zozlak]

Bunch of references:

• problematic regex in EasyRdf n-triples parser
• stackoverflow issue on PREG_JIT_STACKLIMIT_ERROR
• discussion of inefficiet regex patterns

[zozlak]

Summing up:

• The problem is caused by the n-triples parser.
• We want to keep n-triples as the serialization format used by the arche-lib for performance reasons. By the way the same issue would affect the turtle format and rdf/xml and json-ld are significantly slower.
• We want to keep using regular expressions for parsing n-triples (and also turtle) because it's the only performant way to do it (we need state machine and implementing it in PHP is not performant).
  • In the long term an alternative could be to switch to using a native C n-triples parser trough a PHP C bindings (e.g. to the serd) but it will require quite a lot of work and will make user side installation painful (as it will require a custom PHP config).
• While using regular expressions we are constrained by the PCRE library settings (precisely the PCRE's JIT stack size).
  • The way PCRE is compiled in Ubuntu and using current EasyRdf regular expressions we can parse literal of length up to around 6k characters.
  • By optimizing the regular expression used we can extend it to around 25k characters. Which is noticeable but not even close to the length of https://shared.acdh.oeaw.ac.at/failing_sample.rdf (which is around 200k).
  • We can also try to recompile PCRE with a larger JIT stack but this would mean clients have to do it as well so it doesn't really seam useful.
• By the way using huge literal values brings important drawbacks to everyday repository usage. Requesting metadata which includes a resource with such a value significantly slows down the whole operation (serialization/deserialization overhead, network transfer overhead) so there should be some benefit which justifies it.

All in all I would say:

• Yes, theoretically there should be no limit on literal size.
• But from pragmatic perspective it's maybe even good there is one and ~10k characters should suite everyone ;-) If it's bigger, it should go into the binary payload.

[zozlak]

acdh-oeaw/easyrdf 0.14.7 raised the literal value length limit to around 24k characters. There are no plans to push this limit further or provide a workaround for that here (in arche-core).

The design decision in arche-core is "if it's too big to be parsed by the RDF library we depend on, it should go into the binary payload".

The spatial indexing feature planned for the arche-core 2.0.0 will cover also binary payloads.