search

LibreCat / Catmandu

Catmandu - a data processing toolkit
https://librecat.org
177 stars 31 forks source link

Rewrite JSON importer #35

Closed nichtich closed 10 years ago

nichtich commented 11 years ago

The current JSON importer can only handle line-based JSON as emitted by the JSON exporter. One should also be able to parse arbitrary JSON documents, including multiple objects in one file. For instance the following file includes three JSON objects:

{ "id": 1 }
{ 
  "id:" 2
}

{ "id": 
3 }

In addition the importer should support a path option as introduced in Catmandu-Importer-XML. For instance the following file with option path=/record.* could import the same three objects:

{
  "records": [ { "id": 1 }, { "id": 2 }, { "id": 3 } ]
}

This Perl Module may help implementing.

nichtich commented 11 years ago

Looks like incremental parsing can solve this, see http://search.cpan.org/~makamaka/JSON-PP-2.27203/lib/JSON/PP.pm#INCREMENTAL_PARSING

nics commented 10 years ago

incremental parsing has been commited, the path option still has to be done.

nichtich commented 10 years ago

We should better support a general method to split items (in the fix language) instead of this path idea, so I close this issue.