PHP Warning: array_slice() expects parameter 1 to be array, null given in /html/typo3/typo3conf/ext/external_import/Classes/Step/ReadDataStep.php line 221

[AstCommodore]

Hello !

I've got a problem reading a JSON file and it seems to be a matter of file size. The JSON import was successful some month ago and nothing changed in the meantime (base programming, extension versions) excluding TYPO3 version, it's 8.7.17 now.

The JSON structure (Array with many many objects) itself has grown and the file size is about 2.5Mb, the array has grown by round about 100 new entries/objects.

When I read the file by simple PHP it don't fail, see:

<?php
$data = file_get_contents("URL");

var_dump($data) . "<br>";

$json = json_decode($data);

if ($json)
    echo $json;
else
    echo json_last_error_msg();

When I start the import by external_import (4.0.2) + SV connector (3.2.4) + JSON connector (2.2.2), anything is imported and the TYPO3 log gets a new entry:

PHP Warning: array_slice() expects parameter 1 to be array, null given in /html/typo3/typo3conf/ext/external_import/Classes/Step/ReadDataStep.php line 221

Devlog shows the following entries:

svconnector_json:

Call parameters and headers -> Valid extra data Structured data -> No extra data

extrernal_import:

Data received (sample) -> No extra data Data received for storage -> No extra data TCEmain data -> Empty extra data, only an empty array for the mapped table fe_users TCEmain commands -> No extra data New IDs -> No extra data Synchronisation of table fe_users -> Report

When I cut the JSON and imported it, I've got the following:

svconnector_json:

Structured data -> False (as info in extra data)

extrernal_import:

Data received (sample) -> Valid example Data received for storage -> Valid mapped data TCEmain data -> Valid mapped data TCEmain commands -> No extra data New IDs -> Array with mapped data Synchronisation of table fe_users -> Report

Looks great with a smaller file! So maybe you've got a hint to me how to handle the bigger file? Have I split it up into more or is there a (programming) solution?

[fsuter]

Hi. It seems indeed like svconnector_json is chocking on a larger file, but I don't know why. I can't see any easy way out of this. One possibility would be to creation your own JSON connector and adapt the code to use the plain PHP you mention above and that works for you.

Could you maybe send me your file by PM so that I can run tests (if it does not contain sensitive data)?

[AstCommodore]

Hi @fsuter,

I'm sorry, the file contains sensitive data, so I can't provide it to you. Meanwhile I figured out, that it's possible to import the file locally: I saved the file form the external source and changed the import URL to my local dev. The import works like charm!

TYPO3 runs at 8.7.12, external_import and the connectors all at the latest versions, PHP 7.2.4 with cURL 7.54.0 and JSON 1.6.0. I couldn't repeat the test on staging / production system yet. All systems couldn't import the file from the origin source, but – by now – one with a local copy.

So it seems that it could be a fault on my origin source server? Maybe a timeout? I will catch up the tests on staging soon an report you. Hope this is a little help.

[fsuter]

Hmm, could be a timeout indeed. I took a look at the source code and the fact is that the file is fetched using TYPO3's GeneralUtility::getUrl(), so if there's an issue, it is not specific to svconnector IMO. Especially since svconnector_json used the same method directly before the FileUtility class was introduced.

[AstCommodore]

Hi @fsuter

the source server deliver a cached JSON by now, no dynamic generated output anymore. This is the key: The timeout is wind up by this solution and there is no failure in your extension.

Maybe you could add a warning for this or similar cases, so it gets more clear for the admin if something is wrong with the source. Otherhand you may close this issue.

[fsuter]

I see your point. It would be interesting to check if we get a significant information when a timeout occurs. As said before, most connector services rely on the Core's GeneralUtility::getUrl(), but I don't really check the return status codes. If there's something significant, the connector service could throw an exception, which could then be used by External Import to abort the process.

This is interesting to investigate and I'll keep this issue open as a reminder.