Overwrite of exisiting entries does not work

[metapublic-gbr]

Typo3 11.5.34 xls_import 4.0.3

I tried to overwrite data of an existing entry in tt_address. Imported csv contained: last_name, uid.

Import didn't work.

Problem occurs in line 425 in XlsimportController: $inserts[$table][$update ? $import['uid'] : uniqid('NEW_', true)] = $insertArray;

Changed it to: $inserts[$table][$update ? $import[arraysearch('uid',$fields)] : uniqid('NEW', true)] = $insertArray;

Now the overwrite works.

This seems to be a rather bold bug. Since I have no overview of the extension, I don't know, if I am missing something.

If this can be confirmed, I could create a pull request.

[calien666]

Thank you for your issue report.

Some hours ago I released 5.0 of the extension, with a complete rework of the ImportController. Would be nice to know, if the problem occurs there, too.

[metapublic-gbr]

Now it doesn't work at all anymore; I can't choose tables to import to, even if I set the tablenames in

module.tx_xlsimport {
  settings {
    allowedTables := addToList(tt_address,pages)
  }
}

The dropdown in BE remains empty. Did something change in this respect?

[calien666]

Yes, we dropped TypoScript support completely, as TypoScript is for Frontend rendering nad configuring. Just take this setup and move it to page TSconfig, either in your page settings or in an extension, like your flavour.

[metapublic-gbr]

Ok. Moving the config to page TSconfig helped a step further.

Now uploading a csv-file works.

importing one line with existing uid and changed field "last_name" into tt_address fails with:

(1/1) #1476107295 TYPO3\CMS\Core\Error\Exception

PHP Warning: array_replace_recursive(): Expected parameter 1 to be an array, null given in /var/www/html/site/private/typo3/sysext/core/Classes/DataHandling/DataHandler.php line 1779

Looks like an expected array is not filled for the dataHandler

[calien666]

Okay, I'll take a look. Can you share the stack trace here? As I see, you want to change address data, not import new ones?

Should be a small problem, I guess. Will write a test for that and then fix the issue.

[metapublic-gbr]

...hmm... Don't know what is the best way.... Here is the full error message from Typo3:

PHP Warning: array_replace_recursive(): Expected parameter 1 to be an array, null given in /var/www/html/site/private/typo3/sysext/core/Classes/DataHandling/DataHandler.php line 1779 in /var/www/html/site/private/typo3/sysext/core/Classes/Error/ErrorHandler.php line 137

    }

    $message = self::ERROR_LEVEL_LABELS[$errorLevel] . ': ' . $errorMessage . ' in ' . $errorFile . ' line ' . $errorLine;
    if ($errorLevel & $this->exceptionalErrors) {
        throw new Exception($message, 1476107295);
    }

    $message = $this->getFormattedLogMessage($message);

at TYPO3\CMS\Core\Error\ErrorHandler->handleError() at array_replace_recursive() in /var/www/html/site/private/typo3/sysext/core/Classes/DataHandling/DataHandler.php line 1779

protected function checkValueForSlug(string $value, array $tcaFieldConf, string $table, $id, int $realPid, string $field, array $incomingFieldArray = []): array
{
    $workspaceId = $this->BE_USER->workspace;
    $helper = GeneralUtility::makeInstance(SlugHelper::class, $table, $field, $tcaFieldConf, $workspaceId);
    $fullRecord = array_replace_recursive($this->checkValue_currentRecord, $incomingFieldArray ?? []);
    // Generate a value if there is none, otherwise ensure that all characters are cleaned up
    if ($value === '') {
        $value = $helper->generate($fullRecord, $realPid);
    } else {

at TYPO3\CMS\Core\DataHandling\DataHandler->checkValueForSlug() in /var/www/html/site/private/typo3/sysext/core/Classes/DataHandling/DataHandler.php line 1537

        case 'input':
            $res = $this->checkValueForInput($value, $tcaFieldConf, $table, $id, $realPid, $field);
            break;
        case 'slug':
            $res = $this->checkValueForSlug((string)$value, $tcaFieldConf, $table, $id, (int)$realPid, $field, $additionalData['incomingFieldArray'] ?? []);
            break;
        case 'language':
            $res = $this->checkValueForLanguage((int)$value, $table, $field);
            break;

at TYPO3\CMS\Core\DataHandling\DataHandler->checkValue_SW() in /var/www/html/site/private/typo3/sysext/core/Classes/DataHandling/DataHandler.php line 1471

        $recFID = '';
    }

    // Perform processing:
    $res = $this->checkValue_SW($res, $value, $tcaFieldConf, $table, $id, $curValue, $status, $realPid, $recFID, $field, $tscPID, ['incomingFieldArray' => $incomingFieldArray]);
    return $res;
}

/**

at TYPO3\CMS\Core\DataHandling\DataHandler->checkValue() in /var/www/html/site/private/typo3/sysext/core/Classes/DataHandling/DataHandler.php line 1328

                break;
            default:
                if (isset($GLOBALS['TCA'][$table]['columns'][$field])) {
                    // Evaluating the value
                    $res = $this->checkValue($table, $field, $fieldValue, $id, $status, $realPid, $tscPID, $incomingFieldArray);
                    if (array_key_exists('value', $res)) {
                        $fieldArray[$field] = $res['value'];
                    }
                    // Add the value of the original record to the diff-storage content:

at TYPO3\CMS\Core\DataHandling\DataHandler->fillInFieldArray() in /var/www/html/site/private/typo3/sysext/core/Classes/DataHandling/DataHandler.php line 1066

                // This allows to set TCAdefaults (for example) without having to use FormEngine to have the fields available first.
                $incomingFieldArray = array_replace_recursive($fieldArray, $incomingFieldArray);
            }
            // Processing of all fields in incomingFieldArray and setting them in $fieldArray
            $fieldArray = $this->fillInFieldArray($table, $id, $fieldArray, $incomingFieldArray, $theRealPid, $status, $tscPID);
            // NOTICE! All manipulation beyond this point bypasses both "excludeFields" AND possible "MM" relations to field!
            // Forcing some values unto field array:
            // NOTICE: This overriding is potentially dangerous; permissions per field is not checked!!!
            $fieldArray = $this->overrideFieldArray($table, $fieldArray);

at TYPO3\CMS\Core\DataHandling\DataHandler->process_datamap() in /var/www/html/site/private/typo3conf/ext/xlsimport/Classes/Service/ImportService.php line 105

    }
    $dataHandler = GeneralUtility::makeInstance(DataHandler::class);
    $dataHandler->start($importJob->getData(), $cmd);
    $dataHandler->process_cmdmap();
    $dataHandler->process_datamap();
}

/**
 * @param array<string, array<int|string, array<string, mixed>>> $insertData

at SUDHAUS7\Xlsimport\Service\ImportService->writeImport() in /var/www/html/site/private/typo3conf/ext/xlsimport/Classes/Controller/DataSheetImportController.php line 379

        return new RedirectResponse($uri);
    }

    $importService->prepareImport($importJob);
    $importService->writeImport($importJob);

    $message = GeneralUtility::makeInstance(
        FlashMessage::class,
        $this->languageService->sL('LLL:EXT:xlsimport/Resources/Private/Language/locallang.xlf:success'),

at SUDHAUS7\Xlsimport\Controller\DataSheetImportController->importAction() in /var/www/html/site/private/typo3conf/ext/xlsimport/Classes/Controller/DataSheetImportController.php line 110

    /**
     * Call the passed in action
     */
    return $this->{$action . 'Action'}($pageId, $moduleTemplate, $request);
}

private function indexAction(
    int $pageId,

at SUDHAUS7\Xlsimport\Controller\DataSheetImportController->handleRequest() in /var/www/html/site/private/typo3/sysext/backend/Classes/Http/RouteDispatcher.php line 91

    }
    $targetIdentifier = $route->getOption('target');
    $target = $this->getCallableFromTarget($targetIdentifier);
    $arguments = [$request];
    return $target(...$arguments);
}

/**
 * Wrapper method for static form protection utility

at TYPO3\CMS\Backend\Http\RouteDispatcher->dispatch() in /var/www/html/site/private/typo3/sysext/backend/Classes/Http/RequestHandler.php line 110

    // there are Core classes that need the Request object but do not get it handed in
    $this->resetGlobalsToCurrentRequest($request);
    try {
        // Check if the router has the available route and dispatch.
        return $this->dispatcher->dispatch($request);
    } catch (MissingRequestTokenException $e) {
        // When token was missing, then redirect to login, but keep the current route as redirect after login
        $loginUrl = $this->uriBuilder->buildUriWithRedirect(
            'login',

at TYPO3\CMS\Backend\Http\RequestHandler->handle() in /var/www/html/site/private/typo3/sysext/core/Classes/Middleware/ResponsePropagation.php line 34

{ public function process(ServerRequestInterface $request, RequestHandlerInterface $handler): ResponseInterface { try { $response = $handler->handle($request); } catch (PropagateResponseException $e) { $response = $e->getResponse(); }

at TYPO3\CMS\Core\Middleware\ResponsePropagation->process() in /var/www/html/site/private/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php line 172

            if (!$middleware instanceof MiddlewareInterface) {
                throw new \InvalidArgumentException(get_class($middleware) . ' does not implement ' . MiddlewareInterface::class, 1516821342);
            }
            return $middleware->process($request, $this->next);
        }
    };
}

}

at class@anonymous/var/www/html/site/private/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php:138$2c3f->handle() in /var/www/html/site/private/typo3/sysext/backend/Classes/Middleware/SiteResolver.php line 69

        }
        $site = $this->siteMatcher->matchByPageId($pageId, $rootLine);
        $request = $request->withAttribute('site', $site);
    }
    return $handler->handle($request);
}

}

at TYPO3\CMS\Backend\Middleware\SiteResolver->process() in /var/www/html/site/private/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php line 172

            if (!$middleware instanceof MiddlewareInterface) {
                throw new \InvalidArgumentException(get_class($middleware) . ' does not implement ' . MiddlewareInterface::class, 1516821342);
            }
            return $middleware->process($request, $this->next);
        }
    };
}

}

at class@anonymous/var/www/html/site/private/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php:138$2c3f->handle() in /var/www/html/site/private/typo3/sysext/backend/Classes/Middleware/AdditionalResponseHeaders.php line 41

 * @return ResponseInterface
 */
public function process(ServerRequestInterface $request, RequestHandlerInterface $handler): ResponseInterface
{
    $response = $handler->handle($request);
    foreach ($GLOBALS['TYPO3_CONF_VARS']['BE']['HTTP']['Response']['Headers'] ?? [] as $header) {
        [$headerName, $value] = explode(':', $header, 2);
        $response = $response->withAddedHeader($headerName, trim($value));
    }

at TYPO3\CMS\Backend\Middleware\AdditionalResponseHeaders->process() in /var/www/html/site/private/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php line 172

            if (!$middleware instanceof MiddlewareInterface) {
                throw new \InvalidArgumentException(get_class($middleware) . ' does not implement ' . MiddlewareInterface::class, 1516821342);
            }
            return $middleware->process($request, $this->next);
        }
    };
}

}

at class@anonymous/var/www/html/site/private/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php:138$2c3f->handle() in /var/www/html/site/private/typo3/sysext/backend/Classes/Middleware/OutputCompression.php line 47

    ob_clean();
    // Initialize output compression if configured
    $this->initializeOutputCompression();

    return $handler->handle($request);
}

/**
 * Initialize output compression if configured

at TYPO3\CMS\Backend\Middleware\OutputCompression->process() in /var/www/html/site/private/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php line 172

            if (!$middleware instanceof MiddlewareInterface) {
                throw new \InvalidArgumentException(get_class($middleware) . ' does not implement ' . MiddlewareInterface::class, 1516821342);
            }
            return $middleware->process($request, $this->next);
        }
    };
}

}

at class@anonymous/var/www/html/site/private/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php:138$2c3f->handle() in /var/www/html/site/private/typo3/sysext/backend/Classes/Middleware/BackendUserAuthenticator.php line 166

    }
    $GLOBALS['LANG'] = $this->languageServiceFactory->createFromUserPreferences($GLOBALS['BE_USER']);
    // Re-setting the user and take the workspace from the user object now
    $this->setBackendUserAspect($GLOBALS['BE_USER']);
    $response = $handler->handle($request);
    $this->sessionGarbageCollection();
    return $this->enrichResponseWithHeadersAndCookieInformation($response, $GLOBALS['BE_USER']);
}

at TYPO3\CMS\Backend\Middleware\BackendUserAuthenticator->process() in /var/www/html/site/private/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php line 172

            if (!$middleware instanceof MiddlewareInterface) {
                throw new \InvalidArgumentException(get_class($middleware) . ' does not implement ' . MiddlewareInterface::class, 1516821342);
            }
            return $middleware->process($request, $this->next);
        }
    };
}

}

at class@anonymous/var/www/html/site/private/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php:138$2c3f->handle() in /var/www/html/site/private/typo3/sysext/backend/Classes/Middleware/BackendRouteInitialization.php line 86

        $uri = GeneralUtility::makeInstance(UriBuilder::class)->buildUriFromRoute('login');
        return new RedirectResponse($uri);
    }

    return $handler->handle($request);
}

}

at TYPO3\CMS\Backend\Middleware\BackendRouteInitialization->process() in /var/www/html/site/private/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php line 172

            if (!$middleware instanceof MiddlewareInterface) {
                throw new \InvalidArgumentException(get_class($middleware) . ' does not implement ' . MiddlewareInterface::class, 1516821342);
            }
            return $middleware->process($request, $this->next);
        }
    };
}

}

at class@anonymous/var/www/html/site/private/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php:138$2c3f->handle() in /var/www/html/site/private/typo3/sysext/backend/Classes/Middleware/ForcedHttpsBackendRedirector.php line 55

        [$server, $address] = explode('/', $url, 2);
        return new RedirectResponse('https://' . $server . $sslPortSuffix . '/' . $address);
    }

    return $handler->handle($request);
}

}

at TYPO3\CMS\Backend\Middleware\ForcedHttpsBackendRedirector->process() in /var/www/html/site/private/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php line 172

            if (!$middleware instanceof MiddlewareInterface) {
                throw new \InvalidArgumentException(get_class($middleware) . ' does not implement ' . MiddlewareInterface::class, 1516821342);
            }
            return $middleware->process($request, $this->next);
        }
    };
}

}

at class@anonymous/var/www/html/site/private/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php:138$2c3f->handle() in /var/www/html/site/private/typo3/sysext/backend/Classes/Middleware/LockedBackendGuard.php line 75

        $request->getAttribute('normalizedParams')->getRemoteAddress(),
        trim((string)$GLOBALS['TYPO3_CONF_VARS']['BE']['IPmaskList'])
    );

    return $handler->handle($request);
}

/**
 * Check adminOnly configuration variable and redirects to an URL in file typo3conf/LOCK_BACKEND

at TYPO3\CMS\Backend\Middleware\LockedBackendGuard->process() in /var/www/html/site/private/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php line 172

            if (!$middleware instanceof MiddlewareInterface) {
                throw new \InvalidArgumentException(get_class($middleware) . ' does not implement ' . MiddlewareInterface::class, 1516821342);
            }
            return $middleware->process($request, $this->next);
        }
    };
}

}

at class@anonymous/var/www/html/site/private/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php:138$2c3f->handle() in /var/www/html/site/private/typo3/sysext/core/Classes/Middleware/NormalizedParamsAttribute.php line 45

 */
public function process(ServerRequestInterface $request, RequestHandlerInterface $handler): ResponseInterface
{
    $request = $request->withAttribute('normalizedParams', NormalizedParams::createFromRequest($request));
    return $handler->handle($request);
}

}

at TYPO3\CMS\Core\Middleware\NormalizedParamsAttribute->process() in /var/www/html/site/private/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php line 172

            if (!$middleware instanceof MiddlewareInterface) {
                throw new \InvalidArgumentException(get_class($middleware) . ' does not implement ' . MiddlewareInterface::class, 1516821342);
            }
            return $middleware->process($request, $this->next);
        }
    };
}

}

at class@anonymous/var/www/html/site/private/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php:138$2c3f->handle() in /var/www/html/site/private/typo3/sysext/core/Classes/Middleware/VerifyHostHeader.php line 55

            1396795884
        );
    }

    return $handler->handle($request);
}

/**
 * Checks if the provided host header value matches the trusted hosts pattern.

at TYPO3\CMS\Core\Middleware\VerifyHostHeader->process() in /var/www/html/site/private/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php line 172

            if (!$middleware instanceof MiddlewareInterface) {
                throw new \InvalidArgumentException(get_class($middleware) . ' does not implement ' . MiddlewareInterface::class, 1516821342);
            }
            return $middleware->process($request, $this->next);
        }
    };
}

}

at class@anonymous/var/www/html/site/private/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php:138$2c3f->handle() in /var/www/html/site/private/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php line 78

 * @return ResponseInterface
 */
public function handle(ServerRequestInterface $request): ResponseInterface
{
    return $this->tip->handle($request);
}

/**
 * Seed the middleware stack with the inner request handler

at TYPO3\CMS\Core\Http\MiddlewareDispatcher->handle() in /var/www/html/site/private/typo3/sysext/core/Classes/Http/AbstractApplication.php line 86

 */
public function handle(ServerRequestInterface $request): ResponseInterface
{
    try {
        $response = $this->requestHandler->handle($request);
    } catch (ImmediateResponseException $exception) {
        $response = $exception->getResponse();
    }
    return $response;

at TYPO3\CMS\Core\Http\AbstractApplication->handle() in /var/www/html/site/private/typo3/sysext/backend/Classes/Http/Application.php line 72

    $request = $request->withAttribute('applicationType', $applicationType);

    // Set up the initial context
    $this->initializeContext();
    return parent::handle($request);
}

/**
 * Create a PSR-7 Response that redirects to the install tool

at TYPO3\CMS\Backend\Http\Application->handle() in /var/www/html/site/private/typo3/sysext/core/Classes/Http/AbstractApplication.php line 100

 * @param callable $execute Deprecated, will be removed in TYPO3 v12.0
 */
final public function run(callable $execute = null)
{
    $response = $this->handle(ServerRequestFactory::fromGlobals());
    if ($execute !== null) {
        trigger_error('Custom execution of Application code will be removed in TYPO3 v12.0, use PSR-15 Middlewares instead.', E_USER_DEPRECATED);
        $execute();
    }

at TYPO3\CMS\Core\Http\AbstractApplication->run() in /var/www/html/site/public/typo3/index.php line 21

call_user_func(static function () { $classLoader = require dirname(dirname(DIR)).'/vendor/autoload.php'; chdir(getenv('TYPO3_PATH_ROOT')); \TYPO3\CMS\Core\Core\SystemEnvironmentBuilder::run(1, \TYPO3\CMS\Core\Core\SystemEnvironmentBuilder::REQUESTTYPE_BE); \TYPO3\CMS\Core\Core\Bootstrap::init($classLoader)->get(\TYPO3\CMS\Backend\Http\Application::class)->run(); });

at {closure}() in /var/www/html/site/public/typo3/index.php line 22

$classLoader = require dirname(dirname(__DIR__)).'/vendor/autoload.php';
chdir(getenv('TYPO3_PATH_ROOT'));
\TYPO3\CMS\Core\Core\SystemEnvironmentBuilder::run(1, \TYPO3\CMS\Core\Core\SystemEnvironmentBuilder::REQUESTTYPE_BE);
\TYPO3\CMS\Core\Core\Bootstrap::init($classLoader)->get(\TYPO3\CMS\Backend\Http\Application::class)->run();

});

[calien666]

Hi @metapublic-gbr. I found hopefully the leading issue via test case and fixed it. Would be nice, if you could test the pr. Feedback is welcome, if this works for you now or not. If not, I think, I need more information, which type of data you want to import and which backend user imports.

[metapublic-gbr]

Hi, sorry, I don't manage to pull the branch. in my central composer.json, I added:

"repositories": [
        {
            "type": "vcs",
            "url": "https://github.com/sudhaus7/typo3-xlsimport.git"
        }
    ],
"require": {
        "sudhaus7/xlsimport": "typo3-xlsimport-47"
    },

but I keep getting the error-message 'Invalid version string "typo3-xlsimport-47"' Any idea?

[calien666]

As packagist.org shows the branch as dev-typo3-xlsimport-47 try this as version string. With this it is not needed using the repository directly.

[metapublic-gbr]

that works. Thank you. Now, my small test, importing a new last_name into an existing tt_address-entry worked well. Also, importing categories worked - if I give the category-ids, not the category-titles.

[calien666]

Nice, that this worked. Then I will release a new version now. Other thing we will look up in a separate issue.