Is there a way to directly persist data into the target table instead of using an intermediate import table?

[Saami783]

Hello,

I would like to know if there is a possibility for us, the users of the bundle, to change the way imports are managed into the database? Typically, instead of creating an import table with all the columns corresponding to the target table in addition to information related to the file and the state of the import, and then inserting the data there for the target table to later retrieve the saved data from the import table, I would like to persist the data directly into the target table and leave the import table to record important import information (path, error, number of lines, etc.). This approach is simpler and allows me to have a better overview of all imports performed on each table.

Thank you in advance for your response.

Sami

[thislg]

Hello,

We use a two-step import for several reasons:

• You can add custom validation via an event subscriber after data is loaded in the temporary table;
• Any issue with initial load would leave the temporary table in a broken state which is not something you want on the final target table;
• Load uses COPY FROM (https://www.postgresql.org/docs/current/sql-copy.html) which is very fast but doesn't allow setting any conflict target.

Error recording is not in the scope of this bundle: you can use an event subscriber for that (to send a custom email or log in a table for example). Basic error information is already logged with a monolog Logger as well as other import steps, and there is an option to send an import report with technical details.