ONYX-862: Data exportation should rollback...

[ymarcon]

Jira issue originally created by user smani:

When exporting data from Onyx, if there is an issue concerning a given participant data, all the exportation process is cancelled. It should be interesting that Onyx isolates problematic cases in a kind of log files while continuing data exportation for non problematic participants.

[ymarcon]

Comment created by nemond:

Yannick,

Peux-tu voir avec Serge. Est-ce faisable facilement. Je trouve ca requete legitime...

[ymarcon]

Comment created by @ymarcon:

There are in fact 2 problems:

1. there is no UI feedback whether the export was successful or failed.
2. there is no middle result: all participants were exported or none. This is due to the fact that the exported data are written on a stream.

Several improvements are possible:

1. Simple (1d):
   • add a progress and a report popup as it is already done for update appointments or purge data in 1.6.
   • do not change the backend (all or none are exported).
2. Complex (at least 5d):
   • introduce spring batch, to export participants by chunks. If for any reason a participant fails, only the concerned chunk fails, next chunks can be processed. This will generate one export zip file by chunk.
   • improve UI:
   • report chunk status in a popup,
   • and/or add an administration page, displaying the status of all spring batch jobs (allowing asynchronous job execution).

[ymarcon]

Comment created by nemond:

Yannick,

With Magma, do we have the same results (as the one you descripbe on September 23)?

[ymarcon]

Comment created by @ymarcon:

Nothing changed: error is only reported in the log, and the transaction covers the whole export process.

[ymarcon]

Comment created by nemond:

Serge,

Do you want to try it before doing something?

[ymarcon]

Comment created by nemond:

Philippe,

Qu'en penses-tu? Serge continue de dire que ca devrait seulement rollbacker le participant en probleme plutot que tout l'export.

[ymarcon]

Comment created by smani:

Nathalie,

It should interesting to implement the "complex" solution to be sure that data exportion works for interviews completed correctly. It should also be nice to send the log file to the DCC; in this cas, the log file should not contain any interview data.

[ymarcon]

Comment created by @plaflamme:

We could implement some improvements, but the fact remains: without some kind of layer on top, a file-system is not transactional.

If an operation fails during writing to the file-system (lack of space for example), then we have to implement our own transactional behaviour to detect the problem, rollback any modifications since the last "checkpoint" then commit.

To implement all of this correctly, we need to have distributed transactions because we are writing to the database (export log) and the file system at the same time. This involves integrating JTA into Onyx. Furthermore, the issue requires the ability to issue "checkpoints" which is not supported by all databases.

In conclusion, this is doable but is not trivial at all.

An improvement that is smaller is to output in blocks of participants. Each block of participant (say 15) would be persisted in its own file. If one block fails, others can still be exported. The downside of course is the fact that it would produce lots of files, but the export process would be more robust.