Open ymarcon opened 14 years ago
Comment created by nemond:
Yannick,
Peux-tu voir avec Serge. Est-ce faisable facilement. Je trouve ca requete legitime...
Comment created by @ymarcon:
There are in fact 2 problems:
Several improvements are possible:
Comment created by nemond:
Yannick,
With Magma, do we have the same results (as the one you descripbe on September 23)?
Comment created by @ymarcon:
Nothing changed: error is only reported in the log, and the transaction covers the whole export process.
Comment created by nemond:
Serge,
Do you want to try it before doing something?
Comment created by nemond:
Philippe,
Qu'en penses-tu? Serge continue de dire que ca devrait seulement rollbacker le participant en probleme plutot que tout l'export.
Comment created by smani:
Nathalie,
It should interesting to implement the "complex" solution to be sure that data exportion works for interviews completed correctly. It should also be nice to send the log file to the DCC; in this cas, the log file should not contain any interview data.
Comment created by @plaflamme:
We could implement some improvements, but the fact remains: without some kind of layer on top, a file-system is not transactional.
If an operation fails during writing to the file-system (lack of space for example), then we have to implement our own transactional behaviour to detect the problem, rollback any modifications since the last "checkpoint" then commit.
To implement all of this correctly, we need to have distributed transactions because we are writing to the database (export log) and the file system at the same time. This involves integrating JTA into Onyx. Furthermore, the issue requires the ability to issue "checkpoints" which is not supported by all databases.
In conclusion, this is doable but is not trivial at all.
An improvement that is smaller is to output in blocks of participants. Each block of participant (say 15) would be persisted in its own file. If one block fails, others can still be exported. The downside of course is the fact that it would produce lots of files, but the export process would be more robust.
Jira issue originally created by user smani:
When exporting data from Onyx, if there is an issue concerning a given participant data, all the exportation process is cancelled. It should be interesting that Onyx isolates problematic cases in a kind of log files while continuing data exportation for non problematic participants.