Reason (Why?)
We want to be able see and track what happens while the ETL process and afterwards (especially when something went wrong).
Solution (What?)
Add more extensive logging to all ETL steps (xml-import, elasticsearch-upload). The logging should be visible while execution (i.e. directly as shell output) and afterwards in a dedicated logging file. The log should include timestamps for all the steps and a summary at the end (e.g. listing how many entities where parsed (x artworks, y artists, ...) and the total time taken.
Acceptance criteria
The ETL process shows more extensive and detailed output while running and a log file is created containing the same output.
Reason (Why?) We want to be able see and track what happens while the ETL process and afterwards (especially when something went wrong).
Solution (What?) Add more extensive logging to all ETL steps (xml-import, elasticsearch-upload). The logging should be visible while execution (i.e. directly as shell output) and afterwards in a dedicated logging file. The log should include timestamps for all the steps and a summary at the end (e.g. listing how many entities where parsed (x artworks, y artists, ...) and the total time taken.
Acceptance criteria The ETL process shows more extensive and detailed output while running and a log file is created containing the same output.