Documentation for ETL5

[Hobbesball]

Detailed description of the issue. Since our ETLs will run periodically on a remote server, we need the capability to monitor their performance and catch any issues that might occur in them. Detailed logging information is necessary for the VKC to maintain the ETLs after they have been delivered by Kitania.

Additional context The main needed goals of logging are to:

• give insights into the volume and performance of the ETLs for monitoring and reporting
• raise errors and warnings that give a clear cause and context so that they can be resolved easily by VKC staff

Possible implementation

• We recommend using monolog's implementation in Symfony , monolog-bundle. You can find documentation on how to initialise it here.
• Logs should be stored in a logfile on the server, rotating out every two months
• Logs should raise errors and warnings in every important step of the ETL process, and give the ID of the objects that failed.
• Logs should count the amount of objects taken as input, how many were successfully put through the ETL, and how many were output.

[Hobbesball]

Adding to the need for logging, documentation for ETL5 should also include:

• a detailed description of how the ETL works, how to configure it, and how to use it in the README
• sufficient in-code comments that break down the process and allow for easy debugging