nplhse / cois

Web application used for creating collaborative IVENA statistics.
https://versorgungsforschung.io
MIT License
0 stars 0 forks source link

Refactor everything related to Imports #1314

Open nplhse opened 1 year ago

nplhse commented 1 year ago

Is your feature request related to a problem? Please describe. The current solution for the Import is really complex and more complex than it needs to be with the ImportService, the ImportWriters and the ImportReaders... This needs to be more easy and understandable for potential users and participants.

Describe the solution you'd like At first there should be a ImportFactory that orchestrates how the Import is going to happen, at the moment a single AllocationImportFactory is enough. This factory manages the workflow via middlewares that the Factory controls and that accepts settings from the Factory.

The first middleware fetches the data and stores it as iterable associative array. It then passes the data along and the following middlewares create an entity, manipulate& validate it and finally sends the entity to the EntityManager to be flushed as batch. Errors that occur during this process result in a SkippedRow that gets also saved in the database for later review.

Required steps

  1. Fetch data from CSV
  2. Create Entity (Allocation or something else like MANV?) and add basic data like times& hospital
  3. FieldParser for boolean parameters
  4. Connect ValueObjects (needs one Middleware per ValueObject)
  5. Pass Entity to EntityManager and flush every X row
nplhse commented 1 year ago

To improve the speed of this process every row gets skipped if a ValueObject can`t be connected because a value is missing. In a special "run" of the Import all ValueObjects listen for new values and add them to the database. Afterwards within a new run of the Importer these rows can be properly connected.