search

ddionrails / steering

0 stars 0 forks source link

Import nach Änderungen benötigt zu viel manuelle Nacharbeit #27

Closed hansendx closed 3 years ago

hansendx commented 4 years ago
  1. Es gibt keinen standardisierten Prozess für die Vorbereitung der Daten.
  2. Die vorhandene "automatische" Vorverarbeitung der Daten ist veraltet und mittlerweile nicht mehr funktionsfähig.

Related: #3 #8 #9

issue-label-bot[bot] commented 4 years ago

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.67. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

kwenzig commented 4 years ago

@hansendx will analyse the current situation and make suggestions how to streamline the process.

hansendx commented 3 years ago

Data processing code pieces were refactored and collected in the paneldata_pipeline python package. The package provides a command line interface entry point. The old scripts needed manual changes to the code to control the program flow whereas the new entry point takes several arguments for this instead. Twinlife was the first study to be processed with the package. soep-core metadata still contains old structures and needs refactoring to work with the package.

hansendx commented 3 years ago

TODO:

hansendx commented 3 years ago

TODO?:

Relational test will take a lot of time. Goodtables might not be optimized enough for this. It could be necessary to implement this functionality instead.

hansendx commented 3 years ago

TODO?:

* [x]  Test relations in data compliance tests

Relational test will take a lot of time. Goodtables might not be optimized enough for this. It could be necessary to implement this functionality instead.

Relation control functionality was implemented as part of the paneldata_pipeline package: https://github.com/ddionrails/paneldata_pipeline