achetverikov / visual_search_db

Visual Search Database
3 stars 2 forks source link

Flexible structured data import #2

Open achetverikov opened 7 years ago

achetverikov commented 7 years ago

What we need is organized data import. Right now, for each experiment (or series of experiments), there is a separate import script. Ideally, this needs to be split into data preparation and data import stages. During data preparation stage, some kind of structured output will be created. Right now it seems to me that separating data in accordance with experiment - subject - block - trial - stimuli logic seems to be valid. Maybe we don't need separate files for each node type, but on the other hand, if we do it all the same, why not? In addition to the files themselves, there should be a translation between csv columns and DB properties.

During data import pre-prepared files should be imported in DB. This should work for every possible import as long as it is correctly prepared.

achetverikov commented 7 years ago

now there's a config for each experiment that describes the mapping of csv fields to DB