Easy export of data - Githubissues

open-cogsci / omm-server

Open Monkey Mind server

Apache License 2.0

0 stars 1 forks source link

Easy export of data #166

Open kristian-lange opened 3 months ago

kristian-lange commented 1 month ago

Here is a repo with a Python script that can be a template for an database exporter. https://github.com/kristian-lange/omm_db_exporter

I'm not sure though whether the exported data are enough. They only contain the JSON data from the job_results table.

kristian-lange commented 1 week ago

Updated to work with the new database schema

kristian-lange commented 1 week ago

@nclaidiere @smathot Please have a look at project https://github.com/kristian-lange/omm_db_exporter.

This simple Python script can be extended to your needs, e.g. take the database's host, username, pw as arguments or use other IDs than the study ID to query the database.

So far it takes a study ID as an argument and writes a file in CSV format with the job result data. It can deal with inconsistencies in the job variables, e.g. if new variables were added during running a study.

Is this okay? Any wishes?

Edit: The CSV contains additionally to the job result data and all job variables the following fields:

participant_name
participant_identifier
participant_meta
job_id
job_position
study_id
study_name

smathot commented 6 days ago

This looks ok to me. It will be a bit difficult for @nclaidiere to have an opinion before he can actually use it to export data, so let's pick this up when we've migrated the system to the new server. But in general terms, this is indeed what we need.

nclaidiere commented 6 days ago

I agree, difficult to know but this looks good. Two questions though:

the goal was to be able to download data from a date range, a list of study IDs or a list of individuals or a combination of those, is that easily implemented here?
we agree there will still be the download data button on the server UI to download the results of a specific experiment directly right?

kristian-lange commented 2 days ago

Hi @nclaidiere and @smathot!

I addressed the missing points and added:

support for lists of study IDs
support for date ranges
command argument parsing

$ ./export.py -h
usage: export.py [-h] -i IDS [-f FROM] [-t TO]

Exports job result data into a file in CSV format

options:
  -h, --help            show this help message and exit
  -i IDS, --ids IDS     list of study IDs, comma-separated
  -f FROM, --from FROM  'from' date in format YYYY-MM-DD
  -t TO, --to TO        'to' date in format YYYY-MM-DD

Example: ./export.py --ids=1,2 --from=2024-11-01 --to=2024-11-03

smathot commented 2 days ago

Nice touch with the command-line options!