TeamINTERACT / IngestScripts

Jupyter Notebooks that govern the telemetry ingest process for Ethica and SenseDoc data
0 stars 0 forks source link

Set up the new permanent archive #1

Open RileyLazarou opened 1 year ago

RileyLazarou commented 1 year ago

It will be called data_archive.

data_archive/
    <city>/  # lowercase
        wave_<n:02d>/
            linkage.csv
            ethica/
                <study_id>/  # one dir per study (e.g. Montreal has 2)
                    accelerometer.csv
                    gps.csv
                    # other ethica files, such as battery.csv, wifi.csv, etc.
            sensedoc/
                <user_id>_<device_id>/  # one dir per user and device
                    SD<device_id>fw<REVNO><DATE><TIME>.sdb  # used by the pipeline
            output/  # This is where consumable output files will live (e.g. ToP csv's)
temp/  # temporary files go here 
code/
    IngestScripts/  # repo
    table_of_power/  # repo
RileyLazarou commented 1 year ago

Data archive and temp set up. Data archive still needs to be filled with data. Code still needs to have repos set up

RileyLazarou commented 1 year ago

Raw treksoft data cannot be stored here due to confidentiality reasons. Processed (anonymized) treksoft data may still be added to the output/ directories. Removing treksoft/ from the dataplan.