search

inbo / fish-tracking

🐟 Collection of scripts for processing and analysing fish tracking data
3 stars 0 forks source link

Change data flow for tracking data #2

Open peterdesmet opened 10 years ago

peterdesmet commented 10 years ago

Current data flow

  1. Tracking data are collected by receivers
  2. Data are periodically retrieved from receivers in the field
  3. Data are consolidated in VEMCO software
  4. Data are exported to csv
  5. Data are imported in Access
  6. Data are manually corrected (e.g. station names)
  7. Data are exported to R for analysis

    New data flow

  8. Idem
  9. Idem
  10. Idem
  11. Idem
  12. A script will parse the data and:
    • Update incorrect station names based on a mapping.
    • Match the data with the metadata on tracker_id, receiver_id and timestamp.
    • Aggregate the data in time windows.
    • Identify ghost trackers
    • Assign coordinates
  13. Data are imported in a relational database on a server.
  14. Data can be exported from the database to R for analysis and for data publication.

This process will run and overwrite all data initially.

peterdesmet commented 10 years ago

@ammouton, can you check if the current workflow is described correctly and if the new workflow is what we discussed?

ammouton commented 10 years ago

looks ok