Review project organisation chapter (2.2)

AbdnCHDS / guidebook

Our guide for open and reproducible ways of working in ACHDS

Other

3 stars 5 forks source link

Review project organisation chapter (2.2) #27

Open dblana opened 3 years ago

dblana commented 3 years ago

Over in our collaborative project repo we thought that a better project structure might be to separate the input into raw and clean data. Currently we have an Input folder (that includes data, metadata and definitions file).

Should we just rename this "data_raw"? Is that name ok even if we can't share the data so we have a toy dataset?
Would the "data_clean" folder contain anything other than the clean data file?

Are there any other changes we want to make to the recommended folder structure?

will-ball commented 3 years ago

We briefly discussed the suggested folder structure yesterday (16/04/21) at the Data Lab meeting. Some points that were raised:

We should consult with DaSH and others with a view to aligning the guidebook suggestion with potential new DaSH folder structure.
The suggested executable compendium may not be fully appropriate given that 1) DaSH require a folder for releases (including non-data items) 2) the compendium model is focused on 'automated' reproducibility & may be overly prescriptive for wider use 3) it's a suggestion & people will do what they like, so maybe a simpler version is better?

The general principle of separating inputs, analysis & outputs is great. I think some of us agreed that separating the 'raw data' from working/cleaned data is also sensible.

dblana commented 3 years ago

Thanks Will! Are DaSH working on a new folder structure at the moment, is there a timeline for that? If it's not going to be very soon, should we implement the raw/clean data change for now, and revisit later to align with DaSH?

will-ball commented 3 years ago

I would be happy with this. @ClaraGlocke or @BScheliga do either of you have strong opinions on this?

ClaraGlocke commented 3 years ago

Hello, Sorry for some reason just became aware of this conversation thread. As discussed at the lab meeting, the DaSH Archiving group hope to look at folder structures soon, as in the next 2-3 weeks, and had hoped to do this with input from ACHDS. I am just out of a meeting with Gary Cooper. @dblana I believe he is going to contact you Dimitra, to see if the ACHDS lab group and DaSH Archive Group could come together to discuss this.

In the brief discussions in the Archive Group so far, thoughts run along the same lines as Will's, that is using the general principle of separating inputs, analysis & outputs, and the separation of 'raw data' from working/cleaned data.

ClaraGlocke commented 3 years ago

Queries from Folder Structure Working group to be returned to by participants:

[ ] Consider adding project number to header DaSH_PrimaryResearchData, i.e. 'DaSHXXX where XXX is project number.
[ ] Summary Statistics - This information is also in the Data Documentation Template. Do the statistics contain anything that is not in the template or should this folder be removed?
[ ] (DaSH) Methods - specify type of documentation rather than just using a generic catch all term 'documentation'.