CaitlinPries
commented
5 years ago Jeff,
I see some issues with what you are proposing.
1,3: First, if you want to make incubationInfo only for data that is consistent across the entire study, then why would you move depth and experimental treatment to the incubationInfo section as those are variables that are not always consistent across the entire study?
Second, I don’t see what the issue is with redundancy between the variables section and incubation info section. I was thinking by filling it out, people would better know which subheadings they need to include with each variable. Maybe if we had a separate section that was called treatmentInfo that gave arrays for all variables being tested in the study that are not site related? Or see experimentalTreatment thought below.
2: I agree that constant variables should be unnested.
experimentalTreatment may still have to have multiple responses if multiple factors are being tested. So it may still have to be an array, such as fieldWarming and depth, with the levels for each factor specified in variables.
We want to keep the yaml file, so turning siteInfo to a table does not seem like a good option to me. We already have two table (excel) files. I don’t think we want more.
So overall, if we need to make a change, I think a new category of all the treatment factors would be useful. But I also don not see an issue with then including the levels of each of those as a sub array so that a quick look of the data shows all the treatments. Of course, I say this without having played with the data as much as you have, Jeff.
Best, Caitlin
Caitlin Hicks Pries Assistant Professor of Biological Sciences Dartmouth College Life Sciences Center, room 349 78 College St. Hanover, NH 03755
603-646-2052 http://sites.dartmouth.edu/hicksprieslab/ @Carbon_Cait on Twitter
From: Jeff B notifications@github.com Reply-To: SoilBGC-Datashare/sidb reply@reply.github.com Date: Tuesday, June 25, 2019 at 1:04 PM To: SoilBGC-Datashare/sidb sidb@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: [SoilBGC-Datashare/sidb] Possible template update (#2)
The current version of our template has multiple structural and syntactic inconsistencies, which makes both data analysis and entry challenging.
As an example of structural inconsistency: the variables "temperature" and "moisture" can be found at three different possible levels within the database, with varying degrees of duplication.
As an example of syntactic inconsistency, the variable "landCover" can also be entered as an experimental treatment if it varies across sites, and when entered at that level of the database it may have a unique name that would prevents querying the database for this variable effectively.
I propose the following changes:
- Adopt a rule for the data in the incubationInfo section that data is ONLY entered here if it is consistent across the entire study. Variables that vary (e.g. laboratory treatments) are entered at the variable level of the database (i.e. for each individual timeseries) already, so it is unnecessary and confusing to enter the data here.
- Unnest any constant variables that are currently nested in the sublist "laboratoryTreatment" within the incubationInfo section. Defining laboratoryTreatment as a separate list is unnecessarily complicated and leads to structural inconsistency.
- Reform the siteInfo section so all data has to have the same shape as the site array, i.e. move depth, permafrost, and experimental treatment info to the incubationInfo section.
- Simplify experimentalTreatment to a single categorical response. Timeseries from studies with multiple treatment levels specify the treatment level explicitly in the variables section, and with only a simple categorical response it will be easier to group similar studies, e.g. "Harvard Forest experimental warming" and "fieldWarming" could both be called "fieldWarming".
- If the above changes are adopted, consider renaming incubationInfo experimentInfo
- Consider either reformatting the siteInfo section or changing it to a tabular form for ease of data entry.
I spent a couple days trying to do some simple reports and queries, and found it required a great deal of data manipulation---not at all user friendly. Hence the proposed changes.
Updating the templates isn't difficult, and I have already done so with an offline set of templates as a proof of concept, and written some query/report scripts.
However, if we do decide to adopt these changes it will affect the code that has already been written, as well as require some minor changes to the description of the template in the manuscript, so I realize it's not an easy decision.
Perhaps it's just me, so If you haven't done so, I would encourage you to try doing some data analysis across all studies with the current version of the database and see what you think.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSoilBGC-Datashare%2Fsidb%2Fissues%2F2%3Femail_source%3Dnotifications%26email_token%3DABXVALV5L6HKNCEQIHKO4RLP4JF2RA5CNFSM4H3KM74KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G3TITZA&data=02%7C01%7Ccaitlin.pries%40dartmouth.edu%7Cb6e05ed4af1a40490cff08d6f98f36b7%7C995b093648d640e5a31ebf689ec9446f%7C0%7C0%7C636970790828905745&sdata=W9UXBvNh2vlxNeqQoM5uGC3TukK8qpjOrx7Z1la%2BNEQ%3D&reserved=0, or mute the threadhttps://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABXVALXHQIGFSQ7U7IZHMFLP4JF2RANCNFSM4H3KM74A&data=02%7C01%7Ccaitlin.pries%40dartmouth.edu%7Cb6e05ed4af1a40490cff08d6f98f36b7%7C995b093648d640e5a31ebf689ec9446f%7C0%7C0%7C636970790828905745&sdata=Mgjm45V6RzHB4vLoSUtvXFYP4E0%2BnaTXqBtBJjant8s%3D&reserved=0.
jb388
aahoyt
ShaneStoner
The current version of our template has multiple structural and syntactic inconsistencies, which makes both data analysis and entry challenging.
As an example of structural inconsistency: the variables "temperature" and "moisture" can be found at three different possible levels within the database, with varying degrees of duplication.
As an example of syntactic inconsistency, the variable "landCover" can also be entered as an experimental treatment if it varies across sites, and when entered at that level of the database it may have a unique name that would prevents querying the database for this variable effectively.
I propose the following changes: 1) Adopt a rule for the data in the incubationInfo section that data is ONLY entered here if it is consistent across the entire study. Variables that vary (e.g. laboratory treatments) are entered at the variable level of the database (i.e. for each individual timeseries) already, so it is unnecessary and confusing to enter the data here. 2) Unnest any constant variables that are currently nested in the sublist "laboratoryTreatment" within the incubationInfo section. Defining laboratoryTreatment as a separate list is unnecessarily complicated and leads to structural inconsistency. 3) Reform the siteInfo section so all data has to have the same shape as the site array, i.e. move depth, permafrost, and experimental treatment info to the incubationInfo section. 4) Simplify experimentalTreatment to a single categorical response. Timeseries from studies with multiple treatment levels specify the treatment level explicitly in the variables section, and with only a simple categorical response it will be easier to group similar studies, e.g. "Harvard Forest experimental warming" and "fieldWarming" could both be called "fieldWarming". 5) If the above changes are adopted, consider renaming incubationInfo experimentInfo 6) Consider either reformatting the siteInfo section or changing it to a tabular form for ease of data entry.
I spent a couple days trying to do some simple reports and queries, and found it required a great deal of data manipulation---not at all user friendly. Hence the proposed changes.
Updating the templates isn't difficult, and I have already done so with an offline set of templates as a proof of concept, and written some query/report scripts.
However, if we do decide to adopt these changes it will affect the code that has already been written, as well as require some minor changes to the description of the template in the manuscript, so I realize it's not an easy decision.
Perhaps it's just me, so If you haven't done so, I would encourage you to try doing some data analysis across all studies with the current version of the database and see what you think.