reconcile differences between census_data_for_cored_trees.csv and chronology_list.csv

[teixeirak]

@ValentineHerr , @RHelcoski,

census_data_for_cored_trees.csv has disagreements over n_cores with chronology_list.csv (the master/ basis for tables 2 and S1 in paper) for 4 species. I believe census_data_for_cored_trees.csv missed recent update(s)?

[RHelcoski]

Yes, I believe so. Those were all the species I had to make the last minute changes on before I left.

[RHelcoski]

Yes, census_data_for_cored_trees.csv missed recent updates. I just checked the raw COFECHA files and re-ran the 4 in question. chronology_list.csv is up to date, census_data_for_cored_trees.csv needs to be updated exactly as the comments state. I don't know where the file is though...

[ValentineHerr]

@RHelcoski, does than mean that there is updated .rwl files that Neil needs to have to re-run his part?

[teixeirak]

I believe Neil has run all the latest files.

[ValentineHerr]

OK.

I re-ran all of my scripts and I still have the discrepancy... My script only uses the measurements files (measurement_notes_2010_chronology.csv and measurement_notes_2016_17_chronology.csv) to build census_data_for_cored_trees.csv and the map. Can the problem come from there?

[teixeirak]

Yes, that’s the problem— that file does not incorporate the final adjustments to chronologies. We need to go off the final chronology .rwl files. The code could read from those or we could update the measurement notes.

[ValentineHerr]

I believe I need the measurement notes to know what trees were cored live or dead.

[ValentineHerr]

I am investigating this issue and I found the following so far:

• CACO: Still investigating why there is a difference

• CAGL: The rwl file in raw_data was not the same as in SCBI-ForestGEO-Data_private/tree_cores --> I updated the rwl file in raw_data. It is not a big deal because the rwl files in Dropbox or data folder were the same as in SCBI-ForestGEO-Data_private/tree_cores so Neil ran his part on the good version.

• CAOVL: Still investigating why there is a difference

  • CATO: I noticed the rwl files in climate_sensitivity_core and in Dropbox (used by Neil) are different than the rwl file in SCBI-ForestGEO-Data_private/tree_cores . They have one more tree (Tree 90552) --> @RHelcoski and @mcgregorian1, do you know which version is the good one ? (with or without Tree 90552). If SCBI-ForestGEO-Data_private/tree_cores is the good on, I need to update the file in Dropbox and Neil has to re-run his part for that species.

  • QURU: The rwl file in data folder and in Dropbox (both the same and used by Neil) are different than the rwl file in SCBI-ForestGEO-Data_private/tree_cores . The rwl files in Drobox and data folder both have a duplicated tree (121026 ) --> Need to update the file in Dropbox and Neil has to re-run his part for that species.

IMPORTANT: wait until I am done with this to ask Neil to re-run his part.

[ValentineHerr]

For CAOVL and CACO the problem comes from tree 111190. It is in the CAOVL chronologies and a CAOVL in the measurement notes but it is a CACO in the census data... I don't see any notes about wrong sp ID in the field notes of new census data so I think I'll re-ID that tree as CACO and move its core measurements from caovl_drop.rwl to caco_drop.rwl and ask Neil to re-run his part for those 2 species. Is that ok with everybody, @RHelcoski , @mcgregorian1 and @teixeirak ? @RHelcoski , @mcgregorian1, please answer question in comment above too. Thanks!

[teixeirak]

@ValentineHerr, thanks for working on this. Please wait until @RHelcoski has a chance to review and respond. Some of this sounds like issues we've dealt with already.

[RHelcoski]

Hello @ValentineHerr , sorry it took me a bit to get back to this.

CACO and CAOVL: You're correct that tree 111190 should be in CACO not in CAOVL. I fixed it in \climate sensitivity\results\z_FinalChronologies. This does mean unfortunetly that both need to be rerun.

CAGL - looks good then

CATO - the good one is with tree 90552, meaning Neil ran the good file.

QURU - you're correct that tree 121026 is duplicated. I updated it in the z_FinalChronologies folder along with CACO and CAOVL.

I've also rerun all the COFECHA files so I will have to update the other tables as well

[teixeirak]

Okay, let's go ahead and send these to Neil. @RHelcoski and @ValentineHerr, could you please take care of that and updating the results/figures/tables? @mcgregorian1, I assume we'll also need to update a few of the chronologies that you sent to Neil. We should let him know to wait on the updated versions before running yours. (Please move forward on this without any further input from me during the shutdown.)

[ValentineHerr]

1. @mcgregorian1, can you update me on what you are sending to Neil? We need to ask him to run CACO, CAOVL and QURU using:

   • Dropbox \climate sensitivity\results\z_FinalChronologies\CACO\caco_drop.rwl
   • Dropbox \climate sensitivity\results\z_FinalChronologies\CAOV\caovl_drop.rwl
   • Dropbox \climate sensitivity\results\z_FinalChronologies\QURU\quru_drop.rwl

   Do you ask him to run those files? If yes, I guess it makes sense to have you only communicate with him. But please let me know if you don't need those files or if it makes more sense for me to work with him directly on that.

2. I am replacing SCBI-ForestGEO-Data_private\tree_cores\chronologies\current_chronologies\complete\cato_drop.rwl with my version of the file that includes 90552 now.

3. once 1. is solved I'll re-run the whole project and we should be good to go... hopefully!

[ValentineHerr]

@RHelcoski, Sorry to bug you again. I realize now that since you said that the version with 90552 was the good one for CATO, maybe I made a mistake when I considered cagl_drop.rwl in SCBI-ForestGEO-Data_private/tree_cores as the good version for CAGL. Should CAGL include trees "30567B" "92183B" "172210B" ?

[RHelcoski]

No there should be no B trees in Cagl

El lun., dic. 31, 2018 2:32 PM, Valentine Herrmann notifications@github.com escribió:

@RHelcoski https://github.com/RHelcoski, Sorry to bug you again. I realize now that since you said that the version with 90552 was the good one for CATO, maybe I made a mistake when I considered cagl_drop.rwl in SCBI-ForestGEO-Data_private/tree_cores

https://github.com/SCBI-ForestGEO/SCBI-ForestGEO-Data_private/tree/master/tree_cores/chronologies/current_chronologies/complete as the good version for CAGL. Should CAGL include trees "30567B" "92183B" "172210B" ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SCBI-ForestGEO/climate_sensitivity_cores/issues/52#issuecomment-450688097, or mute the thread https://github.com/notifications/unsubscribe-auth/Aat8Iy5uPy_x-yGFI7AjLOo4oH0sWyQyks5u-nRNgaJpZM4ZYtd5 .

[ValentineHerr]

Ok thanks! I fixed CAGL then. Now we are just waiting on @mcgregorian1 to see if it is necessary for us to ask Neil to run CACO, CAOVL and QURU or if he will do it himself. If I don't hear from @mcgregorian1 in the next couple of days I'll ask Neil myself.

[teixeirak]

This is a task for Neil. @mcgregorian1 does not (yet?) know how to do this. This will affect @mcgregorian1's chronologies (sent to Neil in December). Please note this in your correspondence with him. Better yet, you could fix the chronologies that @mcgregorian1 created, which are here and send fixed files for those that changed to Neil.

[ValentineHerr]

About chronologies created by @mcgregorian1 to separate trees by canopy and subcanpy trees:

• QURU: It looks like file here is correct (does not include twice 121026) --> no need to ask Neil to re-run that.
• CACO and CAOVL: I can't find tree 111190 in any of the caco or caovl files, canopy nor subcanopy...

Since I could not find tree 111190 I ran a script to find if there were more missing trees between files here and canopy and subcanpy folders and bellow are the missing trees I found. @teixeirak and @mcgregorian1, is it normal that those trees are missing? Maybe the canopy position was not known for them? @mcgregorian1, I see the script you wrote to separate files here between live and dead but I don't see the script you used to separate by canopy and subcanopy. Maybe you can push that when you get a chance. That way I can double check what is going on.

missing trees in canopy or subcanopy for:
- CACO: "162252A" "160989A" "142210A" "52059A"  "111190A"
- CAGL:  "70065"    "70332"    "70626A"   "91391"    "80288"    "92183A"   "101159r"  "172218Ar" "33337A"  
- CAOVL: "190929A" "111081r" "60358r" 
- CATO: "182279A" "142538A" "152185A"
- FRAM: "121455A" "192641"  "12267"   "152659"  "102453A" "192592A" "41006A"  "162458r"
- FRNI: "30142A"  "90718A"  "102001A" "12221A" 
- LITU: "50474"   "72212A"  "92442A"  "042565A" "151051A" "111033"  "70469A"  "90607A"  "112421A" "140820A" "82239A"  "82381A"  "110902r"
- PIST:  "122128"
- QUAL:  "80371"   "162397"  "201332"  "161573A" "132177A" "50517r"  "20835r"  "122474r"
- QUPR: "202135"  "32513A"  "72361A"  "182056A" "192083"  "50434"   "80437"   "22600r"  "80465r" 
- QURU: "121310" "50506" 
- QUVE: "80056"  "80156"  "90577"  "130911" "132386" "150957" "82059r"

[mcgregorian1]

I'll look over things. However, I'm having some trouble consolidating dropbox github with github desktop connecting to everything on this computer, so I'm looking at everything via the web client

[mcgregorian1]

Original comment: 7 Jan. Ok after reading through everything, I see the issue.

First off, sorry about all this beforehand. I should've checked email last week. I'll try explaining this in as clear a way I can.

1. The initial issue Krista noticed with the different numbers in census_data_for_cored_trees may partly have been from an edit that I made before I sent the cores to Neil on 10 Dec. After I realized with @RHelcoski that some of the B cores were included in the "complete" chronologies, I removed those so only A cores were in the "complete" subset. Then, I had noticed the numbers were off, and after messaging Ryan about it, I changed them accordingly (seen here, I believe). This seems to have been fixed tho, based on what I've read here.

For my workflow otherwise, this is what I did:

2. First, I knew I'd be sending a list to Neil of the cores separated by canopy positions, for a possible paper for my own analysis. To do this, I needed to take AJ's data and manipulate it.

3. I made a script called dbh_crownposition_by_sp that subsets canopy positions from the dendroband data based on AJ's surveys and which cores in the measurement_notes files were labeled as "complete" (along the way troubleshooting for trees that couldn't be or weren't given canopy positions).

4. After I made a list of trees for my analysis with this script (Sections 1 and 2 in the code) and creating some of the graphs @teixeirak has seen.

5. Then, I made a list of cores that I needed to send to Neil. The script for this is Section 5 in the code in Step 3 above, and the file is called "core_list_for_Neil".

6. I then made another script using "core_list_for_Neil" that created the different rwl files by canopy position. This is called core_list_by_canopy_subcanopy (@ValentineHerr if you think this is one-too-many scripts, I can easily put it in my original script from Step 3. I think I made it separate because I didn't want the original to be too confusing).

7. The files made from that script in Step 6 above are what I sent to Neil. They are currently located in the canopy and subcanopy folders.

8. After that, I put the cores in the Dropbox folder (Dropbox (Smithsonian)\Tree Cores\cores_by_canopy_position), and sent Neil the link on 10 December. I have not received a response from him since then.

   • as far as I know, I don't think I need to contact Neil with updated cores, as I made the list of cores for him based on the complete cores from the beginning. Does this seem correct?

Remaining missing trees

9. Core 111190 was not labeled as "included" in measurement_notes_2010-2011, therefore I did not include it in my lists.

Update 8 Jan:

10. dendro_cored_full was missing AJ's data. I discovered that's my fault and it's been fixed, along with having AJ's raw data uploaded as well.

11. I altered the code standardize_rwl_to_csv to only include cores from the "complete" folder. The reason you were finding the discrepancies, @ValentineHerr, is because when I first made the file, I was using all the cores in the "processed" folder, which included more than we needed. I've uploaded the new all_core_chronologies.

12. Also, yes, I don't know how to run the cores. We asked Neil if he wanted to teach me to do it, but he mentioned it was fine by him to run them.

[ValentineHerr]

• On 01/21/2019: I provided Neil with 3 new .rwl files for CACO, CAOVL and QURU in Dropbox\climate sensitivity\results\NEW CHRONOLOGIES TO RUN BY NEIL. New files include 2 fixes: tree 111190 is correctly in CACO instead of CAOVL and tree 121026 is not duplicated anymore in QURU.

• On 01/25/2019:

  • Neil got back to me. He had detrended the chronologies and put the results in Dropbox\climate sensitivity\results\NEW CHRONOLOGIES TO RUN BY NEIL
  • I took the results and moved them over to Dropbox\climate sensitivity\results\z_FinalChronologies so that all good chronologies are in the same spot now. (I have saved a version of the previous results on my dropbox just in case)
  • [ ] TO DO by @ValentineHerr : delete Dropbox\climate sensitivity\results\NEW CHRONOLOGIES TO RUN BY NEIL when all of this has been "approved" by others
  • [ ] NOTE to @RHelcoski and @mcgregorian1, the COFETCHA outputs were not in what Neil gave. Is that something you guys do? I don't think I need it for the analysis but all other species have the COFETCHA_OUT file so for consistency it might be good to add that file into the Dropbox...
  • I copied Neils' output over to climate sensitivity analysis data folder
  • [ ] TO DO by @ValentineHerr delete the raw_data/core folder from climate sensitivity analysis repository (we don't need it anymore). And find a solution for climate data so that we can remove raw_data all together.
  • I copied corrected caco.rwl and caovl.rwl over to SCBI-ForestGEO-Data_private relevant forlder (QURU was already correct there), @RHelcoski and @mcgregorian1 see this related issue.

Future steps:

• [ ] (after shutdown ends) for @ValentineHerr: run the whole analysis again. Note for myself: I have updated some script on my tablet...

[RHelcoski]

I used the data from here to run the COFECH files. COFECHA is basically just a quick way to summarize the chronologies for any dendrochronologists snooping around.

note that this file doesn't appear to have the newest QURU rwl files yet

[ValentineHerr]

@RHelcoski,

• Where did you put the COFETCHA outputs?
• the QURU file here was already correct (tree 121026 appeared only once there). That file only needed to be updated in Dropbox for Neil to use, and in the climate sensitivity analysis repository.

[ValentineHerr]

@RHelcoski or @mcgregorian1, I understand that line 620 of this file should now read:

Tag	chronology	chronology_status	chronology_notes
111190	CACO	NA

instead of:

Tag	chronology	chronology_status	chronology_notes
111190	CAOVL	NA	core not measured

correct?? Let me know if you update the file. Otherwise I'll do it myself.

[mcgregorian1]

Yep should be caco.

Since it was originally labeled as "core not measured," I didn't include this in my list of cores I sent to Neil.

[ValentineHerr]

I updated the measurement file in this commit

[teixeirak]

Is this issue resolved, then?

[ValentineHerr]

Almost. I am still waiting on @RHelcoski to tell me where he put the COFETCHA outputs.

[RHelcoski]

They are all in the folder

\climate sensitivity\results\z_FinalChronologies

[RHelcoski]

all COFECHA files in that folder are up to date

[ValentineHerr]

Thanks @RHelcoski !

I moved COFTECHA ouput of CACO, CAOVL and QURU here.

@teixeirak, let's close this issue when we are sure that census_data_for_cored_trees.csv matchs updated table 2 ans S1.

[teixeirak]

Okay. @RHelcoski, could you please update those tables?

[RHelcoski]

Yes, table 2 and table S1 are both impacted by this so I will update them both

[teixeirak]

table 2 is updated; closing this.

[teixeirak]

Note: all results in manuscript are now updated.