Closed sync-by-unito[bot] closed 10 months ago
➤ Shannon Dubay commented:
Isaiah Lekay please have a read over this task. Next week, we’ll need to complete at least step 1 and 2. I will need your assistance with step 1, since it will require poking through the code; I will lead Step 2.
➤ Shannon Dubay commented:
Divide and Conquer
Isaiah and Valentine - 2 Shannon - 3-7 Isaiah - 8-12
➤ Isaiah Lekay commented:
Shannon Dubay Still busy with this have a few things in Reco to double check, then will move on to the last few in Occupancy. But will have this task completed by the end of day today.
➤ Isaiah Lekay commented:
Shannon Dubay Valentine Tawira Checked the final functions for Occupancy and SCR, attached the functions in the description which uses DateTimeOriginal. I know it was mostly functions that dealt with the server databases, but I still added them from all the modules.
➤ Shannon Dubay commented:
Awesome work Isaiah Lekay to finish out step 1 of this task.
Step One- complete.
I am prepping the email communication for Step Two.
Then after that goes out, we can work on Step Three with Valentine Tawira and Thabied Majal and change the existing Redshift DateTimeOriginal data using the code they prepared. THIS WILL BE FOR NEXT WEEK’S SPRINT.
➤ Shannon Dubay commented:
Step 2 of this task complete.
Step 3 is beginning this week in https://pantheracorporation.atlassian.net/browse/DV-523 ( https://pantheracorporation.atlassian.net/browse/DV-523|smart-link )
➤ Shannon Dubay commented:
Step 3 was completed over the last 2 weeks.
Step 4 was completed by Isaiah Lekay on Thurs 19 Oct.
Marking this task as complete- finally 🙂
﹍STEP ONE: INVESTIGATION:﹍
We need to determine which date/time variable is used within the various places of IDS that we use date/time. Places within IDS where the date/time is used is: (red = uses DateTimeOriginal, which up until this point has been incorrect)
pttrn_rcgntn_srvr_fltrd_indvdl_tbl
【 highlighted #ff5630 】” (A function to filter a Redshift table and pull it down) - Here we use “DateTimeOriginal“【 end highlighted 】 ”rfrnc_indvdls_srvr_dwnld_rfrnc_tbl
” 【 highlighted #ff5630 】(A function to filter a individual reference table and pull it down) - Here we use “DateTimeOriginal“【 end highlighted 】* ﹍Pattern Matching﹍
【 highlighted #36b37e 】- ”【 end highlighted 】
dttm <- Sys.time()
【 highlighted #36b37e 】” - Used when clicking “Run“ on in the Pattern Matching tab【 end highlighted 】 【 highlighted #36b37e 】- When joining the trigger and individual table we use “DatetimeInput“【 end highlighted 】 【 highlighted #36b37e 】- When matching temporally, we group by Station and CaptureEvent.【 end highlighted 】* ﹍Validate﹍
【 highlighted #ff5630 】- 【 end highlighted 】
rfrnc_indvdls_srvr_dwnld_rfrnc_tbl
【 highlighted #ff5630 】- In this function we use to filter a individual reference table and pull it down. We use DateTimeOriginal.【 end highlighted 】 【 highlighted #ff5630 】-” 【 end highlighted 】blk_img_exprt_srvr_pll_fltr_tbl
【 highlighted #ff5630 】" - A function to filter a Redshift table and pull it down. ← DateTimeOriginal【 end highlighted 】 【 highlighted #ff5630 】- 【 end highlighted 】pttrn_rcgntn_asss_indpndnc_fll
【 highlighted #ff5630 】- A function to compute independent captures, based on a user defined threshold. Returns a full data set. ← This function uses DateTimeOriginal【 end highlighted 】* 【 highlighted #36b37e 】﹍Assignment﹍【 end highlighted 】
* 【 highlighted #36b37e 】﹍Merge﹍【 end highlighted 】
ct_occpncy_anlyss_gnrt_dtctn_rcrd_cstm_bns
"【 highlighted #ff5630 】- A function to generate a detection record for occupancy analysis, using custom bins. We use DateTimeOriginal here as well.【 end highlighted 】 【 highlighted #ff5630 】- “【 end highlighted 】ct_anlyss_asss_indpndnc
【 highlighted #ff5630 】“ - A function to assess temporal independence within camera stations【 end highlighted 】ct_occpncy_anlyss_gnrt_dtctn_rcrd
" 【 highlighted #ff5630 】- A function to generate a detection records for occupancy analyses.【 end highlighted 】ct_occpncy_anlyss_gnrt_dtctn_rcrd_cstm_bns_mltpl_sts
“ -【 highlighted #ff5630 】A function to generate a detection record for multiples survey sites,for Occupancy analysis using custom bins 【 end highlighted 】ct_occpncy_anlyss_gnrt_stan_dtctns_dt
“ 【 highlighted #ff5630 】- Generate detection history input for STAN occupancy model【 end highlighted 】ct_occpncy_anlyss_gnrt_dtctn_tmln_dt
【 highlighted #ff5630 】- A function to generate detection timeline【 end highlighted 】ct_occpncy_mlt_ssn_fltr_date
“ 【 highlighted #ff5630 】- A function to filter data within primary periods or multi-season occupancy modelling.【 end highlighted 】“ct_gnrt_dh
"【 highlighted #ff5630 】- A function to convert station coordinates to UTM, and generate a detection history. Returns a capthist secr object【 end highlighted 】 【 highlighted #ff5630 】-” 【 end highlighted 】ct_secr_anlyss_mnl_srvr_tbl
【 highlighted #ff5630 】" - A function to filter a Redshift table and pull it down for SCR analysis【 end highlighted 】﹍STEP TWO: COMMUNICATE TO USERS:﹍
This needs to be communicated to the users VERY WELL, to alert them of any potential issues within their analyses using server-based data. Also let them know that we’ll be adding the DateTimeFinal column and using that instead.
﹍STEP THREE: FIX EXISTING SERVER DATA:﹍
In addition to the fix being applied in https://pantheracorporation.atlassian.net/browse/DV-523 to stop the data from being converted to the different time zone, we will also be manually creating and filling a new DateTimeFinal attribute for all server-based data on Redshift. All the details for this work are now included in the https://pantheracorporation.atlassian.net/browse/DV-523 task.
﹍STEP FOUR: COMMUNICATE TO USERS AGAIN ONCE FIX IS APPLIED:﹍
This needs to be communicated to the users VERY WELL, to let them know that their data has been changed.
┆Issue is synchronized with this Jira Task by Unito