DV-526 ⁃ Communicate DateTimeOriginal to DateTimeFinal Changes

[sync-by-unito[bot]]

﹍STEP ONE: INVESTIGATION:﹍

We need to determine which date/time variable is used within the various places of IDS that we use date/time. Places within IDS where the date/time is used is: (red = uses DateTimeOriginal, which up until this point has been incorrect)

1. 【 highlighted #ff5630 】Bulk Data Exporter【 end highlighted 】
   1. 【 highlighted #ff5630 】- ct_fltr_date ← A function to filter out unwanted dates within a metadata table【 end highlighted 】
2. Recognition
   1. (within the first tab, where we run the Reco algorithms. Here, we group images taken at the same place and the same time together to make a spatial-tempral match. (note- even though these groups are only finalized and shown to the user in the “assignment” tab (where names are assigned to the groups), I believe the initial grouping starts as early as the first tab). 【 highlighted #ff5630 】”【 end highlighted 】pttrn_rcgntn_srvr_fltrd_indvdl_tbl【 highlighted #ff5630 】” (A function to filter a Redshift table and pull it down) - Here we use “DateTimeOriginal“【 end highlighted 】 ”rfrnc_indvdls_srvr_dwnld_rfrnc_tbl” 【 highlighted #ff5630 】(A function to filter a individual reference table and pull it down) - Here we use “DateTimeOriginal“【 end highlighted 】

* ﹍Pattern Matching﹍

【 highlighted #36b37e 】- ”【 end highlighted 】dttm <- Sys.time()【 highlighted #36b37e 】” - Used when clicking “Run“ on in the Pattern Matching tab【 end highlighted 】 【 highlighted #36b37e 】- When joining the trigger and individual table we use “DatetimeInput“【 end highlighted 】 【 highlighted #36b37e 】- When matching temporally, we group by Station and CaptureEvent.【 end highlighted 】

* ﹍Validate﹍

【 highlighted #ff5630 】- 【 end highlighted 】rfrnc_indvdls_srvr_dwnld_rfrnc_tbl【 highlighted #ff5630 】- In this function we use to filter a individual reference table and pull it down. We use DateTimeOriginal.【 end highlighted 】 【 highlighted #ff5630 】-” 【 end highlighted 】blk_img_exprt_srvr_pll_fltr_tbl【 highlighted #ff5630 】" - A function to filter a Redshift table and pull it down. ← DateTimeOriginal【 end highlighted 】 【 highlighted #ff5630 】- 【 end highlighted 】pttrn_rcgntn_asss_indpndnc_fll【 highlighted #ff5630 】- A function to compute independent captures, based on a user defined threshold. Returns a full data set. ← This function uses DateTimeOriginal【 end highlighted 】

* 【 highlighted #36b37e 】﹍Assignment﹍【 end highlighted 】

* 【 highlighted #36b37e 】﹍Merge﹍【 end highlighted 】

1. 【 highlighted #ff5630 】Descriptive Statistics “Daily Photographic Detections - Grouped Classes” plot【 end highlighted 】
   1. 【 highlighted #ff5630 】(the dates of captures are shown)【 end highlighted 】
2. 【 highlighted #ff5630 】Descriptive Statistics “Species accumulation curve (x-axis = number of occasions)” plot【 end highlighted 】
   1. 【 highlighted #ff5630 】(the number of total/cumulative species captures is shown over “occasions”, aka days of the survey)【 end highlighted 】
3. 【 highlighted #ff5630 】Descriptive Statistics “Activity Patterns | Top 10 Species Detections” plot & “Activity Patterns | User-defined Species” plot & “Activity Patterns | Degree of Overlap” plot【 end highlighted 】
   1. 【 highlighted #ff5630 】(the photographic activity per species is plotted throughout a 24 hour day, in order to see when we are capturing the species most, so we can assume when they are most active in the day)【 end highlighted 】
4. 【 highlighted #36b37e 】Descriptive Statistics “Trap Effort by Occasion” plot -----> uses “Date” attribute from CamAct【 end highlighted 】
   1. 【 highlighted #36b37e 】(the number of active cameras per station (and total) over “occasions”, aka days of the survey)【 end highlighted 】
5. 【 highlighted #ff5630 】Intelligent Queries Result Pivot Table【 end highlighted 】
   1. 【 highlighted #ff5630 】“DateTime” attribute and “Date” attribute are potentially wrong【 end highlighted 】
6. 【 highlighted #ff5630 】Occupancy Single Season, Single Species【 end highlighted 】

• ”ct_occpncy_anlyss_gnrt_dtctn_rcrd_cstm_bns"【 highlighted #ff5630 】- A function to generate a detection record for occupancy analysis, using custom bins. We use DateTimeOriginal here as well.【 end highlighted 】 【 highlighted #ff5630 】- “【 end highlighted 】ct_anlyss_asss_indpndnc【 highlighted #ff5630 】“ - A function to assess temporal independence within camera stations【 end highlighted 】
• “ct_occpncy_anlyss_gnrt_dtctn_rcrd" 【 highlighted #ff5630 】- A function to generate a detection records for occupancy analyses.【 end highlighted 】
• “ct_occpncy_anlyss_gnrt_dtctn_rcrd_cstm_bns_mltpl_sts“ -【 highlighted #ff5630 】A function to generate a detection record for multiples survey sites,for Occupancy analysis using custom bins 【 end highlighted 】
• “ct_occpncy_anlyss_gnrt_stan_dtctns_dt“ 【 highlighted #ff5630 】- Generate detection history input for STAN occupancy model【 end highlighted 】

1. 【 highlighted #ff5630 】Occupancy Single Season, Multi Species【 end highlighted 】

• ct_occpncy_anlyss_gnrt_dtctn_tmln_dt【 highlighted #ff5630 】- A function to generate detection timeline【 end highlighted 】

1. 【 highlighted #ff5630 】Occupancy Multi Season, Single Species【 end highlighted 】

• “ct_occpncy_mlt_ssn_fltr_date“ 【 highlighted #ff5630 】- A function to filter data within primary periods or multi-season occupancy modelling.【 end highlighted 】

1. Occupancy Export Detections
2. 【 highlighted #ff5630 】SCR- Generate Inputs【 end highlighted 】

• “ct_gnrt_dh"【 highlighted #ff5630 】- A function to convert station coordinates to UTM, and generate a detection history. Returns a capthist secr object【 end highlighted 】 【 highlighted #ff5630 】-” 【 end highlighted 】ct_secr_anlyss_mnl_srvr_tbl【 highlighted #ff5630 】" - A function to filter a Redshift table and pull it down for SCR analysis【 end highlighted 】

﹍STEP TWO: COMMUNICATE TO USERS:﹍

This needs to be communicated to the users VERY WELL, to alert them of any potential issues within their analyses using server-based data. Also let them know that we’ll be adding the DateTimeFinal column and using that instead.

﹍STEP THREE: FIX EXISTING SERVER DATA:﹍

In addition to the fix being applied in https://pantheracorporation.atlassian.net/browse/DV-523 to stop the data from being converted to the different time zone, we will also be manually creating and filling a new DateTimeFinal attribute for all server-based data on Redshift. All the details for this work are now included in the https://pantheracorporation.atlassian.net/browse/DV-523 task.

﹍STEP FOUR: COMMUNICATE TO USERS AGAIN ONCE FIX IS APPLIED:﹍

This needs to be communicated to the users VERY WELL, to let them know that their data has been changed.

┆Issue is synchronized with this Jira Task by Unito

[sync-by-unito[bot]]

➤ Shannon Dubay commented:

Isaiah Lekay please have a read over this task. Next week, we’ll need to complete at least step 1 and 2. I will need your assistance with step 1, since it will require poking through the code; I will lead Step 2.

[sync-by-unito[bot]]

➤ Shannon Dubay commented:

Divide and Conquer

Isaiah and Valentine - 2 Shannon - 3-7 Isaiah - 8-12

[sync-by-unito[bot]]

➤ Isaiah Lekay commented:

Shannon Dubay Still busy with this have a few things in Reco to double check, then will move on to the last few in Occupancy. But will have this task completed by the end of day today.

[sync-by-unito[bot]]

➤ Isaiah Lekay commented:

Shannon Dubay Valentine Tawira Checked the final functions for Occupancy and SCR, attached the functions in the description which uses DateTimeOriginal. I know it was mostly functions that dealt with the server databases, but I still added them from all the modules.

[sync-by-unito[bot]]

➤ Shannon Dubay commented:

Awesome work Isaiah Lekay to finish out step 1 of this task.

Step One- complete.

I am prepping the email communication for Step Two.

Then after that goes out, we can work on Step Three with Valentine Tawira and Thabied Majal and change the existing Redshift DateTimeOriginal data using the code they prepared. THIS WILL BE FOR NEXT WEEK’S SPRINT.

[sync-by-unito[bot]]

➤ Shannon Dubay commented:

Step 2 of this task complete.

Step 3 is beginning this week in https://pantheracorporation.atlassian.net/browse/DV-523 ( https://pantheracorporation.atlassian.net/browse/DV-523|smart-link )

[sync-by-unito[bot]]

➤ Shannon Dubay commented:

Step 3 was completed over the last 2 weeks.

Step 4 was completed by Isaiah Lekay on Thurs 19 Oct.

Marking this task as complete- finally 🙂