stratum mislabled in AI gird shapefile

[EmilyMarkowitz-NOAA]

Issue

A few stratum seem to be mislabeled in the AI gird shapefile currently included in akgfmaps. I had encountered this issue last year when I was putting together the survey progression/temperature maps and Mark Z was able to provide me with a new shapefile where these were corrected.

Suggestion

Replace this shapefile with this shapefile.

Notes

Differences between the shapefile in akgfmaps and survey-live-temperature-map As a short hand, "r_" is grid data provided from survey-live-temperature-map and "a" is grid data provided from akgfmaps

When I plot bounding boxes around the corresponding INFPC regions (ID'ed by stratum via goa.goa_strata), you can see that the stratum are in the issue (this is how I noticed the error in the shapefile).

Using the AI Grid shapefile from akgfmaps:

Using the shapefiles in survey-live-temperature-map's shapefile

[sean-rohan-NOAA]

Ok. Is this the official shapefile for the GOA/AI group that is used to produce stock assessment products? Or is this a realignment that Mark did that hasn't been adopted by the group as the official stratum boundary file?

[EmilyMarkowitz-NOAA]

I think the latter. Good point, probably worth checking with @Ned-Laman-NOAA.

[sean-rohan-NOAA]

@Ned-Laman-NOAA Is the shapefile Emily shared the official shapefile for the AI?

[EmilyMarkowitz-NOAA]

I'll add that I double checked that everything else (columns, observations, data dimensions, etc) were otherwise exactly the same between the shapefiles. I took a moment to recalculate the areas and perimeters of each grid from scratch (a great sanity-check suggestion form @sean-rohan-NOAA) and they are the same between the files. I feel confident that, between these two shapefiles, the only difference is in the specification of those few stratum.

[sean-rohan-NOAA]

@Ned-Laman-NOAA @EmilyMarkowitz-NOAA Was there a final decision on this? Noticed it's been a month and a half since this issue was opened.

[EmilyMarkowitz-NOAA]

No updates from me, as I am just the anticipated user. @Ned-Laman-NOAA were you able to talk with people on your end?

[Ned-Laman-NOAA]

Remind me (again) of the origin of the shapefile with the obviously funky strata? E.g., is it U:\alaska\constructs\zones\strata\aigrid? U:\alaska\constructs\zones\strata\ai_strata? Or does it originate from one of the Oracle tables on the AI schema?

[sean-rohan-NOAA]

The AI stratum shapefile is U:\alaska\constructs\zones\strata\ai_strata, which I used based on your recommendation in an email on May 24, 2022.

On Tue, Apr 18, 2023 at 1:26 PM Ned Laman @.***> wrote:

Remind me (again) of the origin of the shapefile with the obviously funky strata? E.g., is it U:\alaska\constructs\zones\strata\aigrid? U:\alaska\constructs\zones\strata\ai_strata? Or does it originate from one of the Oracle tables on the AI schema?

— Reply to this email directly, view it on GitHub https://github.com/afsc-gap-products/akgfmaps/issues/30#issuecomment-1513754649, or unsubscribe https://github.com/notifications/unsubscribe-auth/APULYIL647HLZPPLDUJIEI3XB32IPANCNFSM6AAAAAAVOADPAU . You are receiving this because you were mentioned.Message ID: @.***>

-- Sean K. Rohan, PhD (he/him) Research Fish Biologist Bering Sea Bottom Trawl Survey Group Resource Assessment and Conservation Engineering Division NOAA Alaska Fisheries Science Center 7600 Sand Point Way NE Seattle, WA 98115 @.***

[Ned-Laman-NOAA]

In that recommendation, I cited both ai_strata & aigrid as the authorities for this information. I'm now realizing that those two shapefiles aren't even in synch. The ai_strata shapefile does not include Bowers Ridge as the aigrid shapefile does. I believe that some of the stratum discrepancies are on Bowers Ridge but we haven't sampled out there on an Aleutian survey for decades. Let's first agree on whether or not to include Bowers Ridge in your navmaps package to start narrowing down where we need to address these issues.

[sean-rohan-NOAA]

My suggestion is that Bowers Ridge shouldn't be included because the strata out there aren't included in index calculations and aren't in the standard survey. Based on that logic, they could also be dropped from the survey grid.

As an aside, Bowers Ridge also shouldn't be included in the VAST AI extrapolation grid but was in the grid the last time I checked. Maybe it's changed.

On Tue, Apr 18, 2023 at 4:26 PM Ned Laman @.***> wrote:

In that recommendation, I cited both ai_strata & aigrid as the authorities for this information. I'm now realizing that those two shapefiles aren't even in synch. The ai_strata shapefile does not include Bowers Ridge as the aigrid shapefile does. I believe that some of the stratum discrepancies are on Bowers Ridge but we haven't sampled out there on an Aleutian survey for decades. Let's first agree on whether or not to include Bowers Ridge in your navmaps package to start narrowing down where we need to address these issues.

— Reply to this email directly, view it on GitHub https://github.com/afsc-gap-products/akgfmaps/issues/30#issuecomment-1513907159, or unsubscribe https://github.com/notifications/unsubscribe-auth/APULYIICJRJRF3U2QRKC5TLXB4PKHANCNFSM6AAAAAAVOADPAU . You are receiving this because you were mentioned.Message ID: @.***>

-- Sean K. Rohan, PhD (he/him) Research Fish Biologist Bering Sea Bottom Trawl Survey Group Resource Assessment and Conservation Engineering Division NOAA Alaska Fisheries Science Center 7600 Sand Point Way NE Seattle, WA 98115 @.***

[Ned-Laman-NOAA]

There is a copy of the aigrid shapefile with Bowers Ridge clipped out here (G:\AI-GOA\shapefiles\aigrid_clipped)

[sean-rohan-NOAA]

Ok. So the station grid with Bowers Ridge clipped out (G:\AI-GOA\shapefiles\aigrid_clipped) is the official survey grid shapefile for the AI?

On Tue, Apr 18, 2023 at 4:51 PM Ned Laman @.***> wrote:

There is a copy of the aigrid shapefile with Bowers Ridge clipped out here (G:\AI-GOA\shapefiles\aigrid_clipped)

— Reply to this email directly, view it on GitHub https://github.com/afsc-gap-products/akgfmaps/issues/30#issuecomment-1513920382, or unsubscribe https://github.com/notifications/unsubscribe-auth/APULYIPI4HVAI5WEIBCDBYLXB4SGZANCNFSM6AAAAAAVOADPAU . You are receiving this because you were mentioned.Message ID: @.***>

-- Sean K. Rohan, PhD (he/him) Research Fish Biologist Bering Sea Bottom Trawl Survey Group Resource Assessment and Conservation Engineering Division NOAA Alaska Fisheries Science Center 7600 Sand Point Way NE Seattle, WA 98115 @.***

[Ned-Laman-NOAA]

The shapefile here (G:\AI-GOA\shapefiles\aigrid_clipped) is literally the shapefile from here (U:\alaska\constructs\zones\strata\aigrid) with Bowers Ridge clipped out. The "aigird_clipped" shapefile referenced here should be considered the official Aleutian survey shapefile for akgfmaps.

[sean-rohan-NOAA]

Hi Ned,

It looks like the aigrid_clipped shapefile you reference has grid cells with strata 793 and 794 that are west of the boundaries for strata 793 and 794:

Therefore, I still think it would make sense to adopt Em's suggestion to change stratum designations for grid cells in the shapefile, as described earlier in this thread:

A few stratum seem to be mislabeled in the AI gird shapefile currently included in akgfmaps. I had encountered this issue last year when I was putting together the survey progression/temperature maps and Mark Z was able to provide me with a new shapefile where these were corrected.

[Ned-Laman-NOAA]

Sean,

It sounds like you're right and we should be using the corrected and clipped shapefile Emily references.

Here is where my hesitancy lies. Before we commit to making the change, we need to have written documentation detailing how the corrected stratum labels were assigned and applied. Then, there is a whole system of shapefiles, Oracle tables, and public-facing databases that will need to receive these changes to Aleutian stratum areas. Before that happens, there nees to be a coordinated communication campaign to inform data users about the changes. Clear documentation of how and why these changes are being made will form the basis for communication and rollout of the changes.

In short, I don't think we can simply swap shapefiles in akgfmaps and call it good without doing our diligence among all of our sources of data and data users. In fact, I think all of the other changes need to be in place and clearly communicated before your package distributes the new map.

Thoughts?

[EmilyMarkowitz-NOAA]

Would it be helpful to add a page documenting this change to an akgfmaps pkgdown page or to the gap_products quarto book? @sean-rohan-NOAA I could help create the former (small lift, though you might already have experience) and/or latter (create a page/section for you to edit) if there is interest. Here is what a pkgdown page for the gapindex package looks like.

[Ned-Laman-NOAA]

It is my sense that the akgfmaps package is rapidly becoming the face of our spatial information so I think it would be helpful to add this type of documentation alongside the package. I remain hesitant to try to work backwards from the package to our internal resources when trying to disseminate these changes because the rate of change will be much much slower for the latter.

[sean-rohan-NOAA]

My first impression is that we don't need a coordinated campaign to communicate with end users about changes to shapefiles beyond what is already being done because that isn't consistent with how folks have been using or obtaining shapefiles. Until recent times, many (most?) folks have been using their own bespoke versions of shapefiles or making changes to shapefiles during their data wrangling to handle errors with shapefiles.

Bespoke shapefiles have been used in an operational capacity. There is at least one case where an 'official' shapefile with errors has been used for both research and production (to generate the AI extrapolation grid that has been the default in VAST-- includes Bowers Ridge). There were discrepancies between survey areas for design and model-based indices of abundance. There is a another case where stratum areas were likely incorrectly transcribed to racebase so there is a discrepancy between stratum areas that can be calculated from official shapefiles and official stratum areas in racebase.

In the last couple of years, folks have been switching to using the akgfmaps package. I think that's because it simplifies their R-based workflows and ensures that they are using the same shapefile as others. A core principle for the package is that the information in the shapefiles should match the information in racebase.

Shapefiles in akgfmaps are used to produce the following:

• Survey data reports
• Presentations
• Survey charts
• NRT temperature maps
• Stratum area (EBS/NBS)
• Extrapolation grids for VAST (EBS/NBS)
• Novel research products

One benefit of the package is that a user can determine which version of a shapefile they're using based on which version the akgfmaps package they're using. To communicate about changes, new releases of akgfmaps include change notes with versioned releases and changes are also logged in the NEWS file.

Currently, the AI survey grid shapefile is often not being used because it is not reliable since it has stratum errors. Consequently, package users are creating their own bespoke version of the shapefile to handle the data errors, which I suspect leads to inconsistencies.

On Mon, Oct 2, 2023 at 2:51 PM Ned Laman @.***> wrote:

Sean,

It sounds like you're right and we should be using the corrected and clipped shapefile Emily references.

Here is where my hesitancy lies. Before we commit to making the change, we need to have written documentation detailing how the corrected stratum labels were assigned and applied. Then, there is a whole system of shapefiles, Oracle tables, and public-facing databases that will need to receive these changes to Aleutian stratum areas. Before that happens, there nees to be a coordinated communication campaign to inform data users about the changes. Clear documentation of how and why these changes are being made will form the basis for communication and rollout of the changes.

In short, I don't think we can simply swap shapefiles in akgfmaps and call it good without doing our diligence among all of our sources of data and data users. In fact, I think all of the other changes need to be in place and clearly communicated before your package distributes the new map.

Thoughts?

— Reply to this email directly, view it on GitHub https://github.com/afsc-gap-products/akgfmaps/issues/30#issuecomment-1743820370, or unsubscribe https://github.com/notifications/unsubscribe-auth/APULYIMYJ4MHAAENM6WW7XTX5MZMDAVCNFSM6AAAAAAVOADPAWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONBTHAZDAMZXGA . You are receiving this because you were mentioned.Message ID: @.***>

-- Sean K. Rohan, PhD (he/him) Research Fish Biologist Bering Sea Bottom Trawl Survey Group Resource Assessment and Conservation Engineering Division NOAA Alaska Fisheries Science Center 7600 Sand Point Way NE Seattle, WA 98115 @.***

[sean-rohan-NOAA]

My preferred approach to removing Bowers Ridge from the standard AI grid would be to keep all of the cells in the shapefile itself and filter out Bowers Ridge cells when folks retrieve AI layers using get_base_layers(). Would still need to correct erroneous stratum 793 and 794 cells.

On Mon, Oct 2, 2023 at 3:23 PM Ned Laman @.***> wrote:

It is my sense that the akgfmaps package is rapidly becoming the face of our spatial information so I think it would be helpful to add this type of documentation alongside the package. I remain hesitant to try to work backwards from the package to our internal resources when trying to disseminate these changes because the rate of change will be much much slower for the latter.

— Reply to this email directly, view it on GitHub https://github.com/afsc-gap-products/akgfmaps/issues/30#issuecomment-1743852763, or unsubscribe https://github.com/notifications/unsubscribe-auth/APULYIKBQ7OJXL2IAOIWNNTX5M5D5AVCNFSM6AAAAAAVOADPAWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONBTHA2TENZWGM . You are receiving this because you were mentioned.Message ID: @.***>

-- Sean K. Rohan, PhD (he/him) Research Fish Biologist Bering Sea Bottom Trawl Survey Group Resource Assessment and Conservation Engineering Division NOAA Alaska Fisheries Science Center 7600 Sand Point Way NE Seattle, WA 98115 @.***

[Ned-Laman-NOAA]

@Sean - take a look at my attempt to stratum correct the aigrid shapefile here G:\AI-GOA\shapefiles\aigrid_stratum_corrected. I haven't had a chance to do the same from ai_strata but will get there if this attempt looks like an improvment.

[sean-rohan-NOAA]

Reviewing the shapefile the strata look correct for the AI. However, it looks like there are a couple of new fields in the shapefile named Field1 and Field2.

• Field1 appears to be the same as AIGRID_.
• Field2 contains 1017 'Y's and 56 'N's and 9324 '[NULL]'s.

Can Field1 and Field2 be omitted from the shapefile?

On Thu, Oct 5, 2023 at 4:38 PM Ned Laman @.***> wrote:

@sean https://github.com/sean - take a look at my attempt to stratum correct the aigrid shapefile here G:\AI-GOA\shapefiles\aigrid_stratum_corrected. I haven't had a chance to do the same from ai_strata but will get there if this attempt looks like an improvment.

— Reply to this email directly, view it on GitHub https://github.com/afsc-gap-products/akgfmaps/issues/30#issuecomment-1749796575, or unsubscribe https://github.com/notifications/unsubscribe-auth/APULYIK74EKIQXRZFINXV5TX55AGVAVCNFSM6AAAAAAVOADPAWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONBZG44TMNJXGU . You are receiving this because you were mentioned.Message ID: @.***>

-- Sean K. Rohan, PhD (he/him) Research Fish Biologist Bering Sea Bottom Trawl Survey Group Resource Assessment and Conservation Engineering Division NOAA Alaska Fisheries Science Center 7600 Sand Point Way NE Seattle, WA 98115 @.***

[Ned-Laman-NOAA]

Field1 and Field2 can be omitted from the shapefile. Field 2 is the T/UT designation which we update after each Aleutian survey and so would not normally be part of the standard shapefile delineating the survey area.

[sean-rohan-NOAA]

Alright, I removed Field1 and Field2 from the shapefile and pushed to the slope branch. Now just waiting for tests to finish.