search

pfmc-assessments / PacFIN.Utilities

R code to manipulate data from the PacFIN database for assessments
http://pfmc-assessments.github.io/PacFIN.Utilities
Other
7 stars 1 forks source link

Comprehensive_BDS_Comm table's area column #32

Closed John-R-Wallace-NOAA closed 2 years ago

John-R-Wallace-NOAA commented 3 years ago

The Comprehensive Biological Data Sample (BDS) Commercial Table PDF (https://pacfin.psmfc.org/wp-content/uploads/2019/03/PacFIN_Comprehensive_BDS_Commercial.pdf) implies that the 'BDS Sample - State fishing area block number' and the 'BDS Sample - Agency PSMFC area code' are in separate columns. However, the FISHING_AREA_BLOCK_NUMBER column (at least for Canary rockfish) is all NA's and the PSMFC_CATCH_AREA_CODE column has both the block number and the PSMFC area code together.

I matched in the Comprehensive_FT's PACFIN_CATCH_AREA_CODE column into the BDS Canary data using FTID and created the following table:

                            1B    1C    2A    2B    2E    2F    3A    3B    3S    CL    UP    VN  <NA>
                             0     0     0     0     0     0     0     0     0     0     0     0  2796
   0100,1C                   0     0     0     0     0     0     0     0     0     0     0     0    12
   0108,1C                   0     0     0     0     0     0     0     0     0     0     0     0    24
   0109,1C                   0     0     0     0     0     0     0     0     0     0     0     0    61
   0200,1B                   0     0     0     0     0     0     0     0     0     0     0     0   136
   0200,1C                   0     0     0     0     0     0     0     0     0     0     0     0   564
   0213,1C,CC                0     0     0     0     0     0     0     0     0     0     0     0     1
   0244,1B                   0     0     0     0     0     0     0     0     0     0     0     0     5
   0245,1B                   0     0     0     0     0     0     0     0     0     0     0     0    30
   0250,1B                   0     0     0     0     0     0     0     0     0     0     0     0     6
   0252,1B                   0     0     0     0     0     0     0     0     0     0     0     0     2
   0256,1B                   0     0     0     0     0     0     0     0     0     0     0     0    37
   0257,1B                   0     0     0     0     0     0     0     0     0     0     0     0    33
   0264,1B                   0     0     0     0     0     0     0     0     0     0     0     0    35
   0270,1B                   0     0     0     0     0     0     0     0     0     0     0     0    52
   0270,1B,FB                0     0     0     0     0     0     0     0     0     0     0     0    16
   0275,1B                   0     0     0     0     0     0     0     0     0     0     0     0     3
   0400,1B                   0     0     0     0     0     0     0     0     0     0     0     0    32
   0403,1B                   0     0     0     0     0     0     0     0     0     0     0     0     4
   0404,1B                   0     0     0     0     0     0     0     0     0     0     0     0    57
   0405,1B,FB                0     0     0     0     0     0     0     0     0     0     0     0    40
   0410,1B                   0     0     0     0     0     0     0     0     0     0     0     0    24
   0416,1B                   0     0     0     0     0     0     0     0     0     0     0     0     5
   0418,1B,FB                0     0     0     0     0     0     0     0     0     0     0     0     3
   0425,1B                   0     0     0     0     0     0     0     0     0     0     0     0     7
   0426,1B,FB                0     0     0     0     0     0     0     0     0     0     0     0     2
   0432,1B,FB                0     0     0     0     0     0     0     0     0     0     0     0     8
   0435,1B                   0     0     0     0     0     0     0     0     0     0     0     0    57
   0440,1B                   0     0     0     0     0     0     0     0     0     0     0     0    29
   0443,1B                   0     0     0     0     0     0     0     0     0     0     0     0    30
   0447,1B                   0     0     0     0     0     0     0     0     0     0     0     0    51
   0451,1B                   0     0     0     0     0     0     0     0     0     0     0     0    21
   0453,1B                   0     0     0     0     0     0     0     0     0     0     0     0    53
   0457,1B                   0     0     0     0     0     0     0     0     0     0     0     0    41
   0464,1B                   0     0     0     0     0     0     0     0     0     0     0     0    86
   0466,1B                   0     0     0     0     0     0     0     0     0     0     0     0     5
   0471,1B                  27     0     0     0     0     0     0     0     0     0     0     0     2
   0474,1B                   0     0     0     0     0     0     0     0     0     0     0     0     3
   0485,1B                   0     0     0     0     0     0     0     0     0     0     0     0     2
   0500,1B                   0     0     0     0     0     0     0     0     0     0     0     0     5
   0501,1B                   0     0     0     0     0     0     0     0     0     0     0     0    11
   0514,1B                   0     0     0     0     0     0     0     0     0     0     0     0    86
   0518,1B                   0     0     0     0     0     0     0     0     0     0     0     0    14
   0526                      0     0     0     0     0     0     0     0     0     0     0     0     4
   0526,1B                   4     0     0     0     0     0     0     0     0     0     0     0    18
   0561,1A                   0     0     0     0     0     0     0     0     0     0     0     0     5
   0562,1A,MN                0     0     0     0     0     0     0     0     0     0     0     0     4
   0602,1A                   0     0     0     0     0     0     0     0     0     0     0     0     3
   0607,1A                   0     0     0     0     0     0     0     0     0     0     0     0     2
   0608,1A                   0     0     0     0     0     0     0     0     0     0     0     0     2
   0614,1A                   0     0     0     0     0     0     0     0     0     0     0     0     1
   0615,1A                   0     0     0     0     0     0     0     0     0     0     0     0     4
   0668,1A                   0     0     0     0     0     0     0     0     0     0     0     0     1
   0673,1A                   0     0     0     0     0     0     0     0     0     0     0     0     2
   0674,1A                   0     0     0     0     0     0     0     0     0     0     0     0     6
   0690,1A                   0     0     0     0     0     0     0     0     0     0     0     0     6
   0710,1A                   0     0     0     0     0     0     0     0     0     0     0     0    21
   0712,1A                   0     0     0     0     0     0     0     0     0     0     0     0    27
   0713,1A                   0     0     0     0     0     0     0     0     0     0     0     0     7
   0716,1A                   0     0     0     0     0     0     0     0     0     0     0     0     1
   0720,1A                   0     0     0     0     0     0     0     0     0     0     0     0     1
   0800,1A                   0     0     0     0     0     0     0     0     0     0     0     0     5
   0833,1A                   0     0     0     0     0     0     0     0     0     0     0     0     1
   0855,1A                   0     0     0     0     0     0     0     0     0     0     0     0     2
   0860,1A                   0     0     0     0     0     0     0     0     0     0     0     0    23
   0867,1A                   0     0     0     0     0     0     0     0     0     0     0     0     2
   1,60A2                    0     0     0     0     0     0     0     0     0     0     0     0     4
   1034,1A                   0     0     0     0     0     0     0     0     0     0     0     0     1
   1035,1A                   0     0     0     0     0     0     0     0     0     0     0     0     4
   1036,1A                   0     0     0     0     0     0     0     0     0     0     0     0    28
   1037,1B                   0     0     0     0     0     0     0     0     0     0     0     0    57
   1038,1B                  38     0     0     0     0     0     0     0     0     0     0     0   234
   1040,1B                  15     0     0     0     0     0     0     0     0     0     0     0   280
   1040,1C                   0     0     0     0     0     0     0     0     0     0     0     0    36
   1041                      0     0     0     0     0     0     0     0     0     0     0     0    18
   1041,1C                   0     0     0     0     0     0     0     0     0     0     0     0  1864
   1041,1C,EU                0     0     0     0     0     0     0     0     0     0     0     0     2
   1042,1C                   0    43     0     0     0     0     0     0     0     0     0     0   263
   1042,1C,CC                0     0     0     0     0     0     0     0     0     0     0     0     3
   1A                        0     0     0     0     0     0     0     0     0     0     0     0  1606
   1A,MN                     0     0     0     0     0     0     0     0     0     0     0     0     1
   1B                      327     0     6     0     0     0     0     0     0     0     0     0  7537
   1B,BB                     5     0     0     0     0     0     0     0     0     0     0     0     0
   1B,FB                     0     0     0     0     0     0     0     0     0     0     0     0    35
   1B,SF                     0     0     0     0     0     0     0     0     0     0     0     0     1
   1C                        0   730    26     0     0     0     0     0     0     0     0     0  6807
   1C,92                     0     0     0     0     0     0     0     0     0     0     0     0     6
   1C,EU                     0     0     0     0     0     0     0     0     0     0     0     0     1
   2A                        0    99   650    73     0     0     0     0     0     0     0     0  3426
   2B                        0     0   516   149     0    49     0     0     0     0    41     0 14760
   2C                        0     0     0     0     0     0     0     0     0     0     0     0   174
   2E                        0     0     0    25    94     0     0     0     0     0     0     0   449
   2F                        0     0    12   118   132   796   326    12     0     0   159     0  7419
   3A                        0     0     0     0   202    56   857   273     0     0    43     0 32695
   3B                        0     0     0     0    10     0   107   734   295     0     0     0 20989
   3C                        0     0     0     0     0     0    56   540   240     0     0   255 16059
   3C-S                      0     0     0     0     0     0     0     0     0     0     0     0   617
   3D                        0     0     0     0     0     0     0     0     0     0     0     0  5677
   3S                        0     0     0     0     0     0     0    27     0     0     0     0   960
   4A                        0     0     0     0     0     0     0     0     0     0     0    13     2
   58B,59A2,60A1,60A2,61     0     0     0     0     0     0     0     0     0     0     0     0    22
   58B,61                    0     0     0     0     0     0     0     0     0     0     0     0     3
   5A                        0     0     0     0     0     0     0     0     0     0     0     0  1412
   5B                        0     0     0     0     0     0     0     0     0     0     0     0  1726
   60A1                      0     0     0     0     0     0     0     0     0     8     0     0     0
   60A1,60A2                 0     0     0     0     0     0     0     0     0     0     0     0     1
   60A2,61                   0     0     0     0     0     0     0     0     0     0     0     0     1
   61                        0     0     0     0     0     0     0     0     0     0     0     0   216
   UNKN                      0     0     0     0     0     0     0     0     0     0   107     0   101

The table shows that the PSMFC_CATCH_AREA_CODE is on a finer scale than the FT's PACFIN_CATCH_AREA_CODE and confirms that, in general, they have consistent information. If PacFIN can not provide us with a PSFMC only and Dahl column in a reasonable time frame, it would be possible to get the PSFMC only column close by parsing PSMFC_CATCH_AREA_CODE and making a large re-coding table. The re-coding table is necessary since some blocks (e.g. 0526), do not have the PSMFC code included (e.g. '0518,1B')

FYI, not having a FTID (FALSE) in the BDS is more prominent in the early years:

         1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994
   FALSE 1502 1147  838 2034 1459  793 2667 2756 1083 1216 1444  981 2206 2242 1464 1841 2719 1899 2649 1702 1437 1032  902 2619 1744 2016  831
   TRUE     0    0    0    0    0    0    0    0   89  813  678  640 1299  865 1862 3285 2473 2420 2038 2159 1782 1691 1632 1266 3199 2618 2877

         1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
   FALSE 1254  899  888  199   35    0   25    0    0    1    0    9    2   18    0    0   61   14    0    0    1    0    0    0    0    0
   TRUE  2289 2921 3573 3215 3516 2066 1950 2217  626  956 1047 1134  938 1138 1278 1215 1006 1223 1882 1729 2950 2443 5648 4163 3792 1201
kellijohnson-NOAA commented 3 years ago

Thanks John for looking into this. Unfortunately, FTID does not map one to one with the catch data base and should not be used going further. If the information is not in the bds table then we should not be using it going forward.

John-R-Wallace-NOAA commented 3 years ago

I only used FTID to verify that the PSMFC area that is stuck inside the Comprehensive BDS's PSMFC_CATCH_AREA_CODE column is legitimate. For example, in the large table, if '0471,1B' matched with anything but '1B' or , then that information is clearly incorrect. The large table shows that the information in PSMFC_CATCH_AREA_CODE is consistent and hence extraction of the PSMFC area can be assumed to be correct, but perhaps incomplete. Splitting out the State fishing area block number from the PSFMC area into separate columns (as their pdf claims) may be the best PacFIN can do (and I can do the same). I doubt that the states have any more information then what is given in the PSMFC_CATCH_AREA_CODE column since that is all the area information that was recorded. Note also that the majority of data for this Canary rockfish example is already in just PSFMC areas (without any state fishing area block number). The numbers in the column count to that total, they just didn't get matched by FTID.

kellijohnson-NOAA commented 3 years ago

I am sorry, but I don't understand what you mean. What are you trying to do here? It seems like you are making things more complicated then they need to be.

John-R-Wallace-NOAA commented 3 years ago

I am extracting out the PSMFC areas from the PSMFC_CATCH_AREA_CODE column in the comprehensive BDS table. With a PSMFC only column, I can also get a INPFC column using the 'ar' (area) table. (I am creating a ARID column also.) I started yesterday and things are going well. I anticipate that the results will be more complete than PacFIN's, if they only do the equivalent of a separation of the PSMFC_CATCH_AREA_CODE column. Unless they (or all 3 states) have hidden tables that associates every state fishery block with the PSMFC and INFPC areas. I don't have such a table either, but it could be made if all the vertices for the state blocks could be obtained. Currently, when needed, I am stuck looking at maps that have state blocks on them, and hence the complication and time needed. I plan to create a Wiki so that there can be buy-in to the steps taken, and as a resource for where the state fishery block maps are on the internet.

kellijohnson-NOAA commented 3 years ago

John, this might be something that should be explored later when we aren't under the gun.

I do NOT think it is a good idea to make up new columns from PacFIN at this point in time. It makes it difficult to communicate with states or others that deal with PacFIN because they don't have access to the same data. More specifically, I believe that PacFIN is working really hard to get us what we need and if there is something that we NEED then we should be asking them to place it in one of the two tables that we extract. Otherwise how are they to know if they should update old tables. Rather than going against the grain here and thinking we can do better than PacFIN, I fully support putting more work on PacFIN and asking them to provide information to us.

John-R-Wallace-NOAA commented 3 years ago

I am not making up columns, this will be a comparison on what PacFIN has been asked to provide and what they will add to the comprehensive BDS. This is not against the grain at all, but will provide a check on what PacFIN provides us and will give us more confidence in what they come up with if that comparison is favorable.