Missing TOTAL_WGT - Githubissues

pfmc-assessments / PacFIN.Utilities

R code to manipulate data from the PacFIN database for assessments

http://pfmc-assessments.github.io/PacFIN.Utilities

Other

7 stars 2 forks source link

Missing TOTAL_WGT #46

Closed chantelwetzel-noaa closed 3 years ago

chantelwetzel-noaa commented 3 years ago

The TOTAL_WGT is missing in the updated bds pull (or either not being converted via cleanColumns). The column is used in the EF1_Numerator function for data expansion.

kellijohnson-NOAA commented 3 years ago

I checked and TOTAL_WGT is not included in the convert list, so it should be available upon download. Is this is a California-specific column. I wonder if we can match it from an old data pull that has TOTAL_WGT to see if Pacfin just added it to a different column?

chantelwetzel-noaa commented 3 years ago

Looking at and older bds file all states have some values in this column:

table(!is.na(data$TOTAL_WGT), data$SAMPLE_AGENCY)

               CA        OR      W

FALSE 92 13587 571 TRUE 108068 106506 55586

Looking in the numerator for the expansions it does seem to be the key quantity for California expansions. I have not been able to find an column in the new pull that looks equivalent.

kellijohnson-NOAA commented 3 years ago

According to the PacFIN documentation I think it should be WEIGHT_OF_LANDING_LBS

kellijohnson-NOAA commented 3 years ago

Looks like PacFIN just collapsed the two columns together, which is what I use to do in EF1_Numerator. Definitions provided by PacFIN old document

TOTAL_WGT == "Total landed weight (in pounds) for just that species portion of the landed catch."
EXP_WT == "Weight (pounds) the sample will be expanded to (default is the landing weight). This value may be larger or smaller than TOTAL_WGT depending on various situations documented in the text file ~pacfin/doc/bds_odfw_expanded_weight.txt.

new document

WEIGHT_OF_LANDINGS_LBS == "Total landed weight (in pounds) for just that species portion of the landed catch"
EXPANDED_SAMPLE_WEIGHT == "Oregon only. Weight the sample will be expanded to (default is the landing weight)."

chantelwetzel-noaa commented 3 years ago

Thanks. Based on this information I think issue #47 is related to this one as well. I think this just may require us to ensure old column names are being converted to the matching new column names.

The cleanColumns function is currently converting columns as follows:

EXPANDED_SAMPLE_WEIGHT == EXP_WT
ROUND_WEIGHT_LBS == CATCH.LBS
WEIGHT_OF_LANDING_LBS == lwt_lbs
WEIGHT_OF_LANDING_LBS == RWT_LBS

but should the correct matching be:

EXPANDED_SAMPLE_WEIGHT == EXP_WT
ROUND_WEIGHT_LBS == RWT_LBS
LANDED_WEIGHT_LBS == LWT_LBS
WEIGHT_OF_LANDING_LBS == TOTAL_WGT

Does this look correct?

kellijohnson-NOAA commented 3 years ago

Thank you @chantelwetzel-noaa for all of your work helping figure out the column name mess that I got myself into. I think this one is fixed as well. If yes, feel free to close.