v9 update and questions

[mcmelnychuk]

Nicole found 26 stocks that were previously not on our list. Some have been first assessed in the last couple years, some are state-based assessments that we didn't thoroughly cover before, and some we appear to have previously missed for other reasons. I've updated or added the following files to the Dropbox folder:

v9 dataset - years of first HCR and assessment.xlsx v9 dataset.csv

• to these new files, I added 26 assessed stocks. They should replace "v8..." in the code. No new fields, only added rows.

SpeciesCrossReference.csv:

• added some entries in the "exclude" column. Filled in some more species under existing categories, and also added some exclusion categories: mammals, corals, frogs, shads.

database linkage - landings management.xlsx

• this file contains the two linkage tables for state landings & stock names. Added the 26 assessed stocks and also made some updates throughout.

crossref.csv crosref.csv

• updated these files based on the updated stock-to-state landings linkage table

landings-price.csv

• updated this file based on the state landings-to-stock linkage table

[mcmelnychuk]

I also have some questions:

1) Filtering out of species: beyond the "exclude" column in "SpeciesCrossReference.csv", are there filters in place to exclude some species? For example, are the landed species linked to FishBase habitat categories, and are freshwater fish then excluded? I added some "freshwater" values to the "exclude" column, but probably didn't get them all.

2) Should I send more information for the newly-added stocks? In the two "v9" files, I've only added their year of first assessment, region, Latin names, and SIS stock name. Is any other information read in from these files (e.g. taxonomic, habitat, year of fishery development)? If so I can fill in the missing info.

3) I'm not sure if we need some manual tweaking to accommodate two species that link to more than one entity under AFS_NAME:

• blue crab: 4 names: CRAB, BLUE; CRAB, BLUE, PEELER; CRAB, BLUE, SOFT; CRAB, BLUE, SOFT AND PEELER
• spiny dogfish: 2 names: SHARK, SPINY DOGFISH; SHARK, DOGFISH This may cause mismatches with linkages involving "AFS_NAME" values. Blue crab was on the list before and I think I remember it getting sorted out, but spiny dogfish is a new one with this issue. (Let me know if I can clarify.)

4) Should we update the NMFS landings & price database, or go with the current one (last year 2013)? I'm not sure if that's important for accommodating the species that were first assessed since then. There is also one assessed species without landings data (BSAI Kamchatka flounder) because it didn't split off into a single species assessment until recently, and I guess single species landings data must have also have started recently.

[Philipp-Neubauer]

Hi Mike -

thanks for the thorough QAQC and all the update work! I'll try to run through everything now to check that it all still works with the new files. For questions:

1. No, there are no other species specific filters - I excluded all that had the label 'Freshwater' in the landings DB, but otherwise its only those in the exclude column. Here's the filters - note those are only for the landings data:

filter(!grepl('Turtle',Species,ignore.case = T), !grepl('Alligator',Species,ignore.case = T), !grepl('Coral',Species,ignore.case = T), !grepl('Sponge',Species,ignore.case = T), !grepl('UNC',Species,ignore.case = T), !grepl('WHALES',Species,ignore.case = T), !grepl('FROGS',Species,ignore.case = T), !grepl('finfishes',Species,ignore.case = T), !grepl('groundfishes',Species,ignore.case = T), !grepl('DOLLY VARDEN',Species,ignore.case = T), !(Species %in% spec.ref$AFS_NAME[!spec.ref$exclude=='']), State != "At-Sea Process, Pac.", State != 'Hawaii')

and

filter(Species != 'SHELLFISH', Species != 'TUNAS', Species != 'SHARKS', Species != 'BILLFISHES', Species != 'SHARK, SILKY', Species != 'INCONNU', !grepl('SWORDFISH',Species), !grepl('MARLIN',Species), !grepl('TUNA',Species), !grepl('SALMON',Species), !grepl('TROUT',Species), !grepl('BONITO',Species), !grepl('BIGEYE',Species), !(grepl('KELP',Species) & !grepl('GREENLING, KELP',Species)), !grepl('SEAWEED',Species) )

2. No, that should be fine. All other columns should be retrievable from FB.

3. Yayx. I just checked the dataset.csv and there are indeed three entries for Blue Crab in the NE, all as un-assessed. As you suggest, this needs to be manually corrected in the code. We had a similar issue around white shrimp before - they were aggregated for some time and then dis-aggregated again. I'll add that in now, same for dogfish.

4. Not sure how important this is at this point - happy to use an updated version if that's easy enough, else I'd say we just acknowledge the limits of our dataset and finalize the manuscript.

Phil

On Sun, Dec 4, 2016 at 5:27 PM, Michael Melnychuk notifications@github.com wrote:

I also have some questions:

1.

Filtering out of species: beyond the "exclude" column in "SpeciesCrossReference.csv", are there filters in place to exclude some species? For example, are the landed species linked to FishBase habitat categories, and are freshwater fish then excluded? I added some "freshwater" values to the "exclude" column, but probably didn't get them all. 2.

Should I send more information for the newly-added stocks? In the two "v9" files, I've only added their year of first assessment, region, Latin names, and SIS stock name. Is any other information read in from these files (e.g. taxonomic, habitat, year of fishery development)? If so I can fill in the missing info. 3.

I'm not sure if we need some manual tweaking to accommodate two species that link to more than one entity under AFS_NAME:

• blue crab: 4 names: CRAB, BLUE; CRAB, BLUE, PEELER; CRAB, BLUE, SOFT; CRAB, BLUE, SOFT AND PEELER

• spiny dogfish: 2 names: SHARK, SPINY DOGFISH; SHARK, DOGFISH This may cause mismatches with linkages involving "AFS_NAME" values. Blue crab was on the list before and I think I remember it getting sorted out, but spiny dogfish is a new one with this issue. (Let me know if I can clarify.)

  1. Should we update the NMFS landings & price database, or go with the current one (last year 2013)? I'm not sure if that's important for accommodating the species that were first assessed since then. There is also one assessed species without landings data (BSAI Kamchatka flounder) because it didn't split off into a single species assessment until recently, and I guess single species landings data must have also have started recently.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Philipp-Neubauer/FirstAssessment/issues/17#issuecomment-264683689, or mute the thread https://github.com/notifications/unsubscribe-auth/ACJDC4G9vc4s6LHHyfHfFt-2qB2S0EH3ks5rEkFIgaJpZM4LDfn- .

-- Phil

[mcmelnychuk]

hi Phil,

Sounds good. For (1), a few things regarding filters:

• when you next generate a QAQC file "dataset.csv", could you add on a column for the FishBase habitat category? (If indeed that information is drawn from FishBase.) I can then look through that list and see if I missed any other freshwater species, and then add them to the 'exclude' column.

• It looks like there's an unfortunate spelling error in the NMFS landings database, "SAND LAUNCE". Could you change that 'UNC' filter to 'UNC ', with an added space?

• could you remove the 'BIGEYE' filter? There are other species with 'bigeye' in their name that should not be excluded (sharks, scads, etc); I have bigeye tuna under the exclude list so that one's taken care of.

• could you remove the 'TROUT' filter? That would remove seatrout, which we want to include. I have trout species under the exclude list ('freshwater') so those should be OK.

• I think 'SHARK, SILKY' is one of the HMS coastal sharks that we want included in the analysis.

• there are two other "KELP" species that we shoudn't remove (ROCKFISH, KELP; BASS, KELP). You could remove that line because the other kelps already have a "seaweed" entry in the "exclude" column.

(4) Happy to go with the current version!

Mike

[Philipp-Neubauer]

Perfect, thanks Mike!

On Tue, Dec 6, 2016 at 1:51 PM, Michael Melnychuk notifications@github.com wrote:

hi Phil,

Sounds good. For (1), a few things regarding filters:

• when you next generate a QAQC file "dataset.csv", could you add on a column for the FishBase habitat category? (If indeed that information is drawn from FishBase.) I can then look through that list and see if I missed any other freshwater species, and then add them to the 'exclude' column.

• It looks like there's an unfortunate spelling error in the NMFS landings database, "SAND LAUNCE". Could you change that 'UNC' filter to 'UNC ', with an added space?

• could you remove the 'BIGEYE' filter? There are other species with 'bigeye' in their name that should not be excluded (sharks, scads, etc); I have bigeye tuna under the exclude list so that one's taken care of.

• could you remove the 'TROUT' filter? That would remove seatrout, which we want to include. I have trout species under the exclude list ('freshwater') so those should be OK.

• I think 'SHARK, SILKY' is one of the HMS coastal sharks that we want included in the analysis.

• there are two other "KELP" species that we shoudn't remove (ROCKFISH, KELP; BASS, KELP). You could remove that line because the other kelps already have a "seaweed" entry in the "exclude" column.

(4) Happy to go with the current version!

Mike

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Philipp-Neubauer/FirstAssessment/issues/17#issuecomment-265027326, or mute the thread https://github.com/notifications/unsubscribe-auth/ACJDC7y7cx_HhrS4tUtUduXR3GGBCrCAks5rFLGZgaJpZM4LDfn- .

-- Phil

[Philipp-Neubauer]

OK, V9 model is running, I just pushed the dataset.csv with the updated manual tweaks and filter definitions. It does contain a habitat variable, but no freshwater references - not sure if that's because FB doesn't distinguish freshwater as a habitat or because there are no more freshwater species. Its not too long a list, so between the three of us I imagine that we wouldn't be left with too many species that we'd be uncertain about (all species I've seen so far are marine...).

On Tue, Dec 6, 2016 at 2:00 PM, Philipp Neubauer neubauer.phil@gmail.com wrote:

Perfect, thanks Mike!

On Tue, Dec 6, 2016 at 1:51 PM, Michael Melnychuk < notifications@github.com> wrote:

hi Phil,

Sounds good. For (1), a few things regarding filters:

• when you next generate a QAQC file "dataset.csv", could you add on a column for the FishBase habitat category? (If indeed that information is drawn from FishBase.) I can then look through that list and see if I missed any other freshwater species, and then add them to the 'exclude' column.

• It looks like there's an unfortunate spelling error in the NMFS landings database, "SAND LAUNCE". Could you change that 'UNC' filter to 'UNC ', with an added space?

• could you remove the 'BIGEYE' filter? There are other species with 'bigeye' in their name that should not be excluded (sharks, scads, etc); I have bigeye tuna under the exclude list so that one's taken care of.

• could you remove the 'TROUT' filter? That would remove seatrout, which we want to include. I have trout species under the exclude list ('freshwater') so those should be OK.

• I think 'SHARK, SILKY' is one of the HMS coastal sharks that we want included in the analysis.

• there are two other "KELP" species that we shoudn't remove (ROCKFISH, KELP; BASS, KELP). You could remove that line because the other kelps already have a "seaweed" entry in the "exclude" column.

(4) Happy to go with the current version!

Mike

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Philipp-Neubauer/FirstAssessment/issues/17#issuecomment-265027326, or mute the thread https://github.com/notifications/unsubscribe-auth/ACJDC7y7cx_HhrS4tUtUduXR3GGBCrCAks5rFLGZgaJpZM4LDfn- .

-- Phil

-- Phil

[Philipp-Neubauer]

Model ran, outputs are in Dropbox (still syncing on my end). Thankfully, the update didn't change the outputs much at all - and only Kamchatka flounder was added to dataset misse https://github.com/Philipp-Neubauer/FirstAssessment/blame/master/dataset_missed.csv d.

On Tue, Dec 6, 2016 at 5:57 PM, Philipp Neubauer neubauer.phil@gmail.com wrote:

OK, V9 model is running, I just pushed the dataset.csv with the updated manual tweaks and filter definitions. It does contain a habitat variable, but no freshwater references - not sure if that's because FB doesn't distinguish freshwater as a habitat or because there are no more freshwater species. Its not too long a list, so between the three of us I imagine that we wouldn't be left with too many species that we'd be uncertain about (all species I've seen so far are marine...).

On Tue, Dec 6, 2016 at 2:00 PM, Philipp Neubauer neubauer.phil@gmail.com wrote:

Perfect, thanks Mike!

On Tue, Dec 6, 2016 at 1:51 PM, Michael Melnychuk < notifications@github.com> wrote:

hi Phil,

Sounds good. For (1), a few things regarding filters:

• when you next generate a QAQC file "dataset.csv", could you add on a column for the FishBase habitat category? (If indeed that information is drawn from FishBase.) I can then look through that list and see if I missed any other freshwater species, and then add them to the 'exclude' column.

• It looks like there's an unfortunate spelling error in the NMFS landings database, "SAND LAUNCE". Could you change that 'UNC' filter to 'UNC ', with an added space?

• could you remove the 'BIGEYE' filter? There are other species with 'bigeye' in their name that should not be excluded (sharks, scads, etc); I have bigeye tuna under the exclude list so that one's taken care of.

• could you remove the 'TROUT' filter? That would remove seatrout, which we want to include. I have trout species under the exclude list ('freshwater') so those should be OK.

• I think 'SHARK, SILKY' is one of the HMS coastal sharks that we want included in the analysis.

• there are two other "KELP" species that we shoudn't remove (ROCKFISH, KELP; BASS, KELP). You could remove that line because the other kelps already have a "seaweed" entry in the "exclude" column.

(4) Happy to go with the current version!

Mike

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Philipp-Neubauer/FirstAssessment/issues/17#issuecomment-265027326, or mute the thread https://github.com/notifications/unsubscribe-auth/ACJDC7y7cx_HhrS4tUtUduXR3GGBCrCAks5rFLGZgaJpZM4LDfn- .

-- Phil

-- Phil

-- Phil