search

afsc-gap-products / gapindex

Calculation of Design-Based Indices of Abundance and Composition for AFSC GAP Bottom Trawl Surveys
https://afsc-gap-products.github.io/gapindex/
Other
4 stars 3 forks source link

database query #10

Closed BenWilliams-NOAA closed 7 months ago

BenWilliams-NOAA commented 2 years ago

Seems that ocdb() is faster than Rodbc() may want to switch over given the size of the data you are pulling in. Here is an example https://github.com/BenWilliams-NOAA/swo/blob/09256fbdfe5bcb7d312ef83ba7357d36b8d4ccfe/R/query_data.R#L31

Also, recommend using vroom or data.table::fread for files this size, will speed up your saves and reads substantially.

https://github.com/afsc-gap-products/design-based-indices/blob/09d003fa2a7048298cf5823957d8b93a040fe287/R/00_download_data_from_oracle.R#L30

MargaretSiple-NOAA commented 2 years ago

Ben-- have you already written this script for any of the RACEBASE tables? If so, I have no problem with using yours (less duplication of effort is my goal). If not I will revise this one.

Jason-Conner-NOAA commented 1 year ago

@MargaretSiple-NOAA have you looked at the protocol switching between RODBC and ROracle in sumfish?

MargaretSiple-NOAA commented 1 year ago

@Jason-Conner-NOAA I have not -- the current structure still uses RODBC to retrieve data tables but I think in the future we may switch to a situation where we use {gfdata} or similar to get the data and then use the other scripts Zack developed here to calculate indices.