search

millerlp / rtide

R package to calculate tide heights
https://poissonconsulting.github.io/rtide/
GNU General Public License v3.0
14 stars 4 forks source link

Tool for prediction based on Reference Station #9

Open abfleishman opened 7 years ago

abfleishman commented 7 years ago

It would be cool to access the type of data that is available through this site: https://tidesandcurrents.noaa.gov/noaatidepredictions/NOAATidesFacade.jsp?Stationid=TPT2739)

It has a time and amplitude offset from a reference station

joethorley commented 7 years ago

This is a great idea - is all the information for these sites (name, coordinates, parent station, time offset and amplitude offset) in one place is a machine readable form?

abfleishman commented 7 years ago

I guess I am not sure. This site give a month of data: https://tidesandcurrents.noaa.gov/noaatidepredictions/viewMonthlyPredictions.jsp?bmon=12&bday=20&byear=2016&timelength=monthly&timeZone=1&dataUnits=1&datum=MLLW&timeUnits=2&interval=highlow&format=Submit&Stationid=TPT2739 and seems to be a .jsp form. I am not sure how to access it without downloading the txt or xml with the link on the side. I need to pull down 5 years of tide data ideally so it would be really nice to have a tool to access the data directly from R.

joethorley commented 7 years ago

If you only want for one site why not generate 5 years of data for the parent site from rtide and then manually adjust for the time and amplitude offsets in R?

abfleishman commented 7 years ago

I guess I am not fully understanding those offsets and how to implement them with the rtide output because they are different for high and low tides, and ultimately I want to get tidal heights for individual times (not just the high and low times). Does this make sense?

joethorley commented 7 years ago

Yes it makes sense. I am willing to work out how to implement (it will be relatively straightforward) for all times if someone was to provide the offsets for all US stations of this type together with their name, latitude, longitude and name of their parent station. However, I'm not willing to figure out how to do it for a single station and I'm not willing to figure how to get all the information together for all the US stations of this type.

abfleishman commented 7 years ago

I will look into putting together that info for all stations... Thanks for your cool package by the way!

joethorley commented 7 years ago

That would be great then we'll have a general solution that will be helpful to a lot of people - thanks -appreciation is always appreciated :)

abfleishman commented 7 years ago

I asked the tidesandcurrents.nooa.gov people and they said that they didn't have a table with all the info we needed so I used rvest to scrape the info from the website. I think I put together all the info about the subordinate tide stations. I have attached a compressed rda file with the resulting table and the script I used to scrape the info. All the info has been pulled from the websites in script. Let me know it all the information that you need is here, if not I can try and grab it.

tide.station.zip

joethorley commented 7 years ago

Good work!!!

I'll add you as a contributor.

Do you want to add your tide.station.r script to data-raw as a pull request so it is tracked for all the world to see (I can do if you don't want to spend the time)?

When do you want the functionality implemented by?

abfleishman commented 7 years ago

In my ideal world I guess I might have a tool to use in the next week or two, however, by no means should you drop what you are doing to work on this. Since I have no idea how much work it will be to implement, I am not sure what a reasonable time frame would be?

joethorley commented 7 years ago

I'll see what I can do.

joethorley commented 7 years ago

I'm making progress -

critical questions

1) do you know what the * in some of the height offsets refers to (+ or -)? 2) I'm still trying to figure out how to get the reference number for the stations in rtide::harmonics$Station - do you have any thoughts?

less critically

1) I'm assuming the time off sets are in minutes 2) the time zones simply state LST/LDT for Local time... do you know a way to query the actual timezone for each station?

joethorley commented 7 years ago

I mean seconds for the time zone off sets

joethorley commented 7 years ago

I've been thinking and I can use the time zone for the reference station for the substation.

abfleishman commented 7 years ago
  1. The * in the height offset is multiplying the height of the ref station values
  2. I will work on getting the ref number for the rtide::harmonics$Station my first attempt didn't work but I can spend some time today and write something to scrape them from the website.
  3. time offsets are in minutes
  4. I have code some where to look up time zone based on lat/long so I can add the correct timezone in the OlsonNames() format.

.

abfleishman commented 7 years ago

Attached are two rda files: harmonic_stations.rda has the basic station info for all the NOAA harmonic tide stations. new_harmonic_with_ref_id.rda is the rtide::harmonics$Station table with the ref station id in a new column called Id

I have a script that attaches the ref id to the rtide::harmonics$Station table, however, it appears that many of the ref stations are missing from the rtide::harmonics$Station table (~40%). The noaa site has the harmonic constituents for the missing reference stations e.g https://tidesandcurrents.noaa.gov/harcon.html?id=9751401 . What format would they need to be in to be used in rtide? Is it worth trying to scrape harmonic constituents into a table for all 848 harmonic stations that noaa has on its site? Based on lat/lon rtide currently has only 377 of the noaa harmonic sites. Thoughts? NOAA_harmonic_stations.zip

abfleishman commented 7 years ago

I have also added a script to get the OlsonNames() time zone for a given lat long and It is setup to read the stations table that I made and update the timezone column with the OlsonNames().

joethorley commented 7 years ago

Good work fine sir!

I think it would be well worth trying to scrape the harmonics for all the stations. The file was inherited without a script and it would be much more satisfying to have a script that pulls them plus the complete set of stations.

I'm not sure that the format is anything official (although I should probably declare it as an S3 class) it just makes sense logically - you can query the harmonics object out using str() if you are interested. If I have all the information digitized it is relatively straightforward to reformat.

joethorley commented 7 years ago

Note the output of str is as follows:

> str(harmonics)
List of 4
 $ Station    :Classes ‘tbl_df’, ‘tbl’ and 'data.frame':    637 obs. of  7 variables:
  ..$ Station  : chr [1:637] "Aberdeen, Grays Harbor, Washington" "Agnes Cove, Aialik Peninsula, Alaska" "Aguadilla, Crashboat Beach, Puerto Rico" "Aguchik Island, Kukak Bay, Alaska" ...
  ..$ Units    : chr [1:637] "feet" "feet" "feet" "feet" ...
  ..$ Longitude: num [1:637] -123.9 -149.6 -67.2 -154.3 -149.7 ...
  ..$ Latitude : num [1:637] 47 59.8 18.5 58.3 59.9 ...
  ..$ Hours    : num [1:637] 0 0 0 0 0 0 0 0 0 0 ...
  ..$ TZ       : chr [1:637] "America/Los_Angeles" "America/Anchorage" "America/Puerto_Rico" "America/Anchorage" ...
  ..$ Datum    : num [1:637] 5.604 5.587 0.627 7.221 5.558 ...
 $ Node       :Classes ‘tbl_df’, ‘tbl’ and 'data.frame':    175 obs. of  2 variables:
  ..$ Node : chr [1:175] "2(MN)8" "2(MS)8" "2(MS)N10" "2KM(SN)2" ...
  ..$ Speed: num [1:175] 114.8 118 146.4 30.7 129.9 ...
 $ StationNode: num [1:637, 1:175, 1:2] 0 0 0 0 0 0 0 0 0 0 ...
  ..- attr(*, "dimnames")=List of 3
  .. ..$ : chr [1:637] "Aberdeen, Grays Harbor, Washington" "Agnes Cove, Aialik Peninsula, Alaska" "Aguadilla, Crashboat Beach, Puerto Rico" "Aguchik Island, Kukak Bay, Alaska" ...
  .. ..$ : chr [1:175] "2(MN)8" "2(MS)8" "2(MS)N10" "2KM(SN)2" ...
  .. ..$ : chr [1:2] "A" "Kappa"
 $ NodeYear   : num [1:175, 1:401, 1:2] 1.147 1.071 1.109 0.625 1.025 ...
  ..- attr(*, "dimnames")=List of 3
  .. ..$ : chr [1:175] "2(MN)8" "2(MS)8" "2(MS)N10" "2KM(SN)2" ...
  .. ..$ : chr [1:401] "1700" "1701" "1702" "1703" ...
  .. ..$ : chr [1:2] "NodeFactor" "EquilArg"
 - attr(*, "class")= chr "tide_harmonics"
joethorley commented 7 years ago

Don't worry about formatting the data - I can do this because I am very familiar with the structure of the harmonics data.

joethorley commented 7 years ago

@abfleishman I have forgotten about this.... I've scoped out what I need to do. Now I just need a window to implement.

joethorley commented 7 years ago

Opps @abfleishman I meant to say I haven't forgotten about this....

abfleishman commented 7 years ago

No worries! Thanks for devoting time to it!

joethorley commented 7 years ago

@abfleishman - Apologies for the delay in resolving this issue. I have cleaned up the code you produced for scraping harmonics etc from the NOAA site in the dev branch. It's such useful code we would like to add you as a co-author on the package - if you are amenable to this?

Currently I am trying to use the TideHarmonics package lambdas and nodal_adj functions to correct for the 18.6 year tidal cycle (as opposed to using the lookup table). I am currently unable to determine when t0 is for the NOAA harmonics Phase values. Do you have any information on this?

abfleishman commented 7 years ago

I am not sure what justifies authorship for packages and libraries in open source software development, but I would be honored to be added as a co-author!

As for determining when t0 is for the NOAA data, I am not sure. I am assuming that this is the date/time from when predictions are made? Would this be associated with the current tidal datum?

While poking around the NOAA website I found this document and maybe the info you seek is in there?

joethorley commented 7 years ago

Hi Abram

The code you contributed was substantive and has really improved the package - in my books that is sufficient for co-authorship. I've added you as a co-author in the DESCRIPTION file in the dev branch which will merge with the main branch and get pushed to CRAN once I resolve the question of when t0 is for NOAA data (or give up and use the annual lookup table which includes the nodal adjustments to the phase plus a correction that sets t0 to be Jan 1st for each year).

I'm starting to think I can used the lookup table minus the nodal adjustment to determine the true t0...I'll check it out

mainahandmaker commented 2 years ago

@joethorley @abfleishman - I know this is an old thread, but did this functionality get incorporated into rtide? I am looking to get minute-by-minute tidal heights for NOAA subordinate stations based on offsets from their reference station. As far as I can tell, these subordinate stations are not included in rtide::tide_stations() - but they are included in xtide. I have the time and amplitude offsets from NOAA for all the subordinate stations I need. Is there a way to use rtide to predict tide height at these subordinate stations based on their offsets from a reference station that is included in rtide? Many thanks for your help!

joethorley commented 2 years ago

@mainahandmaker - we began working on a branch but it never made it in. Good question about plugging in time and amplitude offsets - I'm likely going to be doing some work on rtide in the next couple of months - I'll give it some thought.

mainahandmaker commented 2 years ago

thank you @joethorley! If it ended up being possible to plug in time and amplitude offsets it would be an incredible tool for our movement ecology work. I am ultimately trying to estimate tide height at each point in time and space used by the shorebirds we track with GPS transmitters, using the NOAA subordinate stations as proxies for nearby tidal conditions. If it's useful as you ponder this possibility for rtide - this site generates all the predictions I need based on xtide, but as I need to collect data for >200 sites for long time periods I am dreaming of a way to streamline this process in R. Additionally, NOAA provides this metadata API to easily get the name, coordinates, reference station, and time and amplitude offsets for all subordinate stations in the US in one place as a json or xml file. Thank you for creating rtide! I'll stay tuned for future package updates.

abfleishman commented 2 years ago

@joethorley It sounds like there is some interest (other than me) in this idea. Do you have a concept of what is left to do to allow what @mainahandmaker is asking about or what I was originally after? Maybe one of us could do some of the work to push this forward?

joethorley commented 2 years ago

Thanks @abfleishman - I'll sketch out specifics on what I think could be done in the next couple of days - pull requests gratefully received.

joethorley commented 2 years ago

@abfleishman and @mainahandmaker - apologies for the extreme delay. I've been thinking about it and think the best option to plug in time and amplitude offsets would be the addition of an extra argument called something like offsets which would be a data frame with columns named something like OffsetStation, ReferenceStation, TimeOffSet, AmplitudeOffSet and possible more if required. The default argument would be the set for the US but the user could pass their own as they wished. All ReferenceStations would need to be in the harmonics object.

joethorley commented 2 years ago

I've looked through the development code and think the best approach is to mine this issue for useful information based on the discussion and then implement as a clean PR for the current master branch from first principles complete with tests etc. Let me know if any questions.

mainahandmaker commented 2 years ago

Thanks for thinking about solutions for this, @joethorley! I think the idea for an extra argument called offsets should work well - one thing to keep in mind is that there are separate amplitude and time offsets for high vs. low tide, so that data frame will need the columns TimeOffsetLowTide, TimeOffsetHighTide, AmplitudeOffsetLowTide, AmplitudeOffsetHighTide. Other than that consideration I think this plan sounds great.

abfleishman commented 2 years ago

@mainahandmaker can you make an example DF with the info that you have for a station you are interested in?

mainahandmaker commented 2 years ago

Sure @abfleishman - here is a data frame of all stations in South Carolina as an example - with columns for the station's NOAA id code, name, type (S = subordinate station, R = reference station), lat/lon coordinates, NOAA id code of the station's reference station, and the four necessary offset columns. SC_subordinate_stations.txt

mainahandmaker commented 2 years ago

Hi @joethorley and @abfleishman -- just wanted to loop back to see if there was still interest in implementing an extra offsets argument to be able to plugin time and amplitude offsets from reference stations for all subordinate NOAA tide gauges. I am now finished with my field season so am more available to help. I don't have the coding expertise to help on the backend but I am happy to help test, etc. I will be working on some analyses starting in August that will require these time and amplitude offsets and would love to be able to use and cite rtide for this work. Thanks!

joethorley commented 2 years ago

@mainahandmaker - sorry for the delay in responding. I certainly have the interest but not the time to currently do this. I'll let you know if that changes in the near future (unlikely...)

mainahandmaker commented 2 years ago

thanks @joethorley -- please do let me know if that changes, or if you know of anyone else who may be interested/able to work on this together! I know many researchers who would benefit from this added functionality right away. Thanks again.