search

mroberge / hydrofunctions

A suite of convenience functions for working with hydrology data in an interactive Python session.
MIT License
66 stars 28 forks source link

Add functionality for multiple sensor / parameter code combinations #77

Closed ajb474 closed 3 years ago

ajb474 commented 4 years ago

Description

When making a general data request using the 'NWIS' function, I could not differentiate between sensor/parameter code combinations and ultimately unable to retrieve a full dataset for a location. More specifically, I am looking at a site with multiple instances of a parameter code and the NWIS request cannot specify between these. For example, if you refer to USGS site 444306122144600 the measurements of a parameter (say Turbidity / 63680) are taken from different sensors. When querying using the parameter code '63680' there is no way to get either or both instances.

What I Did

Here is a basic query using the above mentioned specifications:

data = hf.NWIS(site = '444306122144600', service = 'iv', parameterCd = '63680', start_date = '2020-03-01', end_date = '2020-10-15') 

Here is a link to the same query via the web interface:

https://nwis.waterdata.usgs.gov/nwis/uv?cb_32295=on&cb_32316=on&cb_32319=on&cb_63680=on&format=gif_stats&site_no=444306122144600&period=&begin_date=2020-03-01&end_date=2020-07-20

Note the multiple instances of the WQ parameters.
mroberge commented 4 years ago

This is interesting!

I never knew it was possible for one site to have two versions of the same parameter code! NWIS is so complex and has so many different applications that it is hard to cover them all.

I'm going to look into this and see if I can come up with a way of dealing with it.

In the meantime, I think you should be able to go data.df('all') to get the full dataframe. Or, you can go data.json to see the raw json response.

I'll take a look into this! Thank you!

ajb474 commented 4 years ago

Thank you for being so responsive.

I just tried to view the full dataframe and the same level of data is available for the parameter code (with the addition of the '_qualifiers' column).

Thanks again and let me know if I can clarify anything!

ajb474 commented 3 years ago

Hello again, I wanted to check back and see if you were able to understand my issue or if you have had the chance to look into this at all.

Thank you!

mroberge commented 3 years ago

Hi Adam! @ajb474 Thank you for your interest in this! So the issue here is that this goes right to the heart of my algorithm for decoding the NWIS response. It is an overly complex algorithm as it is, and I never anticipated that the WaterML format could have more than one sensor with the same parameter at a site... but now that I've made my perfunctory excuses, I think I've got a solution!

It seems too easy, but I just look for multiple 'method'-series inside of a series, and then treat it like another series. It is basically just an extra for loop. You can see my progress on this branch: https://github.com/mroberge/hydrofunctions/tree/bugfix-multiple-methods-in-NWIS-json The problem now (and reason why I'm a little slow) is that I need to come up with some tests to make sure that I'm not breaking anything.

Would you be interested in helping out? I need to:

BTW, I just noticed that you were NAU! I got my PhD from ASU Geography!

Anyway, thank you, and I hope I can get something working soon. You can try using the new branch, if you want. -Marty

mroberge commented 3 years ago

I think I've got this one solved. I'm closing the issue, but feel free to comment if you find another site or situation that has multiple methods for a single parameter.

ajb474 commented 3 years ago

Thank you so much for taking this on! Believe it or not providing you some edge cases was on my to do list this week, I've just been a bit behind. Actually, I was working with the R version (dataRetrieval) last week and found some good examples of the issue. It was the same deal, for a single site and WQ parameter multiple instruments were used for data collection at different times.

Thanks again and if I can be of any help in the future please let me know.

On Tue, Nov 17, 2020, 21:36 Martin Roberge notifications@github.com wrote:

I think I've got this one solved. I'm closing the issue, but feel free to comment if you find another site or situation that has multiple methods for a single parameter.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mroberge/hydrofunctions/issues/77#issuecomment-729400593, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGBCO6QPLSTSHZ56BIOG5F3SQNFOHANCNFSM4TBFQCZQ .