EnviroDIY WOFpy registration with CUAHSI data services

[emiliom]

This issue will be a central discussion thread for getting the EnviroDIY WOFpy endpoint fully registered and in production at the CUAHSI data services.

We'll add more background information soon.

There's relevant past discussions at this issue:

• https://github.com/ODM2/WOFpy/issues/219

cc @jcaraballo17 @aufdenkampe @lsetiawan @miguelcleon

[emiliom]

ODM2 deployments (both live and sandbox) are listed here: https://github.com/BiG-CZ/bigcz_wshp2017/issues/1

[aufdenkampe]

@horsburgh, how can we move forward on this? @emiliom and @miguelcleon have had success with their Luquillo CZO WOFpy registration. It is listed on http://qa-hiswebclient.azurewebsites.net/ , but there is no service metadata or data streams.

[emiliom]

Updated 2/27/2018

There is a registered service on the test CUAHSI system, but it's not working (or at least this page doesn't show anything): http://hiscentral.cuahsi.org/pub_network.aspx?n=100003

The "Service Test Page" is at http://qa-hiscentral.cuahsi.org/testpage.aspx?n=100003. This page does work!

[aufdenkampe]

Deadline: March 13 release of WikiWatershed, with first release of Monitor My Watershed, which we hope to use to feature EnviroDIY.

[miguelcleon]

@emiliom @aufdenkampe @horsburgh on the call today I said the CUAHSI test hydroclient (http://qa-hiswebclient.azurewebsites.net) was erroring when searching for envirodiy data. I just noticed that it is also erroring when searching Luquillo CZO data. I tried searching some other services and they also error out. So I think something is wrong with CUAHSI test hydroclient. I will email @martinseul

[emiliom]

Thanks @miguelcleon. I assume this was working fine on the Luquillo CZO last week?

[miguelcleon]

Yes, it was working yesterday in fact.

[miguelcleon]

from @martinseul

We had a problem with one of our dev servers that got hacked. We had to shut it down and move to a new server. That is completed now and the app should work as expected, sorry for the inconvenience.

Please check and let me know if you find additional problems.

I'm still getting an error, I emailed him back.

[miguelcleon]

@aufdenkampe @horsburgh @emiliom

It is working now. envirodiy also works, you can download envirodiy data from the QA hydroclient http://qa-hiswebclient.azurewebsites.net/ take a look.

I did get an error when downloading some time series others ones worked fine. An error occured downloading file: envirodiy-sara's_desk_at_stroud-relative_humidity.csv

[emiliom]

Great.

[emiliom]

Copying over to here the update from Yaping's email (@ypxiao):

Please go to http://data.cuahsi.org and search for id=5644. You should be able to search/download the data.

I confirmed yesterday that the service and time series were available at data.cuahsi.org. Also, the service info page is looking nice: http://hiscentral.cuahsi.org/pub_network.aspx?n=5644

The service may not have propagated to the catalog API or the BiGCZ portal CUAHSI catalog service cache (updated daily? I can't remember), b/c searching for "envirodiy" or "wikiwatershed" didn't yield anything in the White Clay Creek area, just now; also, searching on "stream" didn't produce envirodiy sites.

I also tried the staging wikiwatershed app, but the search functionality doesn't seem to be working at all there.

[horsburgh]

The EnviroDIY service has now been registered at HIS Central:

http://hiscentral.cuahsi.org/pub_network.aspx?n=5644

I still have some questions for fleshing out the public details page for the service:

• What logo should I use for the service? EnviroDIY or WikiWatershed?
• Do we want to customize the icon for the sites?
• What should we list for the citation for the service?
• Are there additional metadata elements we want to add? You can define whatever you want, you just need an element name and a text value for the element.
• I couldn’t get the button to edit the funding sources to work correctly, but I can try and get that added as well.
• Finally – this is registered using my contact information and account at HIS Central. Should I be the contact or should someone else?

[emiliom]

@aufdenkampe: Data from the envirodiy service is not showing up in CUAHSI WDC searches under either the staging wikiwatershed portal or the BiGCZ portal. That means that the service is either not being advertised yet in the CUAHSI catalog API, or the wikiwatershed/BiGCZ portal is using a stale, (not updated) cache of service endpoints.

[aufdenkampe]

It's been >3 days since the WikiWatershed EnviroDIY Water Quality Data Portal service (http://hiscentral.cuahsi.org/pub_network.aspx?n=5644 and http://data.cuahsi.org & search for id=5644) has been available and functioning.

Unfortunately, we still can't find any of the sites from https://portal.bigcz.org or https://staging.app.wikiwatershed.org.

@ypxiao & @emiliom, can you let us know if the WikiWatershed EnviroDIY service is available via the CUAHSI catalog API? Perhaps a quick test with Python could give this answer.

@ajrobbins, @rajadain, could you let us know if there might be a caching issue in the BiGCZ & WikiWatershed web apps?

We would like to get this figured out by the end of the week, in advance of the March 13 release of the WikiWatershed web app.

[miguelcleon]

@aufdenkampe @emiliom Martin had mentioned to me that service data are cached once a week, I believe on Friday, so maybe it will be updated then.

[ajrobbins]

@rajadain refreshed the cache, which addressed this - services are cached for a week on staging, to avoid requesting them anew every time.

[aufdenkampe]

@horsburgh, here are some responses to the web service metadata.

• Organization:
  • change from WikiWatershed Water Quality Data Portal
  • to WikiWatershed
• Title:
  • change from WikiWatershed EnviroDIY Water Quality Data Portal
  • to WikiWatershed EnviroDIY Data Sharing Portal
• What logo should I use for the service? EnviroDIY or WikiWatershed?
  • For now, let's use EnviroDIY square, at https://drive.google.com/drive/folders/0B0V1GWNa8ZO6NTFDcUQ4MzBLZmM
  • @heatherbrooks, can you create a new combined square logo similar to EnviroDIY but that also WikiWatershed listed under EnviroDIY?
• Do we want to customize the icon for the sites?
  • Sure. What are the options. Where is this visible? In http://data.cuahsi.org?
• What should we list for the citation for the service?
  • WikiWatershed™ and EnviroDIY™ are an initiatives of Stroud Water Research Center, which seeks to advance knowledge and stewardship of fresh water through global research, education, and watershed restoration.
• Are there additional metadata elements we want to add? You can define whatever you want, you just need an element name and a text value for the element.
  • Documents and Publications. Provide links to:
  • WikiWatershed: http://wikiwatershed.org/
  • EnviroDIY Community: https://envirodiy.org/
  • Contributing Organizations/Grants
  • Fields: Organization, Agency Name, Award ID, Start Date, End Date, Grant URL, Completed?
  • ODM2 grants:
    • ​NSF ACI 1339834. Collaborative Research: SI2-SSI: The Community-Driven BiG CZ Software System for Integration and Analysis of Bio- and Geoscience Data in the Critical Zone. http://www.nsf.gov/awardsearch/showAward?AWD_ID=1339834
    • NSF EAR 1224638. Developing a Community Information Model and Supporting Software to Extend Interoperability of Sensor and Sample Based Earth Observations. http://www.nsf.gov/awardsearch/showAward?AWD_ID=1224638
  • EnviroDIY grants:
    • William Penn Foundation. (WPF) grant 158-15 "EnviroDIY CitSci Data Portal, Activity 2.1 in Supporting Citizen Science within the Delaware River Watershed Initiative (DRWI) Cluster Teams". 2016-2019
    • NSF EAR 1331856. Christina River Basin CZO: a whole watershed approach to integrating feedbacks between water, mineral and carbon fluxes in human landscapes. http://www.nsf.gov/awardsearch/showAward?AWD_ID=1331856
  • WikiWatershed Grants:
    • William Penn Foundation. (WPF) Grant 12-17 to "Expand the WikiWatershed web-based tool to enhance its use by conservation organizations in the Delaware River Watershed Initiative (DRWI) to identify scale and type of conservation projects needed to meet water quality goals." 2017-2019
    • NSF DRL 1418133. Collaborative Research: Teaching Environmental Sustainability - Model My Watershed. http://www.nsf.gov/awardsearch/showAward?AWD_ID=1418133
• I couldn’t get the button to edit the funding sources to work correctly, but I can try and get that added as well.
  • Thanks.
• Finally – this is registered using my contact information and account at HIS Central. Should I be the contact or should someone else?
  • Let's change this to the new data portal Google Group that we just set up:
    • WikiWatershed Data Portal Help ww-data-portal-help@googlegroups.com
    • Is phone number required? If so, do we use one for @heatherbrooks

[HeatherBrooks]

@aufdenkampe Combining the logos doesn't give great results ... will this work?

[aufdenkampe]

That's not bad. Can we vertically compress a bit, moving both lines of text upward? Maybe change the blues in the WikiWatershed wave to match the one in the EnviroDIY logo?

[HeatherBrooks]

I don't really have the proper tools for changing the wave colors so the fine wave ends aren't changed. If the logo will be used at a smaller scale, this might work.

[horsburgh]

@aufdenkampe - I was able to add the funding sources to the WOFPy service registration page at HIS Central. The normal "add funding sources" doesn't seem to work. Liza from CUAHSI told me that you have to add funding organizations to the underlying database, have them harvested by the metadata harvester, and then you can use the funding sources editor. But, I'm not sure how that would work because funding agencies listed as sources would never be associated with data and so wouldn't get harvested.

I don't love how it is implemented now. I had to list all of the funding sources in separate metadata elements because if I put them all in one the website strips out any formatting, new lines, etc., so it just get's jumbled into one long continuous text string. Let me know if you want me to do something different.

[aufdenkampe]

I think all this has been resolved. Thanks everyone.