Completing Luquillo CZO WOFpy endpoint integration into CUAHSI WS

[emiliom]

@martinseul, I'm following up on our emails last week about our common goal to identify and fix the remaining issue(s) preventing complete functioning of @miguelcleon's Luquillo CZO WOFpy endpoint in the CUAHSI Water Services. The homepage to that WOFpy endpoint is at http://odm2admin.cuahsi.org/wofpy/odm2lczo/, unless @miguelcleon has changed it. It looks like the test network page at qa-hiscentral is http://qa-hiscentral.cuahsi.org/pub_network.aspx?n=100002, and the corresponding test service is at http://qa-hiscentral.cuahsi.org/testpage.aspx?n=100002

For reference, we've been discussing this for some time at WOFpy issue #196. The last comments were from @lsetiawan (Don) and I on Nov 15:

Based on conversation with Martin, The problem seems to lie on tags not being closed properly for empty attributes. For example <qualityControlLevelCode/> should be <qualityControlLevelCode></qualityControlLevelCode> for the xml to be validated.

and

I spoke with Martin a bit about this. What we're doing is not wrong (ie, <qualityControlLevelCode/> is not incorrect, as demonstrated by our schema validation), but it sounds like there's a client parsing expectation on their end.

As discussed via email last week, we'll use this issue to track progress. Thanks!

[miguelcleon]

seems like a conda environment problem @ocefpaf ?

[miguelcleon]

maybe it has to do with SSL?

[miguelcleon]

I tried downgrading spyne to 2.12.11 that didn't help

[miguelcleon]

OK it was an apache problem. My wsgi.py was set to the wrong application group. Making progress, now I'm getting:

 [Wed Feb 07 19:02:42.952512 2018] [wsgi:error] [pid 97906:tid 139899760125696] [remote 100.14.198.215:33266] mod_wsgi (pid=97906): Target WSGI script '/var/www/wofpy/wsgi.py' cannot be loaded as Python module.
[Wed Feb 07 19:02:42.952575 2018] [wsgi:error] [pid 97906:tid 139899760125696] [remote 100.14.198.215:33266] mod_wsgi (pid=97906): Exception occurred processing WSGI script '/var/www/wofpy/wsgi.py'.
[Wed Feb 07 19:02:42.952606 2018] [wsgi:error] [pid 97906:tid 139899760125696] [remote 100.14.198.215:33266] Traceback (most recent call last):
[Wed Feb 07 19:02:42.952642 2018] [wsgi:error] [pid 97906:tid 139899760125696] [remote 100.14.198.215:33266]   File "/var/www/wofpy/wsgi.py", line 5, in <module>
[Wed Feb 07 19:02:42.954842 2018] [wsgi:error] [pid 97906:tid 139899760125696] [remote 100.14.198.215:33266]     from singlerunserver import app as application
[Wed Feb 07 19:02:42.954902 2018] [wsgi:error] [pid 97906:tid 139899760125696] [remote 100.14.198.215:33266]   File "/var/www/wofpy/singlerunserver.py", line 9, in <module>
[Wed Feb 07 19:02:42.954989 2018] [wsgi:error] [pid 97906:tid 139899760125696] [remote 100.14.198.215:33266]     import argparse
[Wed Feb 07 19:02:42.955004 2018] [wsgi:error] [pid 97906:tid 139899760125696] [remote 100.14.198.215:33266]   File "/home/azureadmin/miniconda2/envs/wofpy5/lib/python2.7/argparse.py", line 86, in <module>
[Wed Feb 07 19:02:42.955475 2018] [wsgi:error] [pid 97906:tid 139899760125696] [remote 100.14.198.215:33266]     import copy as _copy
[Wed Feb 07 19:02:42.955490 2018] [wsgi:error] [pid 97906:tid 139899760125696] [remote 100.14.198.215:33266]   File "/home/azureadmin/miniconda2/envs/wofpy5/lib/python2.7/copy.py", line 52, in <module>
[Wed Feb 07 19:02:42.955599 2018] [wsgi:error] [pid 97906:tid 139899760125696] [remote 100.14.198.215:33266]     import weakref
[Wed Feb 07 19:02:42.955613 2018] [wsgi:error] [pid 97906:tid 139899760125696] [remote 100.14.198.215:33266]   File "/home/azureadmin/miniconda2/envs/wofpy5/lib/python2.7/weakref.py", line 14, in <module>
[Wed Feb 07 19:02:42.955739 2018] [wsgi:error] [pid 97906:tid 139899760125696] [remote 100.14.198.215:33266]     from _weakref import (
[Wed Feb 07 19:02:42.955768 2018] [wsgi:error] [pid 97906:tid 139899760125696] [remote 100.14.198.215:33266] ImportError: cannot import name _remove_dead_weakref

[emiliom]

@miguelcleon, just a heads-up: @lsetiawan and I, and even @ocefpaf, will be largely unavailable for the rest of the week, at a workshop I'm hosting here in Seattle. Don in particular probably won't have any time to do anything related to ODM2, including WOFpy, until Monday.

In the meantime, I'm glad to see you're making progress with debugging!

[miguelcleon]

@emiliom ok, thanks for letting me know.

[miguelcleon]

I had to do the following:

1) downgrade conda environment to python 2.7.13

2) added full path to config file in the runserver file:

parser.add_argument('--config',
                    help='Configuration file',
                    default='/var/www/wofpy/odm2_config_postgresql.cfg')

Now it says the URL is not found. Something must still be wrong with the configuration.

https://dev-odm2admin.cuahsi.org/WOFLCZO/

[miguelcleon]

it's working locally

[miguelcleon]

I had to uncomment the line https://github.com/ODM2/WOFpy/blob/master/wof/flask/__init__.py#L51

Why is this line commented out?

WOFpy for LCZO is now up here https://dev-odm2admin.cuahsi.org/wofpy/

[miguelcleon]

I changed it back to https://dev-odm2admin.cuahsi.org/woflczo/

[emiliom]

Testing on HIS central QA system isn't working so I'm in communication with @martinseul about that.

Darn. @miguelcleon, is this a new problem, or problems you had run into before and were unresolved? Or are you saying the HIS Central QA test system isn't working altogether (not just your endpoint)?

[miguelcleon]

I'm not sure if it is a problem we ran into before. HIS central QA system (http://qa-hiscentral.cuahsi.org/testpage.aspx?n=100002) isn't retrieving siteInfo. I asked Martin for the SOAP envelope and I'm waiting to hear back.

[emiliom]

I'm pretty sure you had gotten past retrieving siteInfo! But anyway ... fingers crossed.

[miguelcleon]

From @martinseul

I checked he soap request and that fails with a 500 error;; Request header: http://dev-odm2admin.cuahsi.org/odm2lczo/odm2lczo/soap/cuahsi_1_1/.wsdl Request body:

<?xml version="1.0" encoding="utf-8"?><soap:Envelope 
xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><soap:Body><GetSiteInfo xmlns="http://www.cuahsi.org/his/1.1/ws/"><site>odm2lczo:Rio Icacos Trib-IO</site></GetSiteInfo></soap:Body></soap:Envelope>

we also had some trouble with routing due to the dot in the url (.wsdl) do you need the dot in the route?

My response:

So if I change the envelope to:

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope 
xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<GetSiteInfo xmlns="http://www.cuahsi.org/his/1.1/ws/">
<site>Rio Icacos Trib-IO</site></GetSiteInfo>
</soap:Body></soap:Envelope>

It works

Where as, in the highlight section, you have: `

odm2lczo:Rio Icacos Trib-IO

` That would seem to be the service title. Can that be handled on your end?

[miguelcleon]

@lsetiawan see my last comment. seems like this <site>odm2lczo:Rio Icacos Trib-IO</site> was working before not sure I named the service odm2lczo though.

[miguelcleon]

From @martinseul

It’s convention to include the service title for the request all WOF services work that way.

So the odm2lczo: should be accetable

[emiliom]

Thanks for those reports, @miguelcleon.

If you're available late Friday morning (Pacific Time), let's plan on touching base then. I'll be out of town Thursday and @lsetiawan will be working for another PI.

I feel I need to first review where we got late last year. I'm pretty sure the . in .wsdl had never been a problem, and we had gotten most of the request types to succeed on the CUAHSI HIS QA system. So, w/o actually having any evidence right now, I feel that there's a regression here, relative to last November or so. I'm confused.

FYI, I don't know that the . in .wsdl is necessary on our end, but it's been like that at least since we took on WOFpy, and possibly much longer.

[miguelcleon]

Yes, OK, looks like my service name is actually woflczo so the below soap envelope works. I had changed the name, but I didn't change it on the HIS central QA system.

This one works http://qa-hiscentral.cuahsi.org/testpage.aspx?n=100005

Looks like I might need to have Martin delete the old one as I can't get it to update http://qa-hiscentral.cuahsi.org/testpage.aspx?n=100002

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope 
xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<GetSiteInfo xmlns="http://www.cuahsi.org/his/1.1/ws/">
<site>woflczo:Rio Icacos Trib-IO</site></GetSiteInfo>
</soap:Body></soap:Envelope>

[miguelcleon]

So things appear to be working after all. I asked @martinseul to ingest the new service.

[emiliom]

Great to hear, @miguelcleon! And also a big relief. Thanks for the updates.

[miguelcleon]

from @martinseul

This looks good! It harvested fine and I can see and visualize on the client. Please check it out http://qa-hiswebclient.azurewebsites.net/#

Almost there!

If you search for Luquillo you can find two time series. There should be a third at another location but that isn't showing up and I've asked Martin about that. WOFpy is pointing to a very limited database for testing.

[emiliom]

@miguelcleon It seems odd or suspicious that the Rio Icacos site only shows DO, and no temperature or conductivity. Is that what you've set up and what you expect? Just checking ...

[miguelcleon]

@emiliom yeah, that is just what I put in this test database. Once the other site shows up on the test his web client I'll try a bigger database with more time series.

[miguelcleon]

Rio Icacos is the site that is not showing up but it only has the one time series associated with it.

[emiliom]

Ok, thanks.

[miguelcleon]

From @martinseul

It seems you have spaces in the sitecode, could you remove them as this seems to create a problem for the harvesting, I’ll see on our end as well if we can mitigate this. In odm 1.1.1 allows sitecode in the range of AZ (case insensitive) , 0-9, and “.”, “-“, and “”. https://www.cuahsi.org/uploads/pages/img/ODM1.1DesignSpecifications.pdf Let me know and I try to reharvest.

Perhaps WOFpy should replace spaces in the sitecode with dashes or underscores. I'm not sure @emiliom ?

[miguelcleon]

From @martinseul

Just harvested and the site does show up now and I can download as well!

Looks good, we will try a more complete database now.

[emiliom]

Perhaps WOFpy should replace spaces in the sitecode with dashes or underscores.

Thanks for that follow up @miguelcleon. Changing WOFpy along those lines is certainly a possibility. But another parallel one is imposing constraints on ODM2 to begin with. I've opened a separate issue (#226) to discuss this.

Just harvested and the site does show up now and I can download as well!

How did you get past the sitecode bad-characters issue? Did you change the sitecode on your database, or did Martin develop mitigation steps on his end?

[miguelcleon]

I just replaced the spaces with dashes.

[miguelcleon]

OK, @martinseul ingested a nearly complete LCZO database, 12 out of 16 sites ingested properly but a few are erroring out. I'm not sure why. The problem would seem to be on the WOFpy side as I'm getting errors via REST. I spent some time trying to debug the problem but I haven't been able to figure it out

this site works and was succesfully ingested: https://dev-odm2admin.cuahsi.org/odm2lczo/odm2lczo/rest/1_1/GetSiteInfo?site=woflczo:RESSH

works: http://dev-odm2admin.cuahsi.org/odm2lczo/odm2lczo/rest/1_1/GetSites?site=woflczo:Prieta

doesn't work: http://dev-odm2admin.cuahsi.org/odm2lczo/odm2lczo/rest/1_1/GetSiteInfo?site=woflczo:Prieta

I can get @lsetiawan a copy of the database, I doubt this problem can be diagnosed without it.

Server side error:

[Mon Feb 26 20:14:33.705298 2018] [wsgi:error] [pid 121966:tid 140214398777088] ERROR:spyne.application:Fault(Server: '17252L')
[Mon Feb 26 20:14:33.705371 2018] [wsgi:error] [pid 121966:tid 140214398777088] Traceback (most recent call last):
[Mon Feb 26 20:14:33.705429 2018] [wsgi:error] [pid 121966:tid 140214398777088]   File "/home/azureadmin/miniconda2/envs/wofpy6/lib/python2.7/site-packages/spyne/application.py", line 151, in process_request
[Mon Feb 26 20:14:33.705442 2018] [wsgi:error] [pid 121966:tid 140214398777088]     ctx.out_object = self.call_wrapper(ctx)
[Mon Feb 26 20:14:33.705448 2018] [wsgi:error] [pid 121966:tid 140214398777088]   File "/home/azureadmin/miniconda2/envs/wofpy6/lib/python2.7/site-packages/spyne/application.py", line 235, in call_wrapper
[Mon Feb 26 20:14:33.705453 2018] [wsgi:error] [pid 121966:tid 140214398777088]     retval = ctx.descriptor.service_class.call_wrapper(ctx)
[Mon Feb 26 20:14:33.705458 2018] [wsgi:error] [pid 121966:tid 140214398777088]   File "/home/azureadmin/miniconda2/envs/wofpy6/lib/python2.7/site-packages/spyne/service.py", line 209, in call_wrapper32u
[Mon Feb 26 20:14:33.705464 2018] [wsgi:error] [pid 121966:tid 140214398777088]     return ctx.function(ctx, *args)
[Mon Feb 26 20:14:33.705469 2018] [wsgi:error] [pid 121966:tid 140214398777088]   File "/home/azureadmin/miniconda2/envs/wofpy6/lib/python2.7/site-packages/wof/apps/spyned_1_1.py", line 107, in GetSiteInfo
[Mon Feb 26 20:14:33.705482 2018] [wsgi:error] [pid 121966:tid 140214398777088]     siteinfoResult = WOFService.GetSiteInfoObject(ctx, site, authToken)
[Mon Feb 26 20:14:33.705487 2018] [wsgi:error] [pid 121966:tid 140214398777088]   File "/home/azureadmin/miniconda2/envs/wofpy6/lib/python2.7/site-packages/wof/apps/spyned_1_1.py", line 97, in GetSiteInfoObject
[Mon Feb 26 20:14:33.705492 2018] [wsgi:error] [pid 121966:tid 140214398777088]     raise Fault(faultstring=str(inst))
[Mon Feb 26 20:14:33.705497 2018] [wsgi:error] [pid 121966:tid 140214398777088] Fault: Fault(Server: '17252L')

[lsetiawan]

I can get @lsetiawan a copy of the database, I doubt this problem can be diagnosed without it.

That'd be great. Thanks.

[emiliom]

That's great news, @miguelcleon!

Regarding failing sites: we know your database has idiosyncracies. Some of those may be tough to identify, and probably even tougher to address -- on your end, most likely, or via a DAO customized to your database. SO:

1. I suggest you go live with the LCZO WOFpy with the sites that work (12 out of 16 is pretty darn good!). We can't know how long it'll take us to address the rest, and we can't say how much time Don and I will have to address these subtleties.
2. Let's close this monster, awful, rambling github issue and start one or more new ones that address specific problems!

[miguelcleon]

@emiliom OK sounds good. Yes, we have made a lot of progress. Thanks much!