Add Conda Channel information to documentation

[emiliom]

Let'sl use this issue that @valentinedwv started to iron out wofpy installation instructions, including problems along the way. I'll start by redirecting @miguelcleon's email from 8/12/2016 to here:

I was able to get the wofpy conda package to install, after I installed miniconda. But now I get an import error: No module named api.ODM2. So I tried installing the odm2api conda package to the same venv. But I still get the same error.

Miguel then said:

This line of code appears correct, but I don’t think it is updated in the conda package. https://github.com/ODM2/WOFpy/blob/master/examples/flask/odm2/timeseries/odm2_timeseries_dao.py#L10

1. conda package installation

Miguel, I'm glad you were able to install miniconda. If you don't mind, please post here a brief summary of the steps you used (web site, etc) to install miniconda, what OS you're using, etc.

Looking into WOFpy, I can see odm2api is not a required dependency. I can see why, b/c you can have a WOFpy deployment not tied to an ODM2 database (like @SRGDamia1 is doing). So, as Miguel pointed out, odm2api must be installed explicitly.

Here's a sample, minimalist statement that will install wofpy and odm2api into the conda environment wofpyodm2api:

conda create -c conda-forge -c odm2 -n wofpyodm2api python=2.7 wofpy odm2api

(do use "-c conda-forge"; it might not be strictly needed, but it's best to do it at this time. I'll add documentation on this later on)

2. wofpy odm2api errors

Now regarding the errors Miguel is still seeing with wofpy, involving ODM2:

• Can you tell us where this error is occurring: "No module named api.ODM2"?
• Your reported a problem in this line: https://github.com/ODM2/WOFpy/blob/master/examples/flask/odm2/timeseries/odm2_timeseries_dao.py#L10. But I was able to issue that import statement (from odm2api.ODM2 import models as odm2_models) w/o any problems on an ipython session using the above conda environment.

@sreeder, you were able to get past this point, weren't you? Can you chime in?

Thanks, all!

cc @SRGDamia1 and @sreeder, since they were/are part of this conversation

[miguelcleon]

The problem I encountered is that when I downloaded the this | linux-64/wofpy-2.0.9b0-py27_0.tar.bz2 from https://anaconda.org/odm2/wofpy/files

Line 10 corresponding to `[https://github.com/ODM2/WOFpy/blob/master/examples/flask/odm2/timeseries/odm2_timeseries_dao.py#L10]

instead reads:

from api.ODM2 import models as odm2_models

Which doesn't work. When I changed it to from odm2api.ODM2 import models as odm2_models

Mannually it worked.

[miguelcleon]

After I manually change that line of code, runserver_odm2_timesser.py seems to be able to succescfully run and read my db connection file and configuration file but I get another error. Seems like I'm getting closer.

[ocefpaf]

@miguelcleon can you post the result of conda list and conda info here. I suspect I know what may be happening there.

[miguelcleon]

Actually I did a reboot and it seems to be working now :) I clicked to first link hopefully it will not crash the browser.

[ocefpaf]

Cool. The shapely issue there is b/c ctypes could not find geos and that usually means another library is in the way. (We patch shapely to always find conda's geos but sometimes that fails due to another installation of some of the libs being present.)

[emiliom]

Great to see your progress, Miguel! Thanks for reporting back on the odm2api issue; that's clear now.

I think you may still be running into "growing pains" from being new to conda. And we still need to create simple instructions on that. I'll try to have something ready by the end of the day.

In the meantime, it sounds like you're almost all set! Though there may be additional customizations to be made to the ODM2 time series WOFpy DAO to refine it for your context (and more generally).

@ocefpaf, thanks for chiming in.

[miguelcleon]

yeah, just using download Anaconda at the top of https://anaconda.org/odm2 seemed to send me astray.

[emiliom]

@miguelcleon and @sreeder: have you gotten farther along with your installation and use of WOFpy?

Miguel, I noticed just now that your successful installation was actually on Jeff's sample Little Bear dataset, not your own Luquillo CZO database. Still, have you been successful with the Little Bear test DB?

Any additional, helpful info you can add about the installation and configuration of WOFpy for ODM2 time series?

Thanks!

[sreeder]

@emiliom I have not gotten any further than we were at our meeting on Friday. I have WOFpy up and running but I cannot get it to successfully return data. There seems to be errors with the queries being run to return data. I am also running Jeff's sample Little Bear River dataset. The Variable and the Site do exist in my database.

[emiliom]

@sreeder, thanks for reporting back. While that may not be positive movement forward, hearing about what problems you're running into is still helpful.

Quick question about "I am also running Jeff's sample Little Bear River dataset." So, you're using exactly the same database dump available online, that Miguel is using, and the San Diego team developed the WOFpy ODM2 timeseries example, right? Not a local copy of the database that might be 98% identical.

[miguelcleon]

@emiliom I haven't made any progress, the services seem to start up but I can never get any of the links to load. I'm connecting to a local copy of the LCZO ODM2 database though.

[sreeder]

@emiliom I am using a local copy that could be 98% identical

[miguelcleon]

I thought I'd reboot and give it another try. Now instead of it seeming to run unendingly doing nothing, I get errors :) from http://127.0.0.1:8080/odm2timeseries/soap/cuahsi_1_0/.wsdl I get: traceback (most recent call last): File "/home/azureadmin/miniconda2/lib/python2.7/site-packages/flask/app.py", line 2000, in __call__ return self.wsgi_app(environ, start_response) File "/home/azureadmin/miniconda2/lib/python2.7/site-packages/werkzeug/wsgi.py", line 659, in __call__ return app(environ, start_response) File "/home/azureadmin/miniconda2/lib/python2.7/site-packages/spyne/server/wsgi.py", line 241, in __call__ return self.handle_wsdl_request(req_env, start_response, url) File "/home/azureadmin/miniconda2/lib/python2.7/site-packages/spyne/server/wsgi.py", line 298, in handle_wsdl_request self.event_manager.fire_event('wsdl', ctx) File "/home/azureadmin/miniconda2/lib/python2.7/site-packages/spyne/_base.py", line 613, in fire_event handler(ctx, *args, **kwargs) File "/home/azureadmin/miniconda2/lib/python2.7/site-packages/wof/WofWsdls.py", line 90, in on_get_wsdl_1_0_ template = env.get_template('wsdl_temp.wsdl') File "/home/azureadmin/miniconda2/lib/python2.7/site-packages/jinja2/environment.py", line 812, in get_template return self._load_template(name, self.make_globals(globals)) File "/home/azureadmin/miniconda2/lib/python2.7/site-packages/jinja2/environment.py", line 774, in _load_template cache_key = self.loader.get_source(self, name)[1] File "/home/azureadmin/miniconda2/lib/python2.7/site-packages/jinja2/loaders.py", line 187, in get_source raise TemplateNotFound(template) TemplateNotFound: wsdl_temp.wsdl

[miguelcleon]

from here: http://127.0.0.1:8080/odm2timeseries/soap/cuahsi_1_1/ I get the following, it seems like it wants me to ask it for some content-type, I wonder how I do that?

`

soap11env:Client.RequestNotAllowed You must issue a POST request with the Content-Type header properly set. `

[valentinedwv]

That template is based on the path from where you start the program to where the templates are located. so where are you tying the command from, and what does the config file look like

On Wed, Aug 17, 2016 at 3:26 PM, Miguel Leon notifications@github.com wrote:

I thought I'd reboot and give it another try. Now instead of it seeming to run unendingly doing nothing, I get errors :) from http://127.0.0.1:8080/odm2timeseries/soap/cuahsi_1_0/.wsdl I get: traceback (most recent call last): File "/home/azureadmin/miniconda2/lib/python2.7/site-packages/flask/app.py", line 2000, in call return self.wsgi_app(environ, start_response) File "/home/azureadmin/miniconda2/lib/python2.7/site-packages/werkzeug/wsgi.py", line 659, in call return app(environ, start_response) File "/home/azureadmin/miniconda2/lib/python2.7/site-packages/spyne/server/wsgi.py", line 241, in call return self.handle_wsdl_request(req_env, start_response, url) File "/home/azureadmin/miniconda2/lib/python2.7/site-packages/spyne/server/wsgi.py", line 298, in handle_wsdl_request self.event_manager.fire_event('wsdl', ctx) File "/home/azureadmin/miniconda2/lib/python2.7/site-packages/spyne/_base.py", line 613, in fire_event handler(ctx, _args, *_kwargs) File "/home/azureadmin/miniconda2/lib/python2.7/site-packages/wof/WofWsdls.py", line 90, in on_get_wsdl_10 template = env.get_template('wsdl_temp.wsdl') File "/home/azureadmin/miniconda2/lib/python2.7/site-packages/jinja2/environment.py", line 812, in get_template return self._load_template(name, self.make_globals(globals)) File "/home/azureadmin/miniconda2/lib/python2.7/site-packages/jinja2/environment.py", line 774, in _load_template cache_key = self.loader.get_source(self, name)[1] File "/home/azureadmin/miniconda2/lib/python2.7/site-packages/jinja2/loaders.py", line 187, in get_source raise TemplateNotFound(template) TemplateNotFound: wsdl_temp.wsdl

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ODM2/WOFpy/issues/59#issuecomment-240568358, or mute the thread https://github.com/notifications/unsubscribe-auth/AAz_SG20BJg89SJPL56gysVn2O-GfPVLks5qg4qMgaJpZM4JjRx1 .

[emiliom]

Thanks for your input, everyone. Lots to chew on. Let's see if there's any progress before Friday, to see if we should talk about this on the BiGCZ call on Friday.

[valentinedwv]

I out on Friday.

On Wed, Aug 17, 2016 at 4:54 PM, Emilio Mayorga notifications@github.com wrote:

Thanks for your input, everyone. Lots to chew on. Let's see if there's any progress before Friday, to see if we should talk about this on the BiGCZ call on Friday.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ODM2/WOFpy/issues/59#issuecomment-240584752, or mute the thread https://github.com/notifications/unsubscribe-auth/AAz_SHgqLJAAAi-ABSJsKcmaWw5_Jvozks5qg58lgaJpZM4JjRx1 .

[miguelcleon]

yes I have no templates in Templates=../../../../wof/apps/templates If that is what the error is about. Should there be templates in here?

[miguelcleon]

I found templates in the github copy of wofpy copied it into the miniconda folder and it works! I see a bunch of xml but it doesn't seem to have to do with my site or anything in my db so far.

[miguelcleon]

I'm making progress, my server connection string was wrong but I fixed it then I figured out I needed to go to someplace like this: http://127.0.0.1:8080/odm2timeseries/rest/1_1/GetSites

But then I got this:

(psycopg2.ProgrammingError) column samplingfeatures.featuregeometrywkt does not exist

it is true my table samplingfeatures does not have a column featuregeometrywkt it does have a column featuregeometry

[miguelcleon]

So if I try:

http://127.0.0.1:8080/odm2timeseries/rest/1_1/GetValuesForASite?site=odm2timeseries:LCZO&startDate=2015-09-01T00:00:00&endDate=2015-10-01T02:30:00

I get:

(psycopg2.ProgrammingError) column "results.resultid" must appear in the GROUP BY clause or be used in an aggregate function LINE 1: SELECT odm2.results.resultid AS odm2_results_resultid, odm2.... ^ [SQL: 'SELECT odm2.results.resultid AS odm2_results_resultid, odm2.results.resultuuid AS odm2_results_resultuuid, odm2.results.featureactionid AS odm2_results_featureactionid, odm2.results.resulttypecv AS odm2_results_resulttypecv, odm2.results.variableid AS odm2_results_variableid, odm2.results.unitsid AS odm2_results_unitsid, odm2.results.taxonomicclassifierid AS odm2_results_taxonomicclassifierid, odm2.results.processinglevelid AS odm2_results_processinglevelid, odm2.results.resultdatetime AS odm2_results_resultdatetime, odm2.results.resultdatetimeutcoffset AS odm2_results_resultdatetimeutcoffset, odm2.results.validdatetime AS odm2_results_validdatetime, odm2.results.validdatetimeutcoffset AS odm2_results_validdatetimeutcoffset, odm2.results.statuscv AS odm2_results_statuscv, odm2.results.sampledmediumcv AS odm2_results_sampledmediumcv, odm2.results.valuecount AS odm2_results_valuecount, CASE WHEN (odm2.results.resulttypecv = %(resulttypecv_1)s) THEN %(param_1)s WHEN (odm2.results.resulttypecv = %(resulttypecv_2)s) THEN %(param_2)s WHEN (odm2.results.resulttypecv = %(resulttypecv_3)s) THEN %(param_3)s WHEN (odm2.results.resulttypecv = %(resulttypecv_4)s) THEN %(param_4)s WHEN (odm2.results.resulttypecv = %(resulttypecv_5)s) THEN %(param_5)s WHEN (odm2.results.resulttypecv = %(resulttypecv_6)s) THEN %(param_6)s WHEN (odm2.results.resulttypecv = %(resulttypecv_7)s) THEN %(param_7)s WHEN (odm2.results.resulttypecv = %(resulttypecv_8)s) THEN %(param_8)s WHEN (odm2.results.resulttypecv = %(resulttypecv_9)s) THEN %(param_9)s WHEN (odm2.results.resulttypecv = %(resulttypecv_10)s) THEN %(param_10)s ELSE %(param_11)s END AS _sa_polymorphic_on \nFROM odm2.results JOIN odm2.featureactions ON odm2.featureactions.featureactionid = odm2.results.featureactionid JOIN odm2.samplingfeatures ON odm2.samplingfeatures.samplingfeatureid = odm2.featureactions.samplingfeatureid \nWHERE odm2.samplingfeatures.samplingfeaturecode = %(samplingfeaturecode_1)s GROUP BY odm2.results.variableid'] [parameters: {'param_11': 'results', 'param_10': 'Measurement', 'resulttypecv_3': 'Category coverage', 'resulttypecv_2': 'Profile Coverage', 'resulttypecv_1': 'Point coverage', 'resulttypecv_7': 'Section coverage', 'resulttypecv_6': 'Time series coverage', 'resulttypecv_5': 'Spectra coverage', 'resulttypecv_4': 'Transect Coverage', 'resulttypecv_9': 'Trajectory coverage', 'resulttypecv_8': 'Profile Coverage', 'resulttypecv_10': 'Measurement', 'param_9': 'Trajectory coverage', 'param_8': 'Profile Coverage', 'param_5': 'Spectra coverage', 'param_4': 'Transect Coverage', 'param_7': 'Section coverage', 'param_6': 'Time series coverage', 'param_1': 'Point coverage', 'param_3': 'Category coverage', 'param_2': 'Profile Coverage', 'samplingfeaturecode_1': u'LCZO'}]

[emiliom]

Miguel: One thing I can say is that samplingfeatures.featuregeometrywkt was added somewhat recently (early this year, I think), together with expectations/recommendations for samplingfeatures.featuregeometrywkt vs samplingfeatures.featuregeometry, where the former is now recommended/expected but the latter is optional and often ignored in ODM2 software. So it's very likely that WOFpy expects featuregeometrywkt but not featuregeometry.

One quick & dirty stop-gap for you would be to manually add that column to the table, using the default field definition in the ODM2 postgresql creation script; then apply a SQL update statement to populate the latter based on the former and standard PostGIS SQL functions. If you'd like to pursue this route, I can help.

[miguelcleon]

Ok I added the featuregeometrywkt field, how should this be used? I'm guressing I can store my coordinates here as plane text so they are human readable? what implications does this have for the mapping on OMD2 Admin?

from here http://127.0.0.1:8080/odm2timeseries/rest/1_1/GetSites I now get the second error starting with (psycopg2.ProgrammingError) column "results.resultid" must appear ....

[sreeder]

this is the issue i get: `Traceback (most recent call last): File "C:\Anaconda\envs\wofpy\lib\site-packages\spyne\application.py", line 151, in process_request ctx.out_object = self.call_wrapper(ctx) File "C:\Anaconda\envs\wofpy\lib\site-packages\spyne\application.py", line 235, in call_wrapper retval = ctx.descriptor.service_class.call_wrapper(ctx) File "C:\Anaconda\envs\wofpy\lib\site-packages\spyne\service.py", line 209, in call_wrapper return ctx.function(ctx, *args) File "C:\Anaconda\envs\wofpy\lib\site-packages\wof\apps\waterml2.py", line 51, in GetValues variable_result = wof_inst.dao.get_variable_by_code(varCode) File "D:\DEV\WOFpy\examples\flask\odm2\timeseries\odm2_timeseries_dao.py", line 138, in get_variable_by_code v_arr = self.get_variables_from_results(var_code) File "D:\DEV\WOFpy\examples\flask\odm2\timeseries\odm2_timeseries_dao.py", line 116, in get_variables_from_results group_by(odm2_models.Variables.VariableID).all() File "C:\Anaconda\envs\wofpy\lib\site-packages\sqlalchemy\orm\query.py", line 2613, in all return list(self) File "C:\Anaconda\envs\wofpy\lib\site-packages\sqlalchemy\orm\query.py", line 2761, in iter return self._execute_and_instances(context) File "C:\Anaconda\envs\wofpy\lib\site-packages\sqlalchemy\orm\query.py", line 2776, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "C:\Anaconda\envs\wofpy\lib\site-packages\sqlalchemy\engine\base.py", line 914, in execute return meth(self, multiparams, params) File "C:\Anaconda\envs\wofpy\lib\site-packages\sqlalchemy\sql\elements.py", line 323, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "C:\Anaconda\envs\wofpy\lib\site-packages\sqlalchemy\engine\base.py", line 1010, in _execute_clauseelement compiled_sql, distilled_params File "C:\Anaconda\envs\wofpy\lib\site-packages\sqlalchemy\engine\base.py", line 1146, in _execute_context context) File "C:\Anaconda\envs\wofpy\lib\site-packages\sqlalchemy\engine\base.py", line 1341, in _handle_dbapi_exception exc_info File "C:\Anaconda\envs\wofpy\lib\site-packages\sqlalchemy\util\compat.py", line 202, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "C:\Anaconda\envs\wofpy\lib\site-packages\sqlalchemy\engine\base.py", line 1139, in _execute_context context) File "C:\Anaconda\envs\wofpy\lib\site-packages\sqlalchemy\engine\default.py", line 450, in do_execute cursor.execute(statement, parameters) File "C:\Anaconda\envs\wofpy\lib\site-packages\pymysql\cursors.py", line 167, in execute result = self._query(query) File "C:\Anaconda\envs\wofpy\lib\site-packages\pymysql\cursors.py", line 323, in _query conn.query(q) File "C:\Anaconda\envs\wofpy\lib\site-packages\pymysql\connections.py", line 836, in query self._affected_rows = self._read_query_result(unbuffered=unbuffered) File "C:\Anaconda\envs\wofpy\lib\site-packages\pymysql\connections.py", line 1020, in _read_query_result result.read() File "C:\Anaconda\envs\wofpy\lib\site-packages\pymysql\connections.py", line 1303, in read first_packet = self.connection._read_packet() File "C:\Anaconda\envs\wofpy\lib\site-packages\pymysql\connections.py", line 982, in _read_packet packet.check_error() File "C:\Anaconda\envs\wofpy\lib\site-packages\pymysql\connections.py", line 394, in check_error err.raise_mysql_exception(self._data) File "C:\Anaconda\envs\wofpy\lib\site-packages\pymysql\err.py", line 120, in raise_mysql_exception _check_mysql_exception(errinfo) File "C:\Anaconda\envs\wofpy\lib\site-packages\pymysql\err.py", line 115, in _check_mysql_exception raise InternalError(errno, errorvalue)

InternalError: (pymysql.err.InternalError) (1066, u"Not unique table/alias: 'results'") [SQL: u'SELECT odm2.timeseriesresultvalues.valueid AS odm2_timeseriesresultvalues_valueid, odm2.timeseriesresultvalues.resultid AS odm2_timeseriesresultvalues_resultid, odm2.timeseriesresultvalues.datavalue AS odm2_timeseriesresultvalues_datavalue, odm2.timeseriesresultvalues.valuedatetime AS odm2_timeseriesresultvalues_valuedatetime, odm2.timeseriesresultvalues.valuedatetimeutcoffset AS odm2_timeseriesresultvalues_valuedatetimeutcoffset, odm2.timeseriesresultvalues.censorcodecv AS odm2_timeseriesresultvalues_censorcodecv, odm2.timeseriesresultvalues.qualitycodecv AS odm2_timeseriesresultvalues_qualitycodecv, odm2.timeseriesresultvalues.timeaggregationinterval AS odm2_timeseriesresultvalues_timeaggregationinterval, odm2.timeseriesresultvalues.timeaggregationintervalunitsid AS odm2_timeseriesresultvalues_timeaggregationintervalunitsid \nFROM odm2.timeseriesresultvalues INNER JOIN (odm2.results INNER JOIN odm2.timeseriesresults ON odm2.results.resultid = odm2.timeseriesresults.resultid) ON odm2.timeseriesresults.resultid = odm2.timeseriesresultvalues.resultid INNER JOIN odm2.results ON odm2.results.resultid = odm2.timeseriesresults.resultid INNER JOIN odm2.variables ON odm2.variables.variableid = odm2.results.variableid \nWHERE odm2.variables.variablecode = %(variablecode_1)s GROUP BY odm2.variables.variableid'] [parameters: {u'variablecode_1': u'USU3'}]`

[emiliom]

In response to @miguelcleon's last post:

Ok I added the featuregeometrywkt field, how should this be used? I'm guessing I can store my coordinates here as plane text so they are human readable? what implications does this have for the mapping on OMD2 Admin?

Regarding implications for ODM2 Admin: I tend to be an incrementalist, so my suggestion for now is to ignore the new featuregeometrywkt field. For now. That way there are no immediate disruptions on ODM2 Admin functionality, including the new mapping functionality. Maybe we can work together with Don and others after next week to discuss featuregeometrywkt usage and incorporate it more fully into ODM2 Admin.

As a I said earlier, you can then manually update featuregeometrywkt from featuregeometry via a SQL statement. Here's the core of it:

UPDATE odm2.samplingfeatures SET featuregeometrywkt = ST_AsText(featuregeometry)

You can add a WHERE filter as needed.

A WKT representation of point geometry looks like this: POINT(-119.195 37.0675)

[emiliom]

@sreeder, thanks for the error report. @valentinedwv won't be at the BiGCZ call today. But we can see if there are any commonalities between the errors you and @miguelcleon are running into, and if there's anything we can do to mover forward a bit.

@sreeder, have you tried running WOFpy against the Little Bear sample database downloaded directly from github, to ensure that it's 100% the same as what the San Diego team deployed with WOFpy on their system?

[lsetiawan]

I think this issue is ironed out in the new documentation: https://github.com/ODM2/WOFpy/pull/124 ... At least the original intent of this issue.

[emiliom]

I'm leaving this open for now. I'd like to go over these discussions next week, to see if there's anything of value for documentation or future discussion ... before closing.