wranglerr commented 5 years ago

Summary

This changeset consist of:

Creating a fabfile.py to automate the deployment of local source code to a remote server.
Convert the existing file structure into a Python package for easier installation. (Still not installing as a package, that will come in a later PR).
Update the configuration files so they pull SMTP secrets from environment variables
Update path names to follow snake_case convention over TitleCasing.

Dev test

Deploying the server code

╰ pipenv run python fabfile.py all -h ubuntu@ec2-52-61-205-101.us-gov-west-1.compute.amazonaws.com  -d /srv/WindNinjaServer         
3.7.3 (default, Mar 26 2019, 21:43:19) 
[GCC 8.2.1 20181127] /home/levi/.local/share/virtualenvs/WindNinja-Server-eDaQoBku/bin/python
Creating tarball of windninja_server..
Successfully created tarball
Deploying windninja server ubuntu@ec2-52-61-205-101.us-gov-west-1.compute.amazonaws.com
Copying windninja_server_2019_06_27_05_08_11.tar.gz to on ec2-52-61-205-101.us-gov-west-1.compute.amazonaws.com
Successfully coped and exactracted windninja_server source.
copying windninjaweb folder: /tmp/windninja_server/windninjaweb to /srv/WindNinjaServer/app/windninjaweb
copying windninjaqueue folder: /tmp/windninja_server/windninjaqueue to /srv/WindNinjaServer/app/windninjaqueue
copying windninjaconfig folder: /tmp/windninja_server/windninjaconfig to /srv/WindNinjaServer/app/windninjaconfig
copying windninjawrapper folder: /tmp/windninja_server/windninjawrapper to /srv/WindNinjaServer/app/windninjawrapper
copying runqueue.py file: /tmp/windninja_server/runqueue.py to /srv/WindNinjaServer/app
copying ../setup.py file: /tmp/windninja_server/../setup.py to /srv/WindNinjaServer/app
creating application folder: /srv/WindNinjaServer/app
copying config file: /tmp/windninja_server/windninjaserver.config.yaml to /srv/WindNinjaServer/app
creating job wrapper folder: /srv/WindNinjaServer/app/windninjawrapper
copying config file: /tmp/windninja_server/windninjawrapper/windninjawrapper.config.yaml to /srv/WindNinjaServer/app/windninjawrapper
creating datastore folder: /srv/WindNinjaServer/data/account
creating datastore folder: /srv/WindNinjaServer/data/feedback
creating datastore folder: /srv/WindNinjaServer/data/job
creating datastore folder: /srv/WindNinjaServer/data/queue
creating datastore folder: /srv/WindNinjaServer/data/notification
copying test job folder: /tmp/windninja_server/../data/dem to /srv/WindNinjaServer/data/dem
copying test job folder: /tmp/windninja_server/../data/job/23becdaadf7c4ec2993497261e63d813 to /srv/WindNinjaServer/data/job/23becdaadf7c4ec2993497261e63d813
copying test job folder: /tmp/windninja_server/../data/job/1a111111111111111111111111111111 to /srv/WindNinjaServer/data/job/1a111111111111111111111111111111
copying apache config file: /tmp/windninja_server/windninjaweb/apache/WindNinjaApp.conf to /etc/apache2/sites-available
copying supervisor config file: /tmp/windninja_server/windninjaqueue/supervisor/WindNinjaApp.conf to /etc/supervisor/conf.d
Reloading apache and supervisor
Site WindNinjaApp already enabled
 * Reloading web server apache2
 * 
Restarted supervisord
wnqueue                          STARTING  
complete!

Confirming the application is running

[2019-06-27T11:35:24.727835]:enqueue job: 1a111111111111111111111111111111
DEBUG   windninja.py    windninja run as main
DEBUG   windninja.py    windninja.main()
DEBUG   windninja.py    Namespace(id='1a111111111111111111111111111111', log_level='debug', pretty_print=True)
INFO    windninja.py    Begin - version 2019.02.21.1
DEBUG   windninja.py    project path: /srv/WindNinjaServer/data/job/1a111111111111111111111111111111
DEBUG   model.py        job json path: /srv/WindNinjaServer/data/job/1a111111111111111111111111111111/job.json
DEBUG   model.py        Job file string:
{
   "status":"new",
   "account":"test@yourdatasmarter.com",
   "name":"Point Six (test 1a)",
   "messages":[
      "2015-02-27T16:48:45.2949952-07:00 | INFO | job created"
   ],
   "id":"1a111111-1111-1111-1111-111111111111",
   "output":{},
   "input":{
      "domain":{
         "xmin":-114.0235465406869,
         "ymin":46.99526210560205,
         "ymax":47.038565315467025,
         "xmax":-113.97925790543299
      },
      "products":"vector:true;raster:false;topofire:false;geopdf:false;clustered:true;weather:true",
      "parameters":"forecast_duration:12;vegetation:trees;mesh_choice:fine",
      "forecast":"NOMADS-NAM-CONUS-12-KM"
   },
   "email":""
}

DEBUG   model.py        Job: {u'status': u'new', u'account': u'test@yourdatasmarter.com', u'name': u'Point Six (test 1a)', u'messages': [u'2015-02-27T16:48:45.2949952-07:00 | INFO | job created'], u'email': u'', u'input': {u'domain': {u'xmin': -114.0235465406869, u'ymin': 46.99526210560205, u'ymax': 47.038565315467025, u'xmax': -113.97925790543299}, u'products': u'vector:true;raster:false;topofire:false;geopdf:false;clustered:true;weather:true', u'parameters': u'forecast_duration:12;vegetation:trees;mesh_choice:fine', u'forecast': u'NOMADS-NAM-CONUS-12-KM'}, u'output': {}, u'id': u'1a111111-1111-1111-1111-111111111111'}
INFO    model.py        Initializing WindNinja Run
DEBUG   model.py        Job: {u'status': 'executing', u'account': u'test@yourdatasmarter.com', u'name': u'Point Six (test 1a)', u'messages': [u'2015-02-27T16:48:45.2949952-07:00 | INFO | job created', '2019-06-27T11:35:25.120879 | info | Initializing WindNinja Run'], u'email': u'', u'input': {u'domain': {u'xmin': -114.0235465406869, u'ymin': 46.99526210560205, u'ymax': 47.038565315467025, u'xmax': -113.97925790543299}, u'products': u'vector:true;raster:false;topofire:false;geopdf:false;clustered:true;weather:true', u'parameters': u'forecast_duration:12;vegetation::
DEBUG   model.py        Job: {u'status': u'new', u'account': u'test@yourdatasmarter.com', u'name': u'Point Six (test 1a)', u'messages': [u'2015-02-27T16:48:45.2949952-07:00 | INFO | job crea
ted'], u'email': u'', u'input': {u'domain': {u'xmin': -114.0235465406869, u'ymin': 46.99526210560205, u'ymax': 47.038565315467025, u'xmax': -113.97925790543299}, u'products': u'vector:true;r
aster:false;topofire:false;geopdf:false;clustered:true;weather:true', u'parameters': u'forecast_duration:12;vegetation:trees;mesh_choice:fine', u'forecast': u'NOMADS-NAM-CONUS-12-KM'}, u'output': {}, u'id': u'1a111111-1111-1111-1111-111111111111'}
INFO    model.py        Initializing WindNinja Run
DEBUG   model.py        Job: {u'status': 'executing', u'account': u'test@yourdatasmarter.com', u'name': u'Point Six (test 1a)', u'messages': [u'2015-02-27T16:48:45.2949952-07:00 | INFO | job created', '2019-06-27T11:35:25.120879 | info | Initializing WindNinja Run'], u'email': u'', u'input': {u'domain': {u'xmin': -114.0235465406869, u'ymin': 46.99526210560205, u'ymax': 47.038565315467025, u'xmax': -113.97925790543299}, u'products': u'vector:true;raster:false;topofire:false;geopdf:false;clustered:true;weather:true', u'parameters': u'forecast_duration:12;vegetation:trees;mesh_choice:fine', u'forecast': u'NOMADS-NAM-CONUS-12-KM'}, u'output': {}, u'id': u'1a111111-1111-1111-1111-111111111111'}
DEBUG   windninja.py    evaluate project forecast: NOMADS-NAM-CONUS-12-KM
DEBUG   gis.py  dem center: [-114.00140222305994, 47.01691371053454]
DEBUG   gis.py  using utm zone: 32611
DEBUG   gis.py  input dem path: /srv/WindNinjaServer/data/dem/windninja_dem.vrt
DEBUG   gis.py  output dem path: /srv/WindNinjaServer/data/job/1a111111111111111111111111111111/wncli/dem.tif
DEBUG   utility.py      Shell command:gdalwarp -tr 85 85 -t_srs epsg:32611 -te 726299 5208938 729480 5213878 /srv/WindNinjaServer/data/dem/windninja_dem.vrt /srv/WindNinjaServer/data/job/1a111111111111111111111111111111/wncli/dem.tif -overwrite
DEBUG   utility.py      Environ: None
DEBUG   utility.py      Shell process result:
Return Code:0
StdOut:Creating output file that is 37P x 58L.
Processing input file /srv/WindNinjaServer/data/dem/windninja_dem.vrt.
Using internal nodata values (e.g. -32768) for image /srv/WindNinjaServer/data/dem/windninja_dem.vrt.
Copying nodata values from source /srv/WindNinjaServer/data/dem/windninja_dem.vrt to destination /srv/WindNinjaServer/data/job/1a111111111111111111111111111111/wncli/dem.tif.
0...10...20...30...40...50...60...70...80...90...100 - done.

StdErr:
INFO    model.py        DEM created
DEBUG   model.py        Job: {u'status': 'executing', u'account': u'test@yourdatasmarter.com', u'name': u'Point Six (test 1a)', u'messages': [u'2015-02-27T16:48:45.2949952-07:00 | INFO | job created', '2019-06-27T11:35:25.120879 | info | Initializing WindNinja Run', '2019-06-27T11:35:25.193766 | info | DEM created'], u'email': u'', u'input': {u'domain': {u'xmin': -114.0235465406869, u'ymin': 46.99526210560205, u'ymax': 47.038565315467025, u'xmax': -113.97925790543299}, u'products': u'vector:true;raster:false;topofire:false;geopdf:false;clustered:true;weather:true', u'parameters': u'forecast_duration:12;vegetation:trees;mesh_choice:fine', u'forecast': u'NOMADS-NAM-CONUS-12-KM'}, u'output': {}, u'id': u'1a111111-1111-1111-1111-111111111111'}
DEBUG   wncli.py        WN CLI output folder: /srv/WindNinjaServer/data/job/1a111111111111111111111111111111/wncli/output
DEBUG   utility.py      Shell command:foamDotFile=$FOAM_INST_DIR/OpenFOAM-2.2.x/etc/bashrc && [ -f $foamDotFile ] && . $foamDotFile ; /usr/local/bin/WindNinja_cli --output_path /srv/WindNinjaServer/data/job/1a111111111111111111111111111111/wncli/output --elevation_file /srv/WindNinjaServer/data/job/1a111111111111111111111111111111/wncli/dem.tif --mesh_choice fine --num_threads 4 --vegetation trees --time_zone auto-detect --initialization_method wxModelInitialization --wx_model_type NOMADS-NAM-CONUS-12-KM --forecast_duration 12 --output_speed_units mph --output_wind_height 20.0 --units_output_wind_height ft --write_shapefile_output true --write_wx_model_shapefile_output true --write_goog_output false --write_ascii_output true --momentum_flag True
DEBUG   utility.py      Environ: {'FOAM_INST_DIR': '/home/ubuntu/src/'}
[2019-06-27T11:35:29.735712]:Jobs - running: 1 ; available: 1; pending:0

Before deploying to production

[x] Add AWS_SMTP_KEY, AWS_SMTP_SECRET, AWS_SMTP_HOST as environment variables on production host

Follow-ups

Add unit testing support with pytest
Update fabfile to install all system dependencies

fspataro-zz commented 5 years ago

@ldmalott Thanks! This is great. I'm out of the office for the next week or so. I'll review with the team when I get back. I don't see any immediate reason why we wouldn't want to go with this approach.

wranglerr commented 5 years ago

@ldmalott Thanks! This is great. I'm out of the office for the next week or so. I'll review with the team when I get back. I don't see any immediate reason why we wouldn't want to go with this approach.

Thanks @fspataro. Any updates on the review? I have quite a few changes for adding and improving the unit tests. I want to hold off on putting those into review until this PR goes through.

nwagenbrenner commented 5 years ago

@ksshannon

ksshannon commented 5 years ago

Looks like better packaging/dependency management, so LGTM

nwagenbrenner commented 5 years ago

I agree we can merge this PR. Who will handle the merge? @fspataro?

wranglerr commented 5 years ago

@fspataro I deployed these changes to the development server. I'm unable to run jobs against it now. Looks like the apache server is not running the Flask app. I'm kind of at a loss since there are no log entries in /var/log/apache2/access.log or /var/log/apache2/error.log. I've been trying to instrument more logging, but have been unsuccessful in moving forward.

Are you able to log into the development server and assist? Are there commands I missed that need to be executed?

fspataro-zz commented 5 years ago

@ldmalott sorry for the delay, just getting back to the office after vacation, crazy busy.... sounds like this got resolved.

firelab / windninja-mobile

Turn server into python package #165

Summary

Dev test

Deploying the server code

Confirming the application is running

Before deploying to production

Follow-ups