search

MAPC / myschoolcommute

Walk Boston Parent Survey
http://myschoolcommute.org
GNU General Public License v3.0
2 stars 1 forks source link

Pull MAPC's changes to templates, R scripts, and .RData files into the production server #76

Open allthesignals opened 9 years ago

allthesignals commented 9 years ago

MAPC will push changes to this Github repo. These changes should be pulled into the repo on the production server and the the app should be restarted to reflect those changes.

Templates: https://github.com/MAPC/myschoolcommute/tree/master/survey/templates R files: https://github.com/MAPC/myschoolcommute/tree/master/R

allthesignals commented 9 years ago

Hi all, Some changes were made in these commits: 2d79fb8e957db999c7b5478c844ad389d53876ce 2c63dc72987d15326ca502bce4edcda946d6da5a

Is it possible for you to pull these into production, @demiurg? If not, do you have a process for this setup in production, or does it need to be done manually?

Thanks!

Matt

demiurg commented 9 years ago

It's possible, and is manual. However, these commits are on a development branch. In general, so far with this project, master is the production branch. To pull something to production I would merge and test, before pulling on production. Is that a task?

allthesignals commented 9 years ago

I will be happy to merge them into master, but as I cannot get the app running locally, I have not been able to test and was hesitant to merge development changes into master.

On Sun, Feb 8, 2015 at 1:07 PM, Pavel notifications@github.com wrote:

It's possible, and is manual. However, these commits are on a development branch. In general, to me, master is the production branch. To pull something to production I would merge and test, before pulling on production. Is that a task?

— Reply to this email directly or view it on GitHub https://github.com/MAPC/myschoolcommute/issues/76#issuecomment-73422979.

Matt Gardner M.S. Urban and Regional Policy Candidate School of Public Policy and Urban Affairs Northeastern University

allthesignals commented 9 years ago

@demiurg Let me see if I can get the app running locally in a Vagrant box. Then I could test it and do this myself. This was part of the first task in the task order we sent out. Thanks!

demiurg commented 9 years ago

Ok, that makes sense. I didn't want to assume anything. For getting a test site I suggest using a dump of the production database, because it has a unique combination of processed geospatial data time consuming to recreate.

allthesignals commented 9 years ago

@demiurg Thanks for the response - that's a good idea. I'll try to run python manage.py dumpdata on it, although I believe I have to be logged in as the django user on the production server. It'd be nice to "fixturize" the data for development purposes. Alternatively, I will just do a pg_dump and try to restore that locally.

demiurg commented 9 years ago

Unfortunately I don't think it's possible to fixture the type of GIS data that the site relies on, because it is topography and other spatial data relationships that no one has really worked on exporting to text form, let alone json. I suggest pg_dump -Fc and before import, pgrouting and postgis extensions should be installed, preferably of the same version as on the server.

allthesignals commented 9 years ago

@demiurg Hi Pavel. I was able to test the templates, and I merged them into master, along with the R changes.

Separately, I am locally seeing column survey_school.shed_05 does not exist so I wondered if the database restoration was incomplete. It sounds like walkshed generation didn't transfer over correctly, but I couldn't be sure. There is nothing in the pg_restore's error log about it. Additionally, greping the sql dump for some reference to the column returns nothing.

Thanks.

demiurg commented 9 years ago

That seems very odd. One thing that might have prevented the sheds from importing is not having SRID '26986' in the spatial_ref_sys or proj4 epsg file. This is a MA local projection. If it's not present, the column datatype would not have been valid, but that should have generated an error somewhere.

allthesignals commented 9 years ago

@demiurg Interesting - in the production database, \d survey_school does not show a shed_05 column either. I can only think that somehow I'm dumping the wrong database (although local_settings.py points to the right database), or I'm using a version of the Django app that differs from the production version. Checking git log on the production server shows the last commit was in 2011, so it's definitely a different version, although it may not somehow be version controlled. Not sure! Thanks for any insight you are able to provide.

demiurg commented 9 years ago

You could be in wrong directory/database. Directory is: /var/www/saferoutes/myschoolcommute

Database is 'mysc', and mysc=# \d survey_school ... shed_05 | geometry(MultiPolygon,26986) |

allthesignals commented 9 years ago

@demiurg Great. I am up and running. Thanks.

08b6236dfadbdcc3894b39551d4b41d4a63d2bee, and the branch merge a036b360b7c34099113dfdf233d05842e6dc84a1 resolves this issue.

allthesignals commented 9 years ago

@demiurg Hi Pavel. Thanks again for your help. One of the tasks includes pulling changes into production. Would you be able to do this after pulling my changes into your local copy and testing the report generation and batch template's map toggle functionalities? As I mentioned, these changes work on my local copy, but I'd be more comfortable if you could test them as well and push them to production. Is there anything I can do to help you accomplish this?

Thanks!

demiurg commented 9 years ago

Yes, I should be able to test and pull on production thereby testing our test and production environments :)

TimReardon commented 9 years ago

Matt has committed and merged a number of changes to the master branch; tested locally and ready to be pulled into production. Please pull these changes ASAP. Thanks!

allthesignals commented 9 years ago

@demiurg Hi Pavel - I just wanted to check on the status of this issue. Let me know if there are any issues that I can help with. Thanks again!

demiurg commented 9 years ago

Sadly, I'm getting errors related to R: Error: 'rho' must be an environment not NULL: detected in C-level eval * stack smashing detected *: /web/myschoolcommute/venv/bin/python terminated Aborted (core dumped)

demiurg commented 9 years ago

There is a chance things might work in production, I can pull there, test, and revert if the issue persists.

allthesignals commented 9 years ago

@demiurg Hmm can you provide some more context? Is this happening when you run the server? What version of R is being used on the production server? That sounds like maybe an issue with R versions, if you've updated R on your development environment.

Also, I updated an .RData workspace image file because that was how the scripts had been retrieving some data in the past. This is most likely worth refactoring, but that may be related to the issue. I included a backup of the old R/.RData.backup. Try seeing if that using that resolves the issue.

allthesignals commented 9 years ago

From what I've found, this is related to how the python package rpy2 interacts with R itself - usually it's a version mismatch. http://sourceforge.net/p/rpy/mailman/rpy-list/thread/5246F9F1.4090108@gmail.com/. It shouldn't be the RData image file. It's not specified in the requirements.txt, so a pip install -r requirements.txt should cause problems on production.

demiurg commented 9 years ago

I'm looking into building rpy2 2.3.8 but I think I need to install some package with headers. These are not easy to find given the name of the r package.

demiurg commented 9 years ago

Ok, so rpy2 is fixed by apt-get installing r-base r-base-core r-base-dev and pip installing rpy2==2.3.8. More errors to look into.

allthesignals commented 9 years ago

Sounds good. Let me know what issues you run into. If R wasn't installed, I've found that there are some required R packages beyond those listed in the README. Here is the output of my installed.packages(), for reference: https://gist.github.com/allthesignals/881aa68c36a868cf33b4

Additionally, apt-get install texlive-binaries will install what you need for LaTex, if you don't have that already.

allthesignals commented 9 years ago

@demiurg I noticed the accessibility changes have been pulled into production - thanks! I'm seeing a 502 Bad Gateway for the report generation, however. What is the output of the logs? Thanks.

allthesignals commented 9 years ago

Nevermind :) Report-generation seems to be working fine. Thanks again, Pavel.

demiurg commented 9 years ago

It's not working, as in off and on. Something is going on with the email system. sendmail is apparently blocking.

----- The following addresses had permanent fatal errors ----- admin@masaferoutessurvey.org (reason: 550 5.7.1 Your email messages have been blocked by the recipient OR by Trend Micro Email Reputation Service. Contact the recipient or his/her administrator using alternate means to resolve the issue.)

----- Transcript of session follows ----- ... while talking to mail.mapc.org.:

DATA <<< 550 5.7.1 Your email messages have been blocked by the recipient OR by Trend Micro Email Reputation Service. Contact the recipient or his/her administrator using alternate means to resolve the issue. 550 5.1.1 admin@masaferoutessurvey.org... User unknown

This could be bad for any other site on the server as well. In this case, I think the best solution is not to mess with domain records and close ports (8000 is open for some reason which might be one of any random cause for security software to flag the sender as spam host), but to have a valid SMTP server with which django project can be configured. I have used gmail apps in the past, but I'm not sure what is best for MAPC. Basically, the masaferoutessurvey.org MX records need to match the email senders info.

allthesignals commented 9 years ago

@demiurg Hmm - I think our web developer @beechnut has set up our SMTP server with another app before, so we can look into it. I know there were a lot of pain points. Using google apps might be a good temporary solution.

I'm currently seeing production behaving as expected - are you experiencing downtime only intermittently?

Thanks for your help!

demiurg commented 9 years ago

The error seems to happen at a whim of the remote remote relay not accepting the email. It's not happening now, but the error "blocked by the recipient OR by Trend Micro Email Reputation Service." doesn't really indicate why. I was simply trying to test when it started working, so I believe whatever the underlying problem was, did not go away.

allthesignals commented 9 years ago

Hey @demiurg, we fixed an issue with the R files. I committed those changes to master. Could you pull those into production? Also, if you have some time, could you document your guidelines for pull code changes into production (just so that I don't have to bug you to do it). I'm hesitant to do it myself because I'm not sure if there are any custom task scripts involved.

Thanks!

demiurg commented 9 years ago

git pull sudo service saferoutes restart

That's all I typically do, and of course I check that whatever is pulled works. Just pulled.