re-structuring cristi.crossculturalconsult.com

[cccs-ip]

Greetings, Team.

As of yesterday, Cristi switched development over to a local test environment. He no longer needs an independent server for development and testing.

I would like to close the server that is currently hosting cristi.crossculturalconsult.com. I would instead like a new setup where the URL cristi.crossculturalconsult.com is hosted on the main CCCS web server, and that URL is automatically updated (via continuous integration) each time that Cristi pushes his local commits to his development branch.

Are we at (or near) a point where this is possible?

[pwhipp]

Yes we are. I'll need to set the site up (~ 30m) and then, after I've got Jenkins working, set up the CI for it (probably less than an hour).

Getting Jenkins working will be the biggie but it is already on the list for the new server and I've done it once now so an hour or two should see it sorted.

[cccs-ip]

Cool. I realize you have other work today; there's no particular rush. Cristi will be ready for the new set-up on Tuesday or thereafter.

[cccs-ip]

Please go ahead with this work as you have time.

[pwhipp]

Cristi, you appear to have deleted your working copy and its related virtualenv from the server. The private key needed to access github is also missing.

Shell in to the new server (ssh crossculturalconsult.com) and put your private key back (the one matching your github id). Use git clone git@github.com:cccs-web/core.git cccs to verify you've set it correctly.

Once done, assign this issue back to me.

I'll setup the virtualenv, nginx and upstart configurations for you.

You can use this to show your work to Aaron as before. Don't hesitate to let me know if you need help with deployment. I've given you the necessary privileges so a catch all deployment would be as follows:

1. shell in
2. switch to the virtualenv workon cccs
3. Check the branch git status (you will probably want to use the css branch but you can use whatever you want to show to Aaron).
4. Pull new code git pull
5. Collect static files (includes changed js files) django collectstatic

If more than the javascript has changed - it may well do, with master updates which are essential - (the git pull response gives you the info needed to know), you may also need to:

1. sudo service cristi_cccs restart

I suggest that you routinely pull and merge master into your local working branches regularly and always do so just before deploying. When you do this locally, if you see models change, you may need to invoke django migrate to bring your local copy of the database up to date.

Your online copy is using the production database so it presents the latest information: A failure to keep your working copy up to date with master could result in confusing errors relating to database updates.

Once we're done and Aaron points cristi.crossculturalconsult.com at the new server, this should all 'just work'. Fingers crossed ;)

[cccs-ip]

I switched over the DNS, but it might take some time before the change goes into effect.

Paul, to be clear:

the branch css is currently the one that cristi.crossculturalconsullt.com is showing, correct? Please only work on that branch for the design; let's keep both 'staging' and 'production' as they are for the moment.

Also, for changes to show on 'cristi.crossculturalconsult.com', one must first connect to the sever to pull in changes from css, right?

thanks everyone!

[pwhipp]

When it is set up, the service points at the working copy. Any working copy can easily switch branch with a single command so the branch is something that can very easily change.

So.... yes I intend to leave it pointing at the css branch but it could be made the same as live with:

git checkout master
sudo service cristi_cccs restart

Or any other branch. This is very handy when reviewing test versions etc. but it does mean that it can be hard to know which branch is being served... I could add an admin utility that would tell you. I'm not sure how long this would take (I could include deployment there rather than using jenkins - added as #135)

[cccs-ip]

Thanks, Paul. No need to worry about the utility now. Cristi: Turning this one over to you...

[cccs-ip]

Hi again, Paul. I just double-checked. You must have accidentally been looking in the production server instead of at cristi.crossculturalconsult.com. You may recall that I had sent you a key for the ubuntu user on this server some time ago, and you appear to have added in your own ssh records to your user. Anyway, all of this is to say that cristi's materials were indeed there.

I moved over his ssh credentials (he's lost a few copies of his keys since we set up the production server), so this should give Cristi back is connection to git.

Cristi: Your keys are password protected, so we can't use them to push and pull on your behalf. To allow us to do so, you can add in a new key records specifically for this site to your GitHub account cccs-web/core. This key would be used only by your CCCS server user, and would be for the purposes of tracking your activities back to your GitHub user. It would be helpful to have this option available so that I can push and pull on your behalf.

[cccs-ip]

Hi again, Paul.

I noticed there were absolutely no folders under user cristi on our re-deloyed server. Your instructions seem to imply that he should at least have a virtualenv set up?

It doesn't appear so:

cristi:~/.ssh $ workon cccs
ERROR: Environment 'cccs' does not exist. Create it with 'mkvirtualenv cccs'.

Are your instructions meaning to say that we have to re-run the entire install process for Crisiti? If so, do we have to do this as his user, or can't we just set it up for him and let him add in his keys later?

Also, I assume we can make it so user 'paul' and user 'aaron' can work on user the 'cristi' files without having to sudo -iu cristi, correct? Should he be on a virtualenv called cccs in that case? Why not call his virtualenv something else, like cristi or css or design ? I mean, can't user 'aaron' workon production or workon staging?

[cccs-ip]

Cristi:

I created a virtualenv for you called css (to match the name of the branch). I don't have your id_rsa password, so I can't run the git clone command for the next step (and I don't want to break anything).

Can you please ssh into your user on crosscuturalconsult.com, and do the following:

1. workon css
2. git clone git@github.com:cccs-web/core.git css

... After that I will try to help finish the migration. This is the first time I have attempted to set up a branch, so fingers crossed!

[cccs-ip]

Hi again, Paul.

Cristi has a CCCS-specific key linked to GitHub. We were able to clone in our repo:

 git clone git@github.com:cccs-web/core.git css
Cloning into 'css'...
The authenticity of host 'github.com (192.30.252.130)' can't be established.
RSA key fingerprint is 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'github.com,192.30.252.130' (RSA) to the list of known hosts.
remote: Counting objects: 2171, done.
remote: Compressing objects: 100% (166/166), done.
remote: Total 2171 (delta 57), reused 0 (delta 0)
Receiving objects: 100% (2171/2171), 11.63 MiB | 13.65 MiB/s, done.
Resolving deltas: 100% (854/854), done.
Checking connectivity... done.

The install, however, failed due to a permissions issue:

error: could not create '/usr/local/lib/python2.7/dist-packages/psycopg2': Permission denied

----------------------------------------
Cleaning up...
Command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip_build_cristi/psycopg2/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-d0BQ7l-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip_build_cristi/psycopg2
Storing debug log for failure in /home/cristi/.pip/pip.log

Please help us to finish the installation / migration of cristi's site by setting up branch css to show at http://cristi.crossculturalconsult.com

FYI: his previous instance is still up and running. You can ssh into that server from the following domain:

http://teardown.crossculturalconsult.com

... I doubt you'll need anything from within it.

[cccs-ip]

further notes: I went ahead added in a secrets.py file for cristi, and set the virtaulenv for css to link to that directory. We still need to:

1. ensure the installation is working
2. link in the cristi user to an appropriate database [I didn't want to mess with this because I don't know how to appropriately set the database up for a specific branch]
3. configure nginx (I copied over the nginx file from the old cristi server to sites-available .. in case it saves any time)

[pwhipp]

Eek... did I really miss the files on the old server? I'm sorry if so - I was using direct IP addresses to avoid the risk of that.

The permission error looks like you are trying to pip install on the main server (not in a virtualenv). I'm looking into this now...

[pwhipp]

FYI as the record:

1. Switched the css working copy to the css branch
2. Renamed the working copy to 'cccs_web' so it is consistent with the existing deploy scripts.
3. Removed the css virtualenv and created cccs_web (to match the working copy it goes with)
4. Installed the requirements into the cccs_web environment
5. Collected the staticfiles
6. Removed /etc/nginx/cristi
7. Linked ln -s /home/cristi/cccs_web/deploy/cristi/nginx.conf /etc/nginx/sites-available/cristi_cccs_web
8. Linked ln -s /etc/nginx/sites-available/cristi_cccs_web /etc/nginx/sites-enabled/cristi_cccs_web
9. reloaded nginx
10. copied upstart cp /home/cristi/cccs_web/deploy/cristi/upstart.conf /etc/init/cristi_cccs_web.conf
11. start the service sudo service cristi_cccs_web start
12. restart the service to check - oops service must have exited so there is a problem
13. review deploy documents:

The nginx.conf is looking at port 8000. We use that one already so this is not good. We got away with this on the old server because we did not go live with the crossculturalconsult site. I've changed the port for cristi's site to 9482.

1. try restarting upstart but still not launching. Continue review of deploy docs... Old virtual env was called 'cccs'. New one is cccs_web. It would be all better called cccs which is what it is so I renamed it to 'cccs' to limit future confusion.
2. pulled updates
3. retried service. failed to start - oops... missed a cccs_web reference.

OK. Cristi's stuff should be good now but something strange is going on with the nginx routing...

[cccs-ip]

Thanks, Paul. Could we not, however, revise the name to css ? This would be consistent with the use of the names production and staging under user cccs.

You had dropped my web-core designations, etc, when migrating the new server, so I would expect it is possible to use css to keep with the branch naming conventions?

Moreover... there was no need to "transfer" materials from Cristi's old site, was there? All we needed to do was to clone the current css branch?

I am still not sure--either from this message or from the documentation on staging and production--how one is supposed to deal with access to its respective database....

[cccs-ip]

Also, with regard to the name of the virtualenv, I still don't see what we are using the name cccs intead of css. Again, for user CCCS, the names of the virtualenv are production and staging...

[cccs-ip]

With regard to point 2, please also clarify what you mean by

consistent with the existing deploy scripts.

We have installed local copies of this site under several names (again, noting 'production' and 'staging', but also as 'cccs' on one of my local machines). I don't know what "deploy scripts" you are referring to with which the name css would be inconsistent.

[cccs-ip]

With regard to "the direct IP address", that was the source of your error. Cristi was on his own server. I never informed you of its direct IP address.

[pwhipp]

All sorted now.