Various errors (and workarounds) upgrading from Birch to Cypress

[fisharebest]

I am upgrading Birch to Cypress, using http://edx.readthedocs.org/projects/edx-installing-configuring-and-running/en/latest/cypress.html#upgrading-from-birch-to-cypress

Here are the errors that I encountered, and how I got past them. Run these before starting the upgrade.

# Fixes "fatal: Unable to create '/edx/app/forum/.rbenv/.git/index.lock': Permission denied"
sudo chown -R forum:www-data /edx/app/forum

# Fixes "msg: Failed to validate the SSL certificate for launchpad.net:443"
cd /var/tmp/configuration
sudo sed -i -e 's/ansible==1.5.5/ansible==1.6.1/' requirements.txt
sudo pip install -r requirements.txt

# Fixes "msg: file (/etc/update-motd.d/51-cloudguest) is absent, cannot continue"
sudo touch /etc/update-motd.d/51-cloudguest

# Fixes "msg: devpi-server: ERROR (already started)"
sudo service supervisor.devpi stop

Still stuck on #2249...

# Fixes "ImportError: no module named _markerlib"
sudo -u edxapp rm -rf /edx/app/edxapp/venvs/edxapp

[dagg]

@fisharebest Thank you for posting this here, it was really helpful and up to a point I had the exact same errors, but after the "msg: devpi-server: ERROR (already started)" I came upon another error and I wonder if you or someone could give me some help because I'm totally stuck there! The error is the following:

TASK: [edxapp | syncdb and migrate] ******************************************* 
failed: [localhost] => (item=lms) => {"changed": true, "cmd": ["/edx/bin/edxapp-syncdb-lms"], "delta": "0:00:00.476233", "end": "2015-12-02 18:26:08.557426", "item": "lms", "rc": 1, "start": "2015-12-02 18:26:08.081193"}
stderr: Traceback (most recent call last):
  File "manage.py", line 111, in <module>
    startup = importlib.import_module(edx_args.startup)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/edx/app/edxapp/edx-platform/lms/startup.py", line 9, in <module>
    settings.INSTALLED_APPS  # pylint: disable=pointless-statement
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/conf/__init__.py", line 48, in __getattr__
    self._setup(name)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/conf/__init__.py", line 44, in _setup
    self._wrapped = Settings(settings_module)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/conf/__init__.py", line 113, in __init__
    raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.
failed: [localhost] => (item=cms) => {"changed": true, "cmd": ["/edx/bin/edxapp-syncdb-cms"], "delta": "0:00:00.486013", "end": "2015-12-02 18:26:09.087089", "item": "cms", "rc": 1, "start": "2015-12-02 18:26:08.601076"}
stderr: Traceback (most recent call last):
  File "manage.py", line 111, in <module>
    startup = importlib.import_module(edx_args.startup)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/edx/app/edxapp/edx-platform/cms/startup.py", line 8, in <module>
    settings.INSTALLED_APPS  # pylint: disable=pointless-statement
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/conf/__init__.py", line 48, in __getattr__
    self._setup(name)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/conf/__init__.py", line 44, in _setup
    self._wrapped = Settings(settings_module)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/conf/__init__.py", line 113, in __init__
    raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.

FATAL: all hosts have already failed -- aborting

PLAY RECAP ******************************************************************** 
           to retry, use: --limit @/home/dagg/vagrant-fullstack.retry

localhost                  : ok=224  changed=52   unreachable=0    failed=1 

Any ideas on how to overcome this?

Thank you!

[fisharebest]

@dagg - if it helps, my final birch-to-cypress script looks like this:

sudo touch /etc/update-motd.d/51-cloudguest
cd /var/tmp/configuration
sudo sed -i -e 's~ansible==1.5.5~git+https://github.com/edx/ansible.git@stable-1.9-plus-edx#egg=ansible==1.9.1-edx.2~' requirements.txt
sudo pip install --upgrade -r requirements.txt
sudo -u edxapp rm -rf /edx/app/edxapp/venvs/edxapp
cd
wget https://raw.githubusercontent.com/edx/configuration/master/util/vagrant/migrate.sh
chmod 755 migrate.sh
./migrate.sh -y -c fullstack

I used this to upgrade several servers. (I also needed to increase the memory on the server from 2GB to 4GB.)

As for your SECRET_KEY error, the command find /edx/etc -type f -exec grep SECRET_KEY {} + should show you where it is being defined.

[dagg]

@fisharebest Thank you once again!

I run your birch-to-cypress script but I got the SECRET_KEY error again, and I looked for the "SECRET_KEY" parameter, it is in the following files: /edx/etc/edxapp/cms.auth.yaml and /edx/etc/edxapp/lms.auth.yaml and it is indeed empty, for example in lms.auth.yaml it is in 2 places: SECRET_KEY: '' and under "CC_PROCESSOR: CyberSource2:" : CC_PROCESSOR: CyberSource2: SECRET_KEY: ''

I found the "/edx/app/edxapp/venvs/edxapp/lib/python2.7/site-packages/django_extensions/management/commands/generate_secret_key.py" but this is supposed to be used from Django itself to produce a secret key.

I put some random key manually at the above yaml files but it didn't work! Do you have any idea why is this happening and maybe a possible workaround?

Again, your help was quite valuable already, and I thank you very much for your time!

[fisharebest]

Do you have any idea why is this happening and maybe a possible workaround?

Sorry - no idea. I've never actually used EDX, and I'm not a python programmer - just the sysadmin who needs to install it...

[dagg]

@fisharebest Thank you anyway for your help :)

[fisharebest]

Another upgrade from birch to cypress and another error:

fatal: Unable to create '/edx/app/forum/.rbenv/.git/index.lock': Permission denied

The migration script is trying to run sudo -u forum git -C /edx/app/forum/.rbenv reset --hard but this folder is owned by www-user, not forum. Can be fixed by sudo chown -R forum:www-data /edx/app/forum.

The next error is

TASK: [rbenv | install build depends] *****************************************
ok: [localhost] => (item=curl,build-essential,libcurl4-openssl-dev,libreadline-dev,libssl-dev,libxml2-dev,libxslt1-dev,zlib1g-dev)

TASK: [rbenv | update rbenv repo] *********************************************
failed: [localhost] => {"failed": true}
msg: Failed to checkout v0.4.0

FATAL: all hosts have already failed -- aborting

Any suggestions anyone?

[yrchen]

@fisharebest Try to remove or rename the directory of forum /edx/app/forum and let script to recreate it. It works on my environment.