overhangio / tutor

The Docker-based Open edX distribution designed for peace of mind
https://docs.tutor.overhang.io/
GNU Affero General Public License v3.0
928 stars 444 forks source link

Build failed running pavelib.assets.update_assets: Subprocess return code: 1 #24

Closed natea closed 6 years ago

natea commented 6 years ago

When I try to run paver update_assets cms --settings=development to update the CMS assets, I get the following error. Do I need to run it with USER=1000 make cms instead?

$ make cms
Starting openedx-docker_rabbitmq_1  ... done
Starting openedx-docker_mysql_1     ... done
Starting openedx-docker_smtp_1      ... done
creating new user 'openedx' with UID 501done
openedx@4d121efb7bef:~/edx-platform$ paver update_assets cms --settings=development
---> pavelib.assets.update_assets
---> pavelib.prereqs.install_node_prereqs
test `npm config get registry` = "https://registry.npmjs.org/" || (echo setting registry; npm config set registry https://registry.npmjs.org/)
npm install
npm WARN deprecated formatio@1.1.1: This package is unmaintained. Use @sinonjs/formatio instead
edx@0.1.0 /openedx/edx-platform
`-- edx-ui-toolkit@1.5.2
  `-- sinon@1.17.7
    +-- formatio@1.1.1
    +-- lolex@1.3.2
    `-- samsam@1.1.2

npm WARN optional Skipping failed optional dependency /chokidar/fsevents:
npm WARN notsup Not compatible with your operating system or architecture: fsevents@1.2.3
npm WARN optional Skipping failed optional dependency /karma/chokidar/fsevents:
npm WARN notsup Not compatible with your operating system or architecture: fsevents@1.2.3
npm WARN eslint-config-airbnb-base@11.3.2 requires a peer of eslint-plugin-import@^2.7.0 but none was installed.
npm WARN eslint-config-airbnb-base@3.0.1 requires a peer of eslint@^2.9.0 but none was installed.
npm WARN edx@0.1.0 No repository field.
npm WARN edx@0.1.0 No license field.
xmodule_assets common/static/xmodule
        Finished processing xmodule assets.
mkdir_p path('common/static/common/js/vendor')
Copying vendor files into static directory
/bin/cp -rf node_modules/backbone.paginator/lib/backbone.paginator.js common/static/common/js/vendor
/bin/cp -rf node_modules/backbone/backbone.js common/static/common/js/vendor
/bin/cp -rf node_modules/bootstrap/dist/js/bootstrap.js common/static/common/js/vendor
/bin/cp -rf node_modules/hls.js/dist/hls.js common/static/common/js/vendor
/bin/cp -rf node_modules/jquery-migrate/dist/jquery-migrate.js common/static/common/js/vendor
/bin/cp -rf node_modules/jquery.scrollto/jquery.scrollTo.js common/static/common/js/vendor
/bin/cp -rf node_modules/jquery/dist/jquery.js common/static/common/js/vendor
/bin/cp -rf node_modules/moment-timezone/builds/moment-timezone-with-data.js common/static/common/js/vendor
/bin/cp -rf node_modules/moment/min/moment-with-locales.js common/static/common/js/vendor
/bin/cp -rf node_modules/picturefill/dist/picturefill.js common/static/common/js/vendor
/bin/cp -rf node_modules/requirejs/require.js common/static/common/js/vendor
/bin/cp -rf node_modules/tether/dist/js/tether.js common/static/common/js/vendor
/bin/cp -rf node_modules/underscore.string/dist/underscore.string.js common/static/common/js/vendor
/bin/cp -rf node_modules/underscore/underscore.js common/static/common/js/vendor
Copying developer vendor files into static directory
/bin/cp -rf node_modules/sinon/pkg/sinon.js common/static/common/js/vendor
/bin/cp -rf node_modules/squirejs/src/Squire.js common/static/common/js/vendor
---> pavelib.assets.compile_coffeescript
node_modules/.bin/coffee --compile `find /openedx/edx-platform/lms /openedx/edx-platform/cms /openedx/edx-platform/common -type f -name "*.coffee"`
python manage.py lms --settings=development print_settings STATIC_ROOT --format=value 2>/dev/null

Captured Task Output:
---------------------

---> pavelib.assets.update_assets
---> pavelib.prereqs.install_node_prereqs
test `npm config get registry` = "https://registry.npmjs.org/" || (echo setting registry; npm config set registry https://registry.npmjs.org/)
npm install
xmodule_assets common/static/xmodule
mkdir_p path('common/static/common/js/vendor')
/bin/cp -rf node_modules/backbone.paginator/lib/backbone.paginator.js common/static/common/js/vendor
/bin/cp -rf node_modules/backbone/backbone.js common/static/common/js/vendor
/bin/cp -rf node_modules/bootstrap/dist/js/bootstrap.js common/static/common/js/vendor
/bin/cp -rf node_modules/hls.js/dist/hls.js common/static/common/js/vendor
/bin/cp -rf node_modules/jquery-migrate/dist/jquery-migrate.js common/static/common/js/vendor
/bin/cp -rf node_modules/jquery.scrollto/jquery.scrollTo.js common/static/common/js/vendor
/bin/cp -rf node_modules/jquery/dist/jquery.js common/static/common/js/vendor
/bin/cp -rf node_modules/moment-timezone/builds/moment-timezone-with-data.js common/static/common/js/vendor
/bin/cp -rf node_modules/moment/min/moment-with-locales.js common/static/common/js/vendor
/bin/cp -rf node_modules/picturefill/dist/picturefill.js common/static/common/js/vendor
/bin/cp -rf node_modules/requirejs/require.js common/static/common/js/vendor
/bin/cp -rf node_modules/tether/dist/js/tether.js common/static/common/js/vendor
/bin/cp -rf node_modules/underscore.string/dist/underscore.string.js common/static/common/js/vendor
/bin/cp -rf node_modules/underscore/underscore.js common/static/common/js/vendor
/bin/cp -rf node_modules/sinon/pkg/sinon.js common/static/common/js/vendor
/bin/cp -rf node_modules/squirejs/src/Squire.js common/static/common/js/vendor
---> pavelib.assets.compile_coffeescript
node_modules/.bin/coffee --compile `find /openedx/edx-platform/lms /openedx/edx-platform/cms /openedx/edx-platform/common -type f -name "*.coffee"`
python manage.py lms --settings=development print_settings STATIC_ROOT --format=value 2>/dev/null

Build failed running pavelib.assets.update_assets: Subprocess return code: 1
natea commented 6 years ago

Even after using USERID=1000 make cms and re-running the update_assets paver command, I still get the same error.

regisb commented 6 years ago

This is a case where /dev/null eats the actual error. Can you try to run python manage.py lms --settings=development print_settings STATIC_ROOT --format=value to find the real error?

natea commented 6 years ago

Here's the output when I run that command:

root@dec35919721e:/openedx/edx-platform# python manage.py lms --settings=development print_settings STATIC_ROOT --format=value
Traceback (most recent call last):
  File "manage.py", line 111, in <module>
    startup = importlib.import_module(edx_args.startup)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/openedx/edx-platform/lms/startup.py", line 12, in <module>
    settings.INSTALLED_APPS  # pylint: disable=pointless-statement
  File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 48, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 44, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 92, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named development

Since I used the EDX_PLATFORM_VERSION environment variables, why when I run make lms is it showing that the edx-platform repo is using the one from edX?

Did I forget a re-build step along the way?

root@dec35919721e:/openedx/edx-platform/lms/envs# git remote -v
origin  https://github.com/edx/edx-platform.git (fetch)
origin  https://github.com/edx/edx-platform.git (push)
root@dec35919721e:/openedx/edx-platform/lms/envs# git branch
* open-release/ginkgo.master

But even if I change it to use my fork, I forgot to check in the two development.py files into my forked repo, so even after I change the repo, it's still not going to find the development.py file.

regisb commented 6 years ago

There is a misunderstanding: EDX_PLATFORM_VERSION is a docker argument, not an environment variable. If you wish to define a different EDX_PLATFORM_VERSION then you need to modify the docker-compose.yml file accordingly (here is the documentation). Instructions about this are purposedly scarce in the README because it's a rather advanced feature.

EDIT: and of course, you would need to add a development.py module in the lms/envs and cms/envs folder of your repo to get it to work in development mode.

natea commented 6 years ago

Sorry, my memory failed me. I meant to type EDX_PLATFORM_REPOSITORY, Which should be my forked repo URL which should match the repo that is checked out to EDX_PLATFORM_PATH, right?

regisb commented 6 years ago

EDX_PLATFORM_REPOSITORY, just like EDX_PLATFORM_VERSION, is also a docker argument, not a docker environment variable. As such, it is only useful during build, not at runtime. Thus, there is no need to define these two arguments when you try to run make lms or make cms.

I realise now that giving instructions about image building is confusing. In the near future images will be downloadable from hub.docker.com and you won't have to build anything locally yourself.

natea commented 6 years ago

Ok, now I'm very confused because 2 days ago you said that these variables are not used during build, only at runtime, and now you're saying that they only useful during build, not at runtime. :/

Can you help me understand which variables are used during build and which are used during runtime?

regisb commented 6 years ago

Ok, my comment was confusing:

EDX_PLATFORM_PATH and EDX_PLATFORM_SETTING are environment variables used at runtime only EDX_PLATFORM_REPOSITORY and EDX_PLATFORM_VERSION are docker args used at build only

natea commented 6 years ago

ok, that makes sense now! Thanks :)

natea commented 6 years ago

@regisb one question since I'm new to using a volume on my mac for edx-platform but having the actual runtime in a Docker container.

Under what circumstances would I need to rebuild the images? If I want to install a new XBlock or add something that has to go into the Python environment then I would need to rebuild the image, whereas if I'm just making SASS or template changes, then that would not require the rebuilding of the image?

regisb commented 6 years ago

Since we are talking about docker volumes, I assume you mean working on Open edX during development.

If I want to install a new XBlock or add something that has to go into the Python environment then I would need to rebuild the image

Correct. Unless the requirement is in the edx-platform repo itself, such as the requirements from local.txt. Another option is to reinstall the requirement every time you start the container, but that is just painful.

if I'm just making SASS or template changes, then that would not require the rebuilding of the image?

No, it would not, because asset collection is not part of the build process.