Memex-explorer install error

[jxu586]

running ./app_setup.sh, one command a time.

command: conda env update --file environment.yml Fetching package metadata: .............. Error: No packages found in current linux-32 channels matching: elasticnutch

Then run: npm install -g log.io npm http GET https://registry.npmjs.org/log.io/log.io

npm ERR! Error: failed to fetch from registry: log.io npm ERR! at /usr/share/npm/lib/utils/npm-registry-client/get.js:139:12 npm ERR! at cb (/usr/share/npm/lib/utils/npm-registry-client/request.js:31:9) npm ERR! at Request._callback (/usr/share/npm/lib/utils/npm-registry-client/request.js:136:18) npm ERR! at Request.callback (/usr/lib/nodejs/request/main.js:119:22) npm ERR! at Request. (/usr/lib/nodejs/request/main.js:212:58) npm ERR! at Request.emit (events.js:88:20) npm ERR! at ClientRequest. (/usr/lib/nodejs/request/main.js:412:12) npm ERR! at ClientRequest.emit (events.js:67:17) npm ERR! at HTTPParser.onIncoming (http.js:1261:11) npm ERR! at HTTPParser.onHeadersComplete (http.js:102:31) npm ERR! You may report this log at: npm ERR! http://bugs.debian.org/npm npm ERR! or use npm ERR! reportbug --attach /home/jxu586/memex-explorer/source/npm-debug.log npm npm ERR! npm ERR! System Linux 3.13.0-63-generic npm ERR! command "node" "/usr/bin/npm" "install" "-g" "log.io" npm ERR! cwd /home/jxu586/memex-explorer/source npm ERR! node -v v0.6.12 npm ERR! npm -v 1.1.4 npm ERR! message failed to fetch from registry: log.io npm ERR! npm ERR! Additional logging details can be found in: npm ERR! /home/jxu586/memex-explorer/source/npm-debug.log npm not ok

[ahmadia]

Thanks for the report @jxu586. I've gone ahead and made a fix in master (that disables that package). Can you try pulling and installing again?

[jxu586]

@ahmadia Hi, I pulled the new version (Ps: I am running on Ubuntu 12.04, 32bit) Then run ./app_setup.sh, Now I have this error. Error: No packages found in current linux-32 channels matching: elasticnutch ~/memex-explorer/source ~/memex-explorer/source/memex ~/memex-explorer/source ~/memex-explorer/source Error: could not find environment: memex Traceback (most recent call last): File "manage.py", line 8, in from django.core.management import execute_from_command_line ImportError: No module named django.core.management Error: No environment to deactivate

I think the problem is:

1. No packages found in current linux-32 channels matching: elasticnutch
2. Conda environment for memex is not created.

[ahmadia]

@jxu586 - We don't support 32-bit Linux. Did you grab the wrong anaconda/miniconda installer or are you truly on a 32-bit system?

[jxu586]

@ahmadia Yes, I am on a 32-bit system. Maybe that's why it doesn't work.Thanks.

[ahmadia]

Sorry. It should theoretically still be possible for you to get Memex Explorer going on a 32-bit system, but you'll need to build any packages yourself that we don't provide.

[antrromet]

@ahmadia Just to continue from above, I am on a mac. And I too pulled the code, just a couple minutes back and I get the following errors.

Installing collected packages: sphinx, sphinx-rtd-theme, billiard, amqp, anyjson, kombu, celery, urllib3, elasticsearch, meld3, supervisor, djangorestframework, django-filter
  Found existing installation: Sphinx 1.2.3
Cannot remove entries from nonexistent file /Users/Antrromet/anaconda/envs/memex/lib/python2.7/site-packages/easy-install.pth
#
# To activate this environment, use:
# $ source activate memex
#
# To deactivate this environment, use:
# $ source deactivate
#
~/Documents/USC/Fall2015/IR/memex-explorer/source
~/Documents/USC/Fall2015/IR/memex-explorer/source/memex ~/Documents/USC/Fall2015/IR/memex-explorer/source
~/Documents/USC/Fall2015/IR/memex-explorer/source
discarding /Users/Antrromet/anaconda/bin from PATH
prepending /Users/Antrromet/anaconda/envs/memex/bin to PATH
Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/Antrromet/anaconda/envs/memex/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/Users/Antrromet/anaconda/envs/memex/lib/python2.7/site-packages/django/core/management/__init__.py", line 345, in execute
    settings.INSTALLED_APPS
  File "/Users/Antrromet/anaconda/envs/memex/lib/python2.7/site-packages/django/conf/__init__.py", line 46, in __getattr__
    self._setup(name)
  File "/Users/Antrromet/anaconda/envs/memex/lib/python2.7/site-packages/django/conf/__init__.py", line 42, in _setup
    self._wrapped = Settings(settings_module)
  File "/Users/Antrromet/anaconda/envs/memex/lib/python2.7/site-packages/django/conf/__init__.py", line 98, in __init__
    % (self.SETTINGS_MODULE, e)
ImportError: Could not import settings 'memex.settings' (Is it on sys.path? Is there an import error in the settings file?): No module named celery
discarding /Users/Antrromet/anaconda/envs/memex/bin from PATH

[ahmadia]

Thanks for the report @antrromet. We're looking into it.

[ahmadia]

@antrromet - we've found an issue in conda env install related to installing the latest Sphinx via pip when it has already been installed with conda. This just turned up recently in our CI tests and turned out to be related to your issue. We've just pushed https://github.com/memex-explorer/memex-explorer/commit/2f0c9b2a2253db23b28b62d92dcb3de95dc183be - which should fix that problem.

[brittainhard]

Build just passed. That should fix the error.

[antrromet]

Am I suppose to run some other command for installation? Because, its giving me the exact same errors as above? I ran ./app_setup.py in /memex-explorer/source

[ahmadia]

Sorry, @antrromet you'll need to delete the existing environment:

rm -rf ~/miniconda/envs/memex

Substitute with anaconda if that's what you installed.

[antrromet]

Sorry for my stupidity, I should have known about deleting the environment. Anyways its installed now, but memex doesn't work :(

I ran the following commands

$ ./app_setup.sh
$ source activate memex
$ supervisord

But I couldn't find anything on http://localhost:8000. Here are the logs.

(memex)Antrromets-MacBook-Pro:source Antrromet$ supervisord
2015-10-09 17:12:45,706 INFO Increased RLIMIT_NOFILE limit to 1024
2015-10-09 17:12:45,734 INFO RPC interface 'supervisor' initialized
2015-10-09 17:12:45,734 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2015-10-09 17:12:45,735 INFO supervisord started with pid 2684
2015-10-09 17:12:46,742 INFO spawned: 'redis' with pid 2687
2015-10-09 17:12:46,745 INFO spawned: 'elasticsearch' with pid 2688
2015-10-09 17:12:46,746 INFO spawnerr: can't find command 'log.io-harvester'
2015-10-09 17:12:46,746 INFO spawnerr: can't find command 'log.io-server'
2015-10-09 17:12:46,748 INFO spawned: 'kibana' with pid 2689
2015-10-09 17:12:46,750 INFO spawned: 'celery' with pid 2690
2015-10-09 17:12:46,753 INFO spawned: 'tika' with pid 2691
2015-10-09 17:12:46,756 INFO spawned: 'django' with pid 2692
2015-10-09 17:12:47,975 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-10-09 17:12:47,975 INFO success: elasticsearch entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-10-09 17:12:47,975 INFO spawnerr: can't find command 'log.io-harvester'
2015-10-09 17:12:47,976 INFO spawnerr: can't find command 'log.io-server'
2015-10-09 17:12:47,976 INFO success: kibana entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-10-09 17:12:47,976 INFO success: celery entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-10-09 17:12:47,976 INFO success: tika entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-10-09 17:12:47,976 INFO success: django entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-10-09 17:12:48,013 INFO exited: django (exit status 1; not expected)
2015-10-09 17:12:48,018 INFO spawned: 'django' with pid 2717
2015-10-09 17:12:48,075 INFO exited: celery (exit status 1; not expected)
2015-10-09 17:12:48,084 INFO spawned: 'celery' with pid 2720
2015-10-09 17:12:48,817 INFO exited: celery (exit status 1; not expected)
2015-10-09 17:12:48,834 INFO exited: elasticsearch (exit status 3; not expected)
2015-10-09 17:12:48,845 INFO spawned: 'elasticsearch' with pid 2727
2015-10-09 17:12:48,873 INFO exited: django (exit status 1; not expected)
2015-10-09 17:12:49,448 INFO exited: elasticsearch (exit status 3; not expected)
2015-10-09 17:12:50,457 INFO spawned: 'elasticsearch' with pid 2738
2015-10-09 17:12:50,459 INFO spawnerr: can't find command 'log.io-harvester'
2015-10-09 17:12:50,459 INFO spawnerr: can't find command 'log.io-server'
2015-10-09 17:12:50,461 INFO spawned: 'celery' with pid 2739
2015-10-09 17:12:50,463 INFO spawned: 'django' with pid 2740
2015-10-09 17:12:50,994 INFO exited: django (exit status 1; not expected)
2015-10-09 17:12:51,043 INFO exited: celery (exit status 1; not expected)
2015-10-09 17:12:51,214 INFO exited: elasticsearch (exit status 3; not expected)
2015-10-09 17:12:53,509 INFO spawned: 'elasticsearch' with pid 2755
2015-10-09 17:12:53,510 INFO spawnerr: can't find command 'log.io-harvester'
2015-10-09 17:12:53,510 INFO gave up: logio-harvester entered FATAL state, too many start retries too quickly
2015-10-09 17:12:53,511 INFO spawnerr: can't find command 'log.io-server'
2015-10-09 17:12:53,511 INFO gave up: logio-server entered FATAL state, too many start retries too quickly
2015-10-09 17:12:53,512 INFO spawned: 'celery' with pid 2756
2015-10-09 17:12:53,514 INFO spawned: 'django' with pid 2757
2015-10-09 17:12:54,090 INFO exited: django (exit status 1; not expected)
2015-10-09 17:12:54,109 INFO exited: celery (exit status 1; not expected)
2015-10-09 17:12:54,271 INFO exited: elasticsearch (exit status 3; not expected)
2015-10-09 17:12:57,534 INFO spawned: 'elasticsearch' with pid 2772
2015-10-09 17:12:57,540 INFO spawned: 'celery' with pid 2773
2015-10-09 17:12:57,544 INFO spawned: 'django' with pid 2774
2015-10-09 17:12:58,096 INFO exited: django (exit status 1; not expected)
2015-10-09 17:12:58,127 INFO gave up: django entered FATAL state, too many start retries too quickly
2015-10-09 17:12:58,168 INFO exited: celery (exit status 1; not expected)
2015-10-09 17:12:58,243 INFO gave up: celery entered FATAL state, too many start retries too quickly
2015-10-09 17:12:58,433 INFO exited: elasticsearch (exit status 3; not expected)
2015-10-09 17:12:59,035 INFO gave up: elasticsearch entered FATAL state, too many start retries too quickly

P.S. Am sorry I tried to attach the bulk of the logs above in a file, but i couldn't upload the file. It gave me error.

[ahmadia]

@antrromet - something is going wrong in your supervisor launch:

2015-10-09 17:12:54,090 INFO exited: django (exit status 1; not expected)
2015-10-09 17:12:54,109 INFO exited: celery (exit status 1; not expected)
2015-10-09 17:12:54,271 INFO exited: elasticsearch (exit status 3; not expected)

tika and redis are staying up so I'm guessing your basic environment is okay. You'll need to go digging into resource/logs or try bringing up the services individually to figure out what's happening. This would be a great piece of advice for a "Troubleshooting" section in our Developer's manual.

[antrromet]

@ahmadia Minor update. I read somewhere, that you might wanna run your server as root. When I do that, these are the logs now.

(memex)Antrromets-MacBook-Pro:source Antrromet$ sudo supervisord
/Users/Antrromet/anaconda/envs/memex/lib/python2.7/site-packages/supervisor/options.py:296: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
  'Supervisord is running as root and it is searching '
2015-10-09 19:22:35,377 CRIT Supervisor running as root (no user in config file)
2015-10-09 19:22:35,379 INFO Increased RLIMIT_NOFILE limit to 1024
2015-10-09 19:22:35,404 INFO RPC interface 'supervisor' initialized
2015-10-09 19:22:35,404 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2015-10-09 19:22:35,405 INFO supervisord started with pid 3730
2015-10-09 19:22:36,408 INFO spawned: 'redis' with pid 3733
2015-10-09 19:22:36,412 INFO spawned: 'elasticsearch' with pid 3734
2015-10-09 19:22:36,415 INFO spawnerr: can't find command 'log.io-harvester'
2015-10-09 19:22:36,415 INFO spawnerr: can't find command 'log.io-server'
2015-10-09 19:22:36,418 INFO spawned: 'kibana' with pid 3735
2015-10-09 19:22:36,420 INFO spawned: 'celery' with pid 3736
2015-10-09 19:22:36,423 INFO spawned: 'tika' with pid 3738
2015-10-09 19:22:36,425 INFO spawned: 'django' with pid 3739
2015-10-09 19:22:37,416 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-10-09 19:22:37,416 INFO success: elasticsearch entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-10-09 19:22:37,416 INFO spawnerr: can't find command 'log.io-harvester'
2015-10-09 19:22:37,417 INFO spawnerr: can't find command 'log.io-server'
2015-10-09 19:22:37,417 INFO success: kibana entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-10-09 19:22:37,420 INFO success: celery entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-10-09 19:22:37,421 INFO success: tika entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-10-09 19:22:37,429 INFO success: django entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-10-09 19:22:40,235 INFO spawnerr: can't find command 'log.io-harvester'
2015-10-09 19:22:40,235 INFO spawnerr: can't find command 'log.io-server'
2015-10-09 19:22:42,821 INFO exited: celery (exit status 1; not expected)
2015-10-09 19:22:43,101 INFO spawned: 'celery' with pid 3794
2015-10-09 19:22:43,352 INFO spawnerr: can't find command 'log.io-harvester'
2015-10-09 19:22:43,352 INFO gave up: logio-harvester entered FATAL state, too many start retries too quickly
2015-10-09 19:22:43,352 INFO spawnerr: can't find command 'log.io-server'
2015-10-09 19:22:43,352 INFO gave up: logio-server entered FATAL state, too many start retries too quickly
2015-10-09 19:22:45,031 INFO success: celery entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-10-09 19:22:45,042 INFO exited: celery (exit status 1; not expected)
2015-10-09 19:22:46,045 INFO spawned: 'celery' with pid 3819
2015-10-09 19:22:47,334 INFO success: celery entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-10-09 19:22:48,399 INFO exited: celery (exit status 1; not expected)
2015-10-09 19:22:49,402 INFO spawned: 'celery' with pid 3833

So now only celery is failing. 2015-10-09 19:22:45,042 INFO exited: celery (exit status 1; not expected)

[ahmadia]

No, please don't run this application as root. This causes all sorts of problems with permissions on the file system when you don't want to run as root, and may be the source of your original problems since you now may have files in your user directory that require administrative privileges to remove.

[antrromet]

Oh man :( Let me remove everything. I'm going to delete the existing environment and do a fresh install and check again.

[ahmadia]

Did you check the logs or just trying to run celery?

[ahmadia]

All supervisord doing is running: celery as you. You'll see the exact same error if you just try to launch celery yourself. Frequently the error is obvious if you check the logs or run the application outside of supervisord.

[antrromet]

@ahmadia Hey it worked :D I just uninstalled everything and the fresh installation works! I think the problem earlier was the sudo command that I might have used while running app_setup.sh. I'm not really sure if that was the problem though, because I followed the exact same steps as mentioned in the wiki.

Thanks a lot. Thanks for the patience. :)

[ahmadia]

We appreciate the feedback. Happy crawling :)

[saloneerege]

Thank you. I had the same error. I will try this if it works.