Create demo instance on CentOS

ajs6f commented 6 years ago

Hi, I'm looking to set up a local demo instance of OSF as part of a project of comparison going on at my institution. I've had very little luck with these instructions although I am still working through them. It's not easy to debug without any documentation of how all the many, many Docker containers fit together.

Is CentOS a supported environment for OSF?

TomBaxter commented 6 years ago

Thank you for your interest in The Center for Open Science. Awesome to hear that you are interested in running you own local instance of OSF. The instructions you referenced should get you a local development environment. Things to keep in mind when working through the instructions.

They are very Mac centric. Most notably is the fact docker-sync is a solution for a mac specific problem. So, you should ignore all related instructions. I am the only developer here running Linux (Ubuntu).
And therefore the Linux instructions are Ubuntu centric. However, I don't imagine there is a ton of difference. You'll need to follow the CentOS docker install instructions. And the loopback alias configuration might be different. I think that should be it for differences.
I've helped several Linux users get OSF running and the part that confuses people the most is the loopback alias. The purpose of the loopback alias is for the docker containers to connect to each other. Any situation where a service is connecting to another service it's doing so on 192.168.168.167 . However, the user interacts with all service on localhost.

I hope you find these tips useful and I hope you enjoy your experience with OSF.

Thanks, Tom Baxter

alsmnn commented 5 years ago

On CentOS 6 you can create a file /etc/sysconfig/network-scripts/ifcfg-lo:0 and insert the values from the README-docker-compose.md. On CentOS 7 the devices are controlled by the NetworkManager. In order to create the ALIAS you have to add the IP addresses by hand, without :

root@osf [~]# cat /etc/sysconfig/network-scripts/ifcfg-lo
DEVICE=lo
IPADDR=127.0.0.1
NETMASK=255.0.0.0
NETWORK=127.0.0.0
# If you're having problems with gated making 127.0.0.0/8 a martian,
# you can change this to something else (255.255.255.255, for example)
BROADCAST=127.255.255.255
ONBOOT=yes
NAME=loopback
**IPADDR1=192.168.168.167**
**NETMASK1=255.255.255.255**
**NETWORK1=192.168.168.167**

Now just restart the network service and your good to go

root@osf [~]# systemctl restart network
root@osf [~]# ip addr show dev lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 192.168.168.167/32 brd 192.168.168.167 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

root@osf [~]# ping -c4 192.168.168.167
PING 192.168.168.167 (192.168.168.167) 56(84) bytes of data.
64 bytes from 192.168.168.167: icmp_seq=1 ttl=64 time=0.053 ms
64 bytes from 192.168.168.167: icmp_seq=2 ttl=64 time=0.046 ms
64 bytes from 192.168.168.167: icmp_seq=3 ttl=64 time=0.044 ms
64 bytes from 192.168.168.167: icmp_seq=4 ttl=64 time=0.043 ms

--- 192.168.168.167 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3041ms
rtt min/avg/max/mdev = 0.043/0.046/0.053/0.007 ms

or you can add the alias ip temporarily with:

ip addr add 192.168.168.167/32 dev lo label lo:0

Note: this won't be available if you reboot or restart the network service *edit: typo

alsmnn commented 5 years ago

I'm sorry for capturing this thread and pls move my comments if they're inappropriate, I'm also trying to setup a local instance of osf on a CentOS7 machine. First of all, I'm a total beginner with dockers, so all they do is nothing but plain magic for me. I would like to run a local instance of osf, because of company regulations, but I think we could really benefit from the osf and I'm really excited about getting this to work.

I'm sorry for throwing all these log-files at you, but I have no idea where to start.

Here is my problem: After installing docker, docker-compose, and disabling SELinux on the machine, I established the loopback alias and tried to start the 'Application Environment' with:

docker-compose up requirements mfr_requirements wb_requirements

unfortunately the container requirements_1 exited with code 1

requirements_1      | Installing requirements for bitbucket
requirements_1      | /usr/local/bin/pip install --exists-action w --upgrade -r /code/addons/bitbucket/requirements.txt  -c /code/requirements/constraints.txt
requirements_1      | Traceback (most recent call last):
requirements_1      |   File "/usr/local/bin/invoke", line 11, in <module>
requirements_1      |     sys.exit(program.run())
requirements_1      |   File "/usr/local/lib/python2.7/site-packages/invoke/program.py", line 293, in run
requirements_1      |     self.execute()
requirements_1      |   File "/usr/local/lib/python2.7/site-packages/invoke/program.py", line 408, in execute
requirements_1      |     executor.execute(*self.tasks)
requirements_1      |   File "/usr/local/lib/python2.7/site-packages/invoke/executor.py", line 114, in execute
requirements_1      |     result = call.task(*args, **call.kwargs)
requirements_1      |   File "/usr/local/lib/python2.7/site-packages/invoke/tasks.py", line 114, in __call__
requirements_1      |     result = self.body(*args, **kwargs)
requirements_1      |   File "/code/tasks/__init__.py", line 247, in requirements
requirements_1      |     addon_requirements(ctx)
requirements_1      |   File "/usr/local/lib/python2.7/site-packages/invoke/tasks.py", line 114, in __call__
requirements_1      |     result = self.body(*args, **kwargs)
requirements_1      |   File "/code/tasks/__init__.py", line 534, in addon_requirements
requirements_1      |     echo=True
requirements_1      |   File "/usr/local/lib/python2.7/site-packages/invoke/context.py", line 70, in run
requirements_1      |     return runner_class(context=self).run(command, **kwargs)
requirements_1      |   File "/usr/local/lib/python2.7/site-packages/invoke/runners.py", line 259, in run
requirements_1      |     return self._run_body(command, **kwargs)
requirements_1      |   File "/usr/local/lib/python2.7/site-packages/invoke/runners.py", line 361, in _run_body
requirements_1      |     raise ThreadException(thread_exceptions)
requirements_1      | invoke.exceptions.ThreadException:
requirements_1      | Saw 1 exceptions within threads (IOError):
requirements_1      |
requirements_1      |
requirements_1      | Thread args: {'kwargs': {'echo': None,
requirements_1      |             'input_': <open file '<stdin>', mode 'r' at 0x7fa315eae0c0>,
requirements_1      |             'output': <open file '<stdout>', mode 'w' at 0x7fa315eae150>},
requirements_1      |  'target': <bound method Local.handle_stdin of <invoke.runners.Local object at 0x7fa30f1f2950>>}
requirements_1      |
requirements_1      | Traceback (most recent call last):
requirements_1      |
requirements_1      |   File "/usr/local/lib/python2.7/site-packages/invoke/util.py", line 142, in run
requirements_1      |     super(ExceptionHandlingThread, self).run()
requirements_1      |
requirements_1      |   File "/usr/local/lib/python2.7/threading.py", line 754, in run
requirements_1      |     self.__target(*self.__args, **self.__kwargs)
requirements_1      |
requirements_1      |   File "/usr/local/lib/python2.7/site-packages/invoke/runners.py", line 624, in handle_stdin
requirements_1      |     data = self.read_our_stdin(input_)
requirements_1      |
requirements_1      |   File "/usr/local/lib/python2.7/site-packages/invoke/runners.py", line 586, in read_our_stdin
requirements_1      |     bytes_ = input_.read(bytes_to_read(input_))
requirements_1      |
requirements_1      |   File "/usr/local/lib/python2.7/site-packages/invoke/platform.py", line 162, in bytes_to_read
requirements_1      |     return struct.unpack('h', fcntl.ioctl(input_, termios.FIONREAD, "  "))[0]
requirements_1      |
requirements_1      | IOError: [Errno 25] Inappropriate ioctl for device
requirements_1      |
requirements_1      |
osfio_requirements_1 exited with code 1

the mfr_requirements container says that flake has a requirement which is incompatible

mfr_requirements_1  | flake8 3.0.4 has requirement mccabe<0.6.0,>=0.5.0, but you'll have mccabe 0.6.1 which is incompatible.
mfr_requirements_1  | flake8 3.0.4 has requirement pyflakes!=1.2.0,!=1.2.1,!=1.2.2,<1.3.0,>=0.8.1, but you'll have pyflakes 2.0.0 which is incompatible.
mfr_requirements_1  | python-coveralls 2.9.1 has requirement coverage==4.0.3, but you'll have coverage 4.5.2 which is incompatible.

ando goes on with

mfr_requirements_1  | Listing '/usr/local/lib/python3.5/site-packages/_pydev_imps'...
mfr_requirements_1  | Compiling '/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_SimpleXMLRPCServer.py'...
mfr_requirements_1  | ***   File "/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_SimpleXMLRPCServer.py", line 272
mfr_requirements_1  |     except Fault, fault:
mfr_requirements_1  |                 ^
mfr_requirements_1  | SyntaxError: invalid syntax
mfr_requirements_1  |
mfr_requirements_1  | Compiling '/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_SocketServer.py'...
mfr_requirements_1  | ***   File "/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_SocketServer.py", line 339
mfr_requirements_1  |     print '-'*40
mfr_requirements_1  |             ^
mfr_requirements_1  | SyntaxError: Missing parentheses in call to 'print'
mfr_requirements_1  |
mfr_requirements_1  | Compiling '/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_inspect.py'...
mfr_requirements_1  | ***   File "/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_inspect.py", line 293
mfr_requirements_1  |     raise TypeError, 'arg is a built-in module'
mfr_requirements_1  |                    ^
mfr_requirements_1  | SyntaxError: invalid syntax
mfr_requirements_1  |
mfr_requirements_1  | Compiling '/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_pkgutil_old.py'...
mfr_requirements_1  | ***   File "/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_pkgutil_old.py", line 543
mfr_requirements_1  |     except IOError, msg:
mfr_requirements_1  |                   ^
mfr_requirements_1  | SyntaxError: invalid syntax
mfr_requirements_1  |
mfr_requirements_1  | Compiling '/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_uuid_old.py'...
mfr_requirements_1  | ***   File "/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_uuid_old.py", line 151
mfr_requirements_1  |     if not 0 <= time_low < 1<<32L:
mfr_requirements_1  |                                 ^
mfr_requirements_1  | SyntaxError: invalid syntax
mfr_requirements_1  |
mfr_requirements_1  | Compiling '/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_xmlrpclib.py'...
mfr_requirements_1  | ***   File "/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_xmlrpclib.py", line 294
mfr_requirements_1  |     True, False = True, False
mfr_requirements_1  |     ^
mfr_requirements_1  | SyntaxError: can't assign to keyword
mfr_requirements_1  |
mfr_requirements_1  | Listing '/usr/local/lib/python3.5/site-packages/_pydev_runfiles'...
mfr_requirements_1  | Listing '/usr/local/lib/python3.5/site-packages/_pydevd_bundle'...
mfr_requirements_1  | Compiling '/usr/local/lib/python3.5/site-packages/_pydevd_bundle/pydevconsole_code_for_ironpython.py'...
mfr_requirements_1  | ***   File "/usr/local/lib/python3.5/site-packages/_pydevd_bundle/pydevconsole_code_for_ironpython.py", line 105
mfr_requirements_1  |     except SyntaxError, err:
mfr_requirements_1  |                       ^
mfr_requirements_1  | SyntaxError: invalid syntax
mfr_requirements_1  |
mfr_requirements_1  | Compiling '/usr/local/lib/python3.5/site-packages/_pydevd_bundle/pydevd_exec.py'...
mfr_requirements_1  | ***   File "/usr/local/lib/python3.5/site-packages/_pydevd_bundle/pydevd_exec.py", line 3
mfr_requirements_1  |     exec exp in global_vars, local_vars
mfr_requirements_1  |            ^
mfr_requirements_1  | SyntaxError: Missing parentheses in call to 'exec'
mfr_requirements_1  |
mfr_requirements_1  | Listing '/usr/local/lib/python3.5/site-packages/_pytest'...

and

mfr_requirements_1  | Listing '/usr/local/lib/python3.5/site-packages/pydev_ipython'...
mfr_requirements_1  | Compiling '/usr/local/lib/python3.5/site-packages/pydev_ipython/inputhookglut.py'...
mfr_requirements_1  | ***   File "/usr/local/lib/python3.5/site-packages/pydev_ipython/inputhookglut.py", line 98
mfr_requirements_1  |     print '\nKeyboardInterrupt'
mfr_requirements_1  |                               ^
mfr_requirements_1  | SyntaxError: Missing parentheses in call to 'print'
mfr_requirements_1  |
mfr_requirements_1  | Listing '/usr/local/lib/python3.5/site-packages/pydevd-0.0.6.dist-info'...

which results in

osfio_mfr_requirements_1 exited with code 0

furthermore the wb_requirements exited with code 0, because of a syntax error

wb_requirements_1   | Listing '/usr/local/lib/python3.5/site-packages/_pydev_imps'...
wb_requirements_1   | Compiling '/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_SimpleXMLRPCServer.py'...
wb_requirements_1   | ***   File "/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_SimpleXMLRPCServer.py", line 272
wb_requirements_1   |     except Fault, fault:
wb_requirements_1   |                 ^
wb_requirements_1   | SyntaxError: invalid syntax
wb_requirements_1   |
wb_requirements_1   | Compiling '/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_SocketServer.py'...
wb_requirements_1   | ***   File "/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_SocketServer.py", line 339
wb_requirements_1   |     print '-'*40
wb_requirements_1   |             ^
wb_requirements_1   | SyntaxError: Missing parentheses in call to 'print'
wb_requirements_1   |
wb_requirements_1   | Compiling '/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_inspect.py'...
wb_requirements_1   | ***   File "/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_inspect.py", line 293
wb_requirements_1   |     raise TypeError, 'arg is a built-in module'
wb_requirements_1   |                    ^
wb_requirements_1   | SyntaxError: invalid syntax
wb_requirements_1   |
wb_requirements_1   | Compiling '/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_pkgutil_old.py'...
wb_requirements_1   | ***   File "/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_pkgutil_old.py", line 543
wb_requirements_1   |     except IOError, msg:
wb_requirements_1   |                   ^
wb_requirements_1   | SyntaxError: invalid syntax
wb_requirements_1   |
wb_requirements_1   | Compiling '/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_uuid_old.py'...
wb_requirements_1   | ***   File "/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_uuid_old.py", line 151
wb_requirements_1   |     if not 0 <= time_low < 1<<32L:
wb_requirements_1   |                                 ^
wb_requirements_1   | SyntaxError: invalid syntax
wb_requirements_1   |
wb_requirements_1   | Compiling '/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_xmlrpclib.py'...
wb_requirements_1   | ***   File "/usr/local/lib/python3.5/site-packages/_pydev_imps/_pydev_xmlrpclib.py", line 294
wb_requirements_1   |     True, False = True, False
wb_requirements_1   |     ^
wb_requirements_1   | SyntaxError: can't assign to keyword
wb_requirements_1   |
wb_requirements_1   | Listing '/usr/local/lib/python3.5/site-packages/_pydev_runfiles'...
wb_requirements_1   | Listing '/usr/local/lib/python3.5/site-packages/_pydevd_bundle'...
wb_requirements_1   | Compiling '/usr/local/lib/python3.5/site-packages/_pydevd_bundle/pydevconsole_code_for_ironpython.py'...
wb_requirements_1   | ***   File "/usr/local/lib/python3.5/site-packages/_pydevd_bundle/pydevconsole_code_for_ironpython.py", line 105
wb_requirements_1   |     except SyntaxError, err:
wb_requirements_1   |                       ^
wb_requirements_1   | SyntaxError: invalid syntax
wb_requirements_1   |
wb_requirements_1   | Compiling '/usr/local/lib/python3.5/site-packages/_pydevd_bundle/pydevd_exec.py'...
wb_requirements_1   | ***   File "/usr/local/lib/python3.5/site-packages/_pydevd_bundle/pydevd_exec.py", line 3
wb_requirements_1   |     exec exp in global_vars, local_vars
wb_requirements_1   |            ^
wb_requirements_1   | SyntaxError: Missing parentheses in call to 'exec'

and another one

wb_requirements_1   | Compiling '/usr/local/lib/python3.5/site-packages/pydev_ipython/inputhookglut.py'...
wb_requirements_1   | ***   File "/usr/local/lib/python3.5/site-packages/pydev_ipython/inputhookglut.py", line 98
wb_requirements_1   |     print '\nKeyboardInterrupt'
wb_requirements_1   |                               ^
wb_requirements_1   | SyntaxError: Missing parentheses in call to 'print'
wb_requirements_1   |
wb_requirements_1   | Listing '/usr/local/lib/python3.5/site-packages/pydevd-0.0.6.dist-info'...

and then it goes on with compiling and exiting

wb_requirements_1   | Compiling '/usr/local/lib/python3.5/xml/dom/xmlbuilder.py'...
wb_requirements_1   | Listing '/usr/local/lib/python3.5/xml/etree'...
wb_requirements_1   | Compiling '/usr/local/lib/python3.5/xml/etree/ElementInclude.py'...
wb_requirements_1   | Listing '/usr/local/lib/python3.5/xml/parsers'...
wb_requirements_1   | Listing '/usr/local/lib/python3.5/xml/sax'...
wb_requirements_1   | Compiling '/usr/local/lib/python3.5/xml/sax/expatreader.py'...
wb_requirements_1   | Listing '/usr/local/lib/python3.5/xmlrpc'...
wb_requirements_1   | Compiling '/usr/local/lib/python3.5/xmlrpc/server.py'...
wb_requirements_1   | Compiling '/usr/local/lib/python3.5/zipapp.py'...
osfio_wb_requirements_1 exited with code 0

Best regards @AljoLe

alsmnn commented 5 years ago

I'm still trying to set up a local instance of osf on a CentOS7 machine. Any thoughts from you guys on this?

CenterForOpenScience / osf.io

Create demo instance on CentOS #7961