ports

[andreasleha]

I am trying to setup a local dev instance following the readme. I'd like to do that in virtual machine. When setting up the virtual machine, I need to specify the ports that need to be opened.

Hence, my question: Which ports need to be accessible? Are all ports in the docker-compse.yml needed to be open?

Thanks in advance!

[Johnetordoff]

It's difficult to say without knowing more specific details, usually our docker-compose container is all the virtualization someone will need to run the OSF locally for standard OSs. If you want to set up up a docker container inside a virtual environment, you're going to have to make sure both are networked correctly to access them. So, for example, you don't need to expose the 5432 (the postgres port number) in your environment because that already exposed within docker. Virtual machines vary widely so this may be different for you, but hopefully that points in the right direction.

[andreasleha]

Thanks for the swift answer! Sorry for the bad phrased question. There will no container within the virtual machine involved. Just the virtual machine. Does that make it easier?

[Johnetordoff]

To set up our entire environment without using Docker is pretty difficult, I wouldn't recommend it. But if you just want to see what the site looks like locally using your virtual machine, just open the ports you need for whatever it is you are doing. So if you need to access the actual site, that port is 5000, the REST server is 8000, file server (waterbutler) is 7777 etc. These services really depend on each other to work, that's why I recommend using docker-compose, it's very difficult to have everything setup under one virtual machine. But it really depends on what your virtualization strategy is.

[andreasleha]

Thanks again. Maybe it is then easier for me to try with docker-compose on a real machine. What ports need to be accessible on the real machine? If I test it locally only, are there still ports that need to be accessible?

[Johnetordoff]

The docker-compose and Dockerfile define the accessible ports. They should be accessible from the browser via your localhost for example url http://localhost:5000/dashboard/ is the front page of the site. Just follow the documentation, the ports should need no additional configuration beyond that.

[andreasleha]

Thanks again for the swift reply. I am now using docker-compose and follow the instructions here. Unfortunately, when running the migrations, there is a problem with the communication to the database:

$ docker-compose run --rm web python3 manage.py migrate                         
Starting osf2_postgres_1 ... done                                                                                                       
Starting osf2_rabbitmq_1 ... done                                                                                                       
Starting osf2_elasticsearch_1 ... done                                                                                                  
[website.util.paths]  WARNING: Skipping load of "webpack-assets.json" in DEBUG_MODE.                                                    
[addons.osfstorage.settings.defaults]  WARNING: No migration settings loaded for OSFStorage, falling back to local dev. module 'website.settings' has no attribute 'MIGRATION_ENV'
[addons.osfstorage.settings]  WARNING: No local.py settings file found
[addons.bitbucket.settings]  WARNING: No local.py settings file found                       
[addons.box.settings]  WARNING: No local.py settings file found
[addons.dataverse.settings]  WARNING: No local.py settings file found                                        
[addons.dropbox.settings]  WARNING: No local.py settings file found
[addons.figshare.settings]  WARNING: No local.py settings file found                              
[addons.github.settings]  WARNING: No local.py settings file found                                                                      
[addons.gitlab.settings]  WARNING: No local.py settings file found                                                                      
[addons.googledrive.settings]  WARNING: No local.py settings file found                                                                 
[addons.owncloud.settings]  WARNING: No local.py settings file found                                      
[addons.s3.settings]  WARNING: No local.py settings file found
[addons.mendeley.settings]  WARNING: No local.py settings file found                   
[addons.onedrive.settings]  WARNING: No local.py settings file found 
[addons.wiki.settings]  WARNING: No local.py settings file found
[addons.zotero.settings]  WARNING: No local.py settings file found
[raven.contrib.django.client.DjangoClient]  INFO: Raven is not configured (logging is disabled). Please see the documentation for more information.
[website.app]  INFO: Sentry disabled; Flask's debug mode enabled
Traceback (most recent call last):
  File "manage.py", line 21, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3.6/site-packages/django/core/management/base.py", line 327, in execute
    self.check()
  File "/usr/lib/python3.6/site-packages/django/core/management/base.py", line 359, in check
    include_deployment_checks=include_deployment_checks,
  File "/usr/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 62, in _run_checks
    issues.extend(super(Command, self)._run_checks(**kwargs))
  File "/usr/lib/python3.6/site-packages/django/core/management/base.py", line 346, in _run_checks
    return checks.run_checks(**kwargs)
  File "/usr/lib/python3.6/site-packages/django/core/checks/registry.py", line 81, in run_checks
    new_errors = check(app_configs=app_configs)
  File "/usr/lib/python3.6/site-packages/django/core/checks/model_checks.py", line 30, in check_all_models
    errors.extend(model.check(**kwargs))
  File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 1284, in check
    errors.extend(cls._check_fields(**kwargs))
  File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 1359, in _check_fields
    errors.extend(field.check(**kwargs))
  File "/usr/lib/python3.6/site-packages/django/db/models/fields/__init__.py", line 913, in check
    errors = super(AutoField, self).check(**kwargs)
  File "/usr/lib/python3.6/site-packages/django/db/models/fields/__init__.py", line 219, in check
    errors.extend(self._check_backend_specific_checks(**kwargs))
  File "/usr/lib/python3.6/site-packages/django/db/models/fields/__init__.py", line 321, in _check_backend_specific_checks
    if router.allow_migrate(db, app_label, model_name=self.model._meta.model_name):
  File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 293, in allow_migrate
    for router in self.routers:
  File "/usr/lib/python3.6/site-packages/django/utils/functional.py", line 35, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 251, in routers
    router = import_string(r)()
  File "/code/osf/db/router.py", line 20, in __init__
    self.CACHED_MASTER = self._get_master()
  File "/code/osf/db/router.py", line 28, in _get_master
    conn = self._get_conn(dsn)
  File "/code/osf/db/router.py", line 59, in _get_conn
    return psycopg2.connect(dsn)
  File "/usr/lib/python3.6/site-packages/psycopg2/__init__.py", line 130, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Host is unreachable
        Is the server running on host "192.168.168.167" and accepting
        TCP/IP connections on port 5432?

But the postgresql container is started without errors:

$ docker-compose up postgres
Pulling postgres (postgres:9.6)...
9.6: Pulling from library/postgres
1cb79db8a9e7: Pull complete
f6bae7873dd7: Pull complete
8f7722dc50a7: Pull complete
e8622b8cb6f3: Pull complete
d6d74bba3a57: Pull complete
874d4d2a09fd: Pull complete
2d87c3a4038c: Pull complete
f955a6cf127b: Pull complete
f62dc55c568d: Pull complete
4e2c4902efbd: Pull complete
01c676df543a: Pull complete
1e3d335ef0b7: Pull complete
11087f2b0d87: Pull complete
4b9a74ac6ea0: Pull complete
Digest: sha256:caddd35b05cdd56c614ab1f674e63be778e0abdf54e71a7507ff3e28d4902698
Status: Downloaded newer image for postgres:9.6
Creating osf2_postgres_1 ... done
Attaching to osf2_postgres_1
postgres_1          | 
postgres_1          | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres_1          | 
postgres_1          | LOG:  database system was shut down at 2022-04-06 14:27:13 UTC
postgres_1          | LOG:  MultiXact member wraparound protections are now enabled
postgres_1          | LOG:  database system is ready to accept connections
postgres_1          | LOG:  autovacuum launcher started

The loopback is created and the port seems open and reachable:

# nmap -sT -p- 192.168.168.167

Starting Nmap 6.40 ( http://nmap.org ) at 2022-04-06 16:51 CEST
Nmap scan report for 192.168.168.167
Host is up (0.00099s latency).
Not shown: 65514 closed ports
PORT      STATE SERVICE
22/tcp    open  ssh
111/tcp   open  rpcbind
2049/tcp  open  nfs
4723/tcp  open  unknown
5000/tcp  open  upnp
5432/tcp  open  postgresql
5666/tcp  open  nrpe
5672/tcp  open  amqp
7007/tcp  open  afs3-bos
7777/tcp  open  cbt
7778/tcp  open  interwise
8080/tcp  open  http-proxy
8787/tcp  open  unknown
8788/tcp  open  unknown
9200/tcp  open  wap-wsp
15672/tcp open  unknown
20048/tcp open  unknown
27017/tcp open  unknown
31416/tcp open  boinc
33543/tcp open  unknown
45138/tcp open  unknown

Nmap done: 1 IP address (1 host up) scanned in 4.61 seconds

I've closely followed the instructions. Only change I had to make was to reduce the max_connections for postgresql in the docker-compose.yml to get rid of FATAL: could not create semaphores: No space left on device.

What could be wrong here?

[Johnetordoff]

Based on these it seems as though postgres, (port 5432) is unaccessible, but it does look based on the postgres log you provide that postgres is actually running fine, so it's difficult to say what the problem is. If you had to change the max_connections for postgres to get this to work, I'd guess there's some other problems related to the platform you are using. Occasionally I personally will get the Is the server running on host "192.168.168.167" and accepting TCP/IP connections on port 5432? error and can fix it by alllowing postgres to wake up and then simply restarting the web or api container and though containers will start working.

[andreasleha]

Thanks again for your comment! Highly appreciated. I've now run the steps on a different (virtual) machine and indeed I can now run all the steps from the README-docker-compose without errors. Unfortunately, I still can't access the local OSF. When browsing to http://ip.address.of.server:5000 I get

requests.exceptions.ReadTimeout
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='192.168.168.167', port=4200): Read timed out. (read timeout=3.05)

(the port is open

# nmap -sT -p- 192.168.168.167
Starting Nmap 7.80 ( https://nmap.org ) at 2022-04-21 16:11 CEST
Nmap scan report for 192.168.168.167
Host is up (0.000071s latency).
Not shown: 65518 closed ports
PORT      STATE SERVICE                                    
22/tcp    open  ssh
4200/tcp  open  vrml-multi-use
4201/tcp  open  vrml-multi-use
5000/tcp  open  upnp
5432/tcp  open  postgresql
5672/tcp  open  amqp
7007/tcp  open  afs3-bos
7777/tcp  open  cbt
7778/tcp  open  interwise
8000/tcp  open  http-alt
8001/tcp  open  vcom-tunnel
8080/tcp  open  http-proxy
9200/tcp  open  wap-wsp
15672/tcp open  unknown
27017/tcp open  mongod
41953/tcp open  unknown
41954/tcp open  unknown

)

Any hint how to proceed here?

[Johnetordoff]

It looks like your ember_osf_web container (at port 4200) is not functioning, it usually takes a couple minutes to build. You should be able to see:

 Slowest Nodes (totalTime => 5% )              | Total (avg)
 ----------------------------------------------+--------------------
 Babel: ember-osf-web (2)                      | 98653ms (49326 ms)
 Babel: registries (2)                         | 67215ms (33607 ms)
 Babel: analytics-page (2)                     | 60147ms (30073 ms)
 Babel: osf-components (1)                     | 39314ms
 Babel: ember-source (5)                       | 32158ms (6431 ms)

in the ember_osf_web logs when it's finished building.

[andreasleha]

Thanks again. I got this resolved! The local instance is now (kind of) running.

The initial page loads a long time, but does not show anything else besides the header. After switching to 'Support' and the back to 'OSFHome' there is some content below some large white space. Is that a known thing?

[Johnetordoff]

The header with no page is actually a unintended result of how the routing is configured locally, admittedly it's a bit confusing, but if you go to localhost:5000/dashboard/ you will be sent to the Dashboard that a typical user would see when logging in. You should also be sure to have the fakecas container up and running, some login redirects require it on the dashboard.

[andreasleha]

Thanks so much for the again swift reply. This is really great!

I am still struggling to set this up properly, however.

The improper loading I can circumvent if I follow the instructions more closely and browse to 'localhost:5000' on the machine with OSF itself. But that is a remote server and I would rather access it from a browser at my desktop. If I go to :5000 this does not work properly (as described in my previous post).

So, my first question would be the steps to make the OSF installation properly accessible from other computers rather than localhost only.

My second issue is, that registering a new user does not work, as no confirmation email is sent out. Any hint, what might cause this?

[Johnetordoff]

Second question first:

My second issue is, that registering a new user does not work, as no confirmation email is sent out. Any hint, what might cause this?

This is because the email service is not set up, you still have many elements to configure, (email, authentication .etc) to run OSF exactly as you would in a finished state, like the virtualization, how you set up the email is up to you. When I am developing for the OSF and want to create a user account locally I will check the the docker-compose logs. The web container's logs are configured to print the info any email it would have sent had the email service been set up, so they are valuable info for testing and development.

Now the first question:

So, my first question would be the steps to make the OSF installation properly accessible from other computers rather than localhost only.

This is a broad question and there's many answers, it really depends on how you want to configure your network. We at COS have a very complex and frequently changing network environment that we are constantly improving so there's not a simple "do this" answer. However if I wanted to set up a small clone of the OSF I would use a cloud service, we are use Google Cloud Service now, but any other implementation is necessarily going to differ from ours because the scale of our operations. It really depends on what your resources are, and what your desired use case is. If you just want to explore the code or test features, I recommend tunneling via ngrok, that can make ports publicly available for testing with limitations. Ultimately there's no single answer to give here.