NGINX related issues

[buckyroberts]

When visiting https://thenewboston.network/admin NGINX is throwing a 502 error. Additionally, when deploying a separate EC2 instance for debugging, in the NGINX error logs there is an error related to loading in the SSL certificate. It seems like NGINX needs to be running so that certbot / Let's Encrypt can validate the domain to obtain a certificate. However NGINX errors out when we first try to run it because the certificates are missing.

Note that we will be deploying this software across multiple servers, each with their own domain name. So the NGINX configuration must be flexible to allow for that.

Also when looking into this issue with Truyen last month, he shared this error message:

Collecting static files...
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/opt/project/core/manage.py", line 22, in <module>
    main()
  File "/opt/project/core/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
    utility.execute()
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/django/core/management/__init__.py", line 440, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/django/core/management/base.py", line 414, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/django/core/management/base.py", line 460, in execute
    output = self.handle(*args, **options)
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 184, in handle
    if self.is_local_storage() and self.storage.location:
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 245, in is_local_storage
    return isinstance(self.storage, FileSystemStorage)
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/django/utils/functional.py", line 258, in inner
    self._setup()
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/django/contrib/staticfiles/storage.py", line 489, in _setup
    self._wrapped = get_storage_class(settings.STATICFILES_STORAGE)()
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/whitenoise/storage.py", line 131, in __init__
    super().__init__(*args, **kwargs)
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/django/contrib/staticfiles/storage.py", line 419, in __init__
    self.hashed_files = self.load_manifest()
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/django/contrib/staticfiles/storage.py", line 440, in load_manifest
    raise ValueError(
ValueError: Couldn't load manifest 'staticfiles.json' (version 1.0)

[dmugtasimov]

dmugtasimov@dmugtasimov-Latitude-E5550:~/gitrep/thenewboston/Core$ curl -v http://thenewboston.network
*   Trying 52.32.121.197:80...
* TCP_NODELAY set
* connect to 52.32.121.197 port 80 failed: Connection timed out
* Failed to connect to thenewboston.network port 80: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to thenewboston.network port 80: Connection timed out

Because server is stopped on AWS. Will start it first and see what happens

[dmugtasimov]

@olegtropinin please, provide a draft PR or a description of the Django issue you found, so I can evaluate it

[dmugtasimov]

After starting the server we get:

dmugtasimov@dmugtasimov-Latitude-E5550:~/gitrep/thenewboston/Core$ curl -v http://thenewboston.network
*   Trying 52.32.121.197:80...
* TCP_NODELAY set
* Connected to thenewboston.network (52.32.121.197) port 80 (#0)
> GET / HTTP/1.1
> Host: thenewboston.network
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.21.6
< Date: Mon, 03 Oct 2022 21:34:01 GMT
< Content-Type: text/html
< Content-Length: 169
< Connection: keep-alive
< Location: https://thenewboston.network/
< 
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.21.6</center>
</body>
</html>
* Connection #0 to host thenewboston.network left intact
dmugtasimov@dmugtasimov-Latitude-E5550:~/gitrep/thenewboston/Core$ curl -v https://thenewboston.network
*   Trying 52.32.121.197:443...
* TCP_NODELAY set
* Connected to thenewboston.network (52.32.121.197) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=thenewboston.network
*  start date: Sep 17 19:26:59 2022 GMT
*  expire date: Dec 16 19:26:58 2022 GMT
*  subjectAltName: host "thenewboston.network" matched cert's "thenewboston.network"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: thenewboston.network
> User-Agent: curl/7.68.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 502 Bad Gateway
< Server: nginx/1.21.6
< Date: Mon, 03 Oct 2022 21:34:09 GMT
< Content-Type: text/html
< Content-Length: 157
< Connection: keep-alive
< 
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.21.6</center>
</body>
</html>
* Connection #0 to host thenewboston.network left intact

[dmugtasimov]

@olegtropinin is right the issue has nothing to do with certbot, because TLS handshake works fine. The reason why it did not work for @truyenhv is that DNS record was not configured to point to core-2 EC2 instance.

[truyenhv]

hey. Core-2 is fully cloned from core server @dmugtasimov. I used that server because I didn't have the private key to access.

[dmugtasimov]

I think this exception is causing the issue:

django.core.exceptions.ImproperlyConfigured: Requested setting REST_FRAMEWORK, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/bin/daphne", line 8, in <module>
    sys.exit(CommandLineInterface.entrypoint())
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/daphne/cli.py", line 170, in entrypoint
    cls().run(sys.argv[1:])
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/daphne/cli.py", line 232, in run
    application = import_by_path(args.application)
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/daphne/utils.py", line 12, in import_by_path
    target = importlib.import_module(module_path)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/project/./core/project/asgi.py", line 6, in <module>
    from core.accounts.routing import websocket_urlpatterns
  File "/opt/project/./core/accounts/routing.py", line 3, in <module>
    from core.accounts.consumers import AccountConsumer
  File "/opt/project/./core/accounts/consumers.py", line 10, in <module>
    from core.core.exceptions import NotAuthenticated
  File "/opt/project/./core/core/exceptions.py", line 2, in <module>
    from rest_framework.views import exception_handler
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/rest_framework/views.py", line 17, in <module>
    from rest_framework.schemas import DefaultSchema
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/rest_framework/schemas/__init__.py", line 25, in <module>
    from . import coreapi, openapi
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/rest_framework/schemas/openapi.py", line 15, in <module>
    from rest_framework import (
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/rest_framework/renderers.py", line 53, in <module>
    class JSONRenderer(BaseRenderer):
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/rest_framework/renderers.py", line 60, in JSONRenderer
    ensure_ascii = not api_settings.UNICODE_JSON
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/rest_framework/settings.py", line 218, in __getattr__
    val = self.user_settings[attr]
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/rest_framework/settings.py", line 209, in user_settings
    self._user_settings = getattr(settings, 'REST_FRAMEWORK', {})
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/django/conf/__init__.py", line 87, in __getattr__
    self._setup(name)
  File "/root/.cache/pypoetry/virtualenvs/core-gHi8t1rX-py3.10/lib/python3.10/site-packages/django/conf/__init__.py", line 67, in _setup
    raise ImproperlyConfigured(

Let's see Oleg's solution before moving forward, so I do not repeat the work already done.

[dmugtasimov]

@truyenhv

DNS A record must point to the EC2 instance for certbot to solve the challange.

[truyenhv]

hm. I thought I updated elastic ip to new server already? Btw the core container can't start with the error that I sent so I didn't check more.

[dmugtasimov]

@olegtropinin I stopped ubuntu-core-1 container, otherwise it keeps restarting:

ubuntu@ip-172-31-32-54:~$ docker ps -a
CONTAINER ID   IMAGE                                                       COMMAND                  CREATED       STATUS                      PORTS                                                                      NAMES
e0c0bf936c33   ghcr.io/thenewboston-developers/core:latest                 "./run.sh"               5 weeks ago   Exited (1) 23 minutes ago                                                                              ubuntu-core-1

[dmugtasimov]

@buckyroberts "Note that we will be deploying this software across multiple servers, each with their own domain name. So the NGINX configuration must be flexible to allow for that." - this should work, but you should test it once this issue is fixed (this issue is not related to domain name)

[olegtropinin]

@dmugtasimov Could you please review my PR and merge it. https://github.com/thenewboston-developers/Core/pull/130

Let's merge it with master and then I will remove core-2 server on AWS. Core-1 should have right configuration of Nginx and Certbot.

[dmugtasimov]

@olegtropinin great job, thank you. I merged with some slight changes

[dmugtasimov]

after upgrading Django we got another issue that requested DRF upgrade: https://github.com/thenewboston-developers/Core/pull/131

[dmugtasimov]

@buckyroberts I was able to login to Django Admin it should work now, please, test