observatorycontrolsystem / ocs_example

An example OCS project
GNU General Public License v3.0
5 stars 2 forks source link

Unhealthy configdb on ocs_example startup #20

Closed jmccormac01 closed 4 days ago

jmccormac01 commented 3 months ago

Hi, I'm having an issue when trying to run the ocs_example via docker compose. I have freshly cloned the latest main branch including the submodule support as per the readme. When I bring the example up I get:

[+] Running 19/19
 ✔ Network ocs_example_default                      Created                                                                                                                                               0.2s 
 ✔ Container ocs_example-db-1                       Healthy                                                                                                                                              12.6s 
 ✔ Container ocs_example-ocs_frontend-1             Started                                                                                                                                               1.3s 
 ✔ Container ocs_example-minio3-1                   Started                                                                                                                                               1.2s 
 ✔ Container ocs_example-minio1-1                   Started                                                                                                                                               1.3s 
 ✔ Container ocs_example-minio2-1                   Started                                                                                                                                               1.1s 
 ✔ Container ocs_example-minio4-1                   Started                                                                                                                                               1.0s 
 ✔ Container ocs_example-redis-1                    Healthy                                                                                                                                              12.6s 
 ✘ Container ocs_example-configdb-1                 Error                                                                                                                                                62.5s 
 ✔ Container ocs_example-observation-portal-1       Created                                                                                                                                               0.1s 
 ✔ Container ocs_example-downtime-1                 Created                                                                                                                                               0.1s 
 ✔ Container ocs_example-dramatiq_worker-1          Created                                                                                                                                               0.1s 
 ✔ Container ocs_example-adaptive_scheduler-1       Created                                                                                                                                               0.1s 
 ✔ Container ocs_example-dramatiq_task_scheduler-1  Created                                                                                                                                               0.0s 
 ✔ Container sciencearchive                         Created                                                                                                                                               0.0s 
 ✔ Container ocs_example-nginx-1                    Created                                                                                                                                               0.0s 
 ✔ Container simbad2k                               Created                                                                                                                                               0.0s 
 ✔ Container thumbnailservice                       Created                                                                                                                                               0.0s 
 ✔ Container ocs_example-science_archive_client-1   Created                                                                                                                                               0.0s 
dependency failed to start: container ocs_example-configdb-1 is unhealthy

Looking at the running containers, configdb is reporting unhealthy.

[ops@localhost ocs_example]$ docker container ls
CONTAINER ID   IMAGE                                                         COMMAND                  CREATED              STATUS                          PORTS                                                 NAMES
75aa600d16f9   ghcr.io/observatorycontrolsystem/configdb:3.0.4               "bash -c 'python man…"   About a minute ago   Up About a minute (unhealthy)   0.0.0.0:7000->7000/tcp, :::7000->7000/tcp, 8080/tcp   ocs_example-configdb-1
3fef7d7ca607   minio/minio:RELEASE.2021-06-17T00-10-46Z                      "/usr/bin/docker-ent…"   About a minute ago   Up About a minute (healthy)     9000/tcp                                              ocs_example-minio3-1
79281ec5574a   minio/minio:RELEASE.2021-06-17T00-10-46Z                      "/usr/bin/docker-ent…"   About a minute ago   Up About a minute (healthy)     9000/tcp                                              ocs_example-minio2-1
701896cc457e   redis:6.2.6                                                   "docker-entrypoint.s…"   About a minute ago   Up About a minute (healthy)     0.0.0.0:6373->6379/tcp, :::6373->6379/tcp             ocs_example-redis-1
dfa88cb44c77   postgis/postgis:14-master                                     "docker-entrypoint.s…"   About a minute ago   Up About a minute (healthy)     5432/tcp                                              ocs_example-db-1
cea907dfa69c   minio/minio:RELEASE.2021-06-17T00-10-46Z                      "/usr/bin/docker-ent…"   About a minute ago   Up About a minute (healthy)     9000/tcp                                              ocs_example-minio1-1
e6ce9e494b32   minio/minio:RELEASE.2021-06-17T00-10-46Z                      "/usr/bin/docker-ent…"   About a minute ago   Up About a minute (healthy)     9000/tcp                                              ocs_example-minio4-1
f46c5d34e1ce   ghcr.io/observatorycontrolsystem/ocs-example-frontend:0.1.3   "/entrypoint.sh"         About a minute ago   Up About a minute               80/tcp, 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp     ocs_example-ocs_frontend-1

I logged into the container and tried the healthcheck command manually and it didn't produce any errors:

[ops@localhost ocs_example]$ docker exec -it 75aa600d16f9 /bin/sh
/app # wget localhost:7000/genericmodes/ -q -O - > /dev/null 2>&1
/app # 

The logs for the container show some warnings but no obvious fatal errors.

[ops@localhost ocs_example]$ docker container logs 75aa600d16f9
System check identified some issues:

WARNINGS:
hardware.Camera: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.CameraType: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.ConfigurationTypeProperties: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.Enclosure: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.GenericMode: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.GenericModeGroup: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.Instrument: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.InstrumentType: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.OpticalElement: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.OpticalElementGroup: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.Site: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.Telescope: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
Operations to perform:
  Apply all migrations: admin, auth, auth_profile, authtoken, contenttypes, hardware, reversion, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying auth_profile.0001_initial... OK
  Applying authtoken.0001_initial... OK
  Applying authtoken.0002_auto_20160226_1747... OK
  Applying authtoken.0003_tokenproxy... OK
  Applying hardware.0001_initial... OK
  Applying hardware.0002_site_elevation... OK
  Applying hardware.0003_horizon_ha... OK
  Applying hardware.0004_instrument_state... OK
  Applying hardware.0005_populate_state... OK
  Applying hardware.0006_remove_instrument_schedulable... OK
  Applying hardware.0007_add_timings... OK
  Applying hardware.0008_auto_20180730_2339... OK
  Applying hardware.0009_auto_20190122_2130... OK
  Applying hardware.0010_remove_camera_modes... OK
  Applying hardware.0011_auto_20190123_1833... OK
  Applying hardware.0012_auto_20190123_2256... OK
  Applying hardware.0013_auto_20190201_2351... OK
  Applying hardware.0014_cameratype_allow_self_guiding... OK
  Applying hardware.0015_telescope_zenith_blind_spot... OK
  Applying hardware.0016_auto_20190814_0723... OK
  Applying hardware.0017_auto_20200521_0229... OK
  Applying hardware.0018_genericmode_validation_schema... OK
  Applying hardware.0018_auto_20200611_2125... OK
  Applying hardware.0019_merge_20200617_2320... OK
  Applying hardware.0020_remove_genericmode_params... OK
  Applying hardware.0021_add_instrument_type... OK
  Applying hardware.0022_populate_instrument_type... OK
  Applying hardware.0023_auto_20200709_0718... OK
  Applying hardware.0024_instrumenttype_validation_schema... OK
  Applying hardware.0025_auto_20210310_0754... OK
  Applying hardware.0026_auto_20210310_0804... OK
  Applying hardware.0027_remove_instrumenttype_configuration_types... OK
  Applying hardware.0028_auto_20210721_0650... OK
  Applying hardware.0029_instrumenttype_default_configuration_type... OK
  Applying hardware.0030_alter_genericmode_validation_schema_and_more... OK
  Applying hardware.0031_auto_20220412_2321... OK
  Applying hardware.0032_telescope_aperture_telescope_telescope_front_padding_and_more... OK
  Applying hardware.0033_alter_instrument_state... OK
  Applying hardware.0034_copy_instrument_state... OK
  Applying hardware.0035_configurationtypeproperties_validation_schema... OK
  Applying reversion.0001_squashed_0004_auto_20160611_1202... OK
  Applying reversion.0002_add_index_on_version_for_content_type_and_db... OK
  Applying sessions.0001_initial... OK
System check identified some issues:

WARNINGS:
hardware.Camera: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.CameraType: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.ConfigurationTypeProperties: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.Enclosure: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.GenericMode: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.GenericModeGroup: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.Instrument: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.InstrumentType: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.OpticalElement: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.OpticalElementGroup: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.Site: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.Telescope: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
Watching for file changes with StatReloader
Performing system checks...

System check identified some issues:

WARNINGS:
hardware.Camera: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.CameraType: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.ConfigurationTypeProperties: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.Enclosure: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.GenericMode: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.GenericModeGroup: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.Instrument: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.InstrumentType: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.OpticalElement: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.OpticalElementGroup: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.Site: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hardware.Telescope: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.

System check identified 12 issues (0 silenced).
June 21, 2024 - 11:26:42
Django version 4.0.3, using settings 'configdb.settings'
Starting development server at http://0.0.0.0:7000/
Quit the server with CONTROL-C.
[ops@localhost ocs_example]$ 

I tried changing the URL in the healthcheck command of configdb section of docker-compose.yml to 0.0.0.0 and 127.0.0.1, both of which solve the issue and configdb then reports healthy.

Have you seen this issue before? I am running the example on Rocky Linux 9.4.

Cheers James