Docker Compose up fails

[loleg]

Following the current instructions on the latest code with an up-to-date Arch Linux machine with 16GB RAM, on which Airflow standalone works fine, docker-compose up fails:

airflow-scheduler-1                                | [2024-06-19T11:57:52.361+0000] {scheduler_job_runner.py:1004} DEBUG - Ran scheduling loop in 0.07 seconds
airflow-triggerer-1                                | [2024-06-19T11:57:52.677+0000] {job.py:209} DEBUG - [heartbeat]
airflow-scheduler-1                                | [2024-06-19T11:57:52.860+0000] {retries.py:93} DEBUG - Running DagWarning._purge_inactive_dag_warnings_with_retry with retries. Try 1 of 3
Gracefully stopping... (press Ctrl+C again to force)
dependency failed to start: container hellodata-hello-data-portal-api-1 is unhealthy

Docker version: 26.1.3 Docker Compose version: 2.27.1 Full Docker Compose log attached: hello-data-docker-compose.log.gz

From a quick run-through, I think the culprit might be

hello-data-portal-api-1 | ...Error creating bean with name 'OAuth2AuthorizedClientManager': Failed to instantiate [org.springframework.security.oauth2.client.OAuth2AuthorizedClientManager]:...Factory method 'clientRegistrationRepository' threw exception with message: Unable to resolve Configuration with the provided Issuer of "http://host.docker.internal:38080/realms/hellodata"

Further up I'm seeing a bunch of Superset Flask 404 not found exception.

I did make sure to set my /etc/hosts as instructed, ping host.docker.internal resolves fine.

[sspaeti]

HEy @loleg 👋🏻

Thanks so much for checking out HelloDATA.

Unfortunately we can't really test on Arch Linux, I ran it with latest version on Mac yesterday, that worked fine. I'll try to test on an Ubuntu next week.

But until then, one thing to try could be to replace host.docker.internal:host-gateway with localhost or the IP? It might be that the pods can not reach the hello-data-portal-API because of that.

[loleg]

Thanks for the tip! I agree, Ubuntu would be a more typical cloud VPS distro. I will try it as you suggested.

[sspaeti]

@loleg Quick Update: It has worked without the IP or workaround mentioned above on an Ubuntu instance. So, in theory, that is not needed. You can always run docker-compose pull; docker-compose up -d --build to re-pull the latest. Sometimes, we also have some faulty images pushed, but as with the latest images as of yesterday, it worked on Mac M1 arm64 and Linux Ubuntu amd64.

Are you using Docker Desktop or Docker native?

[sspaeti]

@loleg Did you had time to give it another try?

[loleg]

I did not have a chance to test on Ubuntu yet. I did see another clue though: a couple of the repos seem to be inaccessible - running on Docker native:

[sspaeti]

Thanks so much for testing. Sorry for the confusion, these new errors are our latest implementation and addition of JupyterHub and Notebooks, these are still in development. Meaning, our develop branch is currently not working. That should be fixed anytime soonish next month.

I'd say the main branch is more stable and should work.

[loleg]

I've switched to main on Ubuntu 24.04.1 LTS, and am getting:

dependency failed to start: container hellodata-keycloak-1 is unhealthy

I think it's the 4GB of RAM I have on the machine, and I see that 10 GB are required. It would be great if I could temporarily remove some of the services (e.g. with a separate docker compose sub-config file) for a more lightweight developer edition.

[paoliniluis]

same thing with the main branch: "jupyterhub-default-data-domain Warning pull access denied for hellodata-jupyterhub-notebook-default-data-domain, repository does not exist or may require 'docker..."

[wieczorslawo]

Hi. Please update your repo and do docker-compose pull. We added some fixes

[wieczorslawo]

Also please don't forget to review the README.md file in the docker-compose directory