search

cnstlungu / portable-data-stack-dagster

A portable Datamart and Business Intelligence suite built with Docker, Dagster, dbt, DuckDB, PostgreSQL and Superset
MIT License
167 stars 34 forks source link

docker-compose up --build fails to get healthy pendulum.Pendulum error #6

Closed thesolution closed 7 months ago

thesolution commented 7 months ago

Introduction

Earlier today I got started with this project and I'd like to express my gratitude for making something like this. However, I ran into issues pretty much immediately due to this dagster project issue 'do you mean _pendulum?'

docker-compose_up_pendulum

Initially I was also worried this error coming from the oltp container was also related.

pg_role_root_dne

But it was just a healthcheck configuration issue.

Looking into and solving these issues for myself made me realize how much effort has already gone into this project and I'd like to help out, though I haven't really contributed to an OSS project before. So this is my attempt at some helpful information and maybe a future PR from my feature branch/commit.

Issue 1 pendulum.Pendulum Not Found

From the dagster project issue 'do you mean _pendulum?' thread the problem arose because dagster didn't version pin the pendulum package which released a V3 package with a new contract.

Dagster Release 1.5.7 fixes the issue by specifying the required pendulum version, but introduces a lot of pydantic warnings related to dagster types inheriting from a generic type before a concrete type(IIRC). I think replacing a fatal error with several warnings isn't ideal for new users like myself so I searched for another release.

Dagster Release 1.5.10 fixes the pydantic issue.

Problematic inheritance that was causing pydantic warnings to be emitted has been corrected.

Updating to this version also required updating dagster-dbt and dagster-duckdb to 0.21.10. I also updated dbt-core==1.7.7 and dbt-duckdb==1.7.1 but I don't think there was any issue with the older versions. Here's a diff of my changes

Minor Issue 2 oltp Service Fatal Logs

This issue isn't breaking or blocking but I think it's worth fixing as a new user isn't likely to distinguish between a real issue and an annoying log message.

I tried several different versions of the shell command arguments but was only able to get the following version to working as desired.

...
  healthcheck:
      test: ["CMD-SHELL", "pg_isready -d sales_oltp --username $POSTGRES_USER"]
...

Minor Issue 3 dagit Is Deprecated

In my attempt to reduce warnings on init I changed the dagster-dbt service's command to:

...
   command: dagster-webserver -h 0.0.0.0
...

Summary

Hopefully these notes are useful. Can I create a PR for these small fixes? Feedback is greatly appreciated.

cnstlungu commented 7 months ago

Hello @thesolution !

Really appreciate you reaching out and the work you put in this.

I'd be more than happy to review and merge a PR with these fixes!

Best regards Constantin

cnstlungu commented 7 months ago

Thanks again @thesolution ! Merged!