Open abi-mutinex opened 9 months ago
Hey @abi-mutinex thanks for reporting this, sorry you're running into trouble. It sounds like this may be caused by a version mismatch between your Dagit JS and the backend + graphQL layer. Could you try doing a hard-reload of the UI (Cmd-Shift-R in your browser), and also opening this lineage page in an incognito window (to ignore any localStorage or cached data) and see if that makes a difference?
We've made some changes to the asset graph in recent releases, and I think it's also likely that upgrading to the latest release (1.6.1) could fix this. (I know you mentioned you just upgraded, maybe the move to 1.6.x is blocked?)
Hello @bengotow, I encountered the same issue today.
I tried doing the hard reload and using incognito window, no success. I tried moving from 1.5.9 to 1.6.1, no success either.
The only information I can add is I just moved to a setup with a separate code-server, whereras before the code server was launched with my webserver I guess. I can reproduce the issue in local with docker-compose.
When I launch using this conf, I have the issue :
version: '3.9'
services:
postgres:
image: postgres:14-alpine
ports:
- 5432:5432
volumes:
- ~/apps/postgres:/var/lib/postgresql/data
environment:
- POSTGRES_PASSWORD=dev
- POSTGRES_USER=dev
- POSTGRES_DB=dev
network_mode: "host"
dagster-grpc:
build: ../
command: ["dagster", "api", "grpc", "--module-name", "orchestrator", "--host", "0.0.0.0", "--port", "4000"]
network_mode: "host"
ports:
- 4000:4000
dagster-webserver:
build: ../
ports:
- 3000:3000
command: ["dagster-webserver" , "-h" , "0.0.0.0" , "-p" , "3000", "--grpc-port", "4000", "--grpc-host", "0.0.0.0"]
network_mode: "host"
environment:
- DB_USER=dev
- DB_PASS=dev
- DB_HOST=0.0.0.0
- DB_NAME=dev
- DB_PORT=5432
dagster-daemon:
build: ../
command: ["dagster-daemon" , "run","--grpc-port", "4000", "--grpc-host", "0.0.0.0"]]
network_mode: "host"
environment:
- DB_USER=dev
- DB_PASS=dev
- DB_HOST=0.0.0.0
- DB_NAME=dev
- DB_PORT=5432
When I launch with this conf, no issue :
version: '3.9'
services:
postgres:
image: postgres:14-alpine
ports:
- 5432:5432
volumes:
- ~/apps/postgres:/var/lib/postgresql/data
environment:
- POSTGRES_PASSWORD=dev
- POSTGRES_USER=dev
- POSTGRES_DB=dev
network_mode: "host"
dagster-webserver:
build: ../
ports:
- 3000:3000
command: ["dagster-webserver" , "-h" , "0.0.0.0" , "-p" , "3000"]
network_mode: "host"
environment:
- DB_USER=dev
- DB_PASS=dev
- DB_HOST=0.0.0.0
- DB_NAME=dev
- DB_PORT=5432
dagster-daemon:
build: ../
command: ["dagster-daemon" , "run"]
network_mode: "host"
environment:
- DB_USER=dev
- DB_PASS=dev
- DB_HOST=0.0.0.0
- DB_NAME=dev
- DB_PORT=5432
Since I wanted to define a workspace with a separated grpc server, I finally ended up using this configuration which seems to solve the issue on my side :
version: '3.9'
services:
postgres:
image: postgres:14-alpine
ports:
- 5432:5432
volumes:
- ~/apps/postgres:/var/lib/postgresql/data
environment:
- POSTGRES_PASSWORD=dev
- POSTGRES_USER=dev
- POSTGRES_DB=dev
network_mode: "host"
dagster-grpc:
build: ../
command: ["dagster", "api", "grpc", "--module-name", "orchestrator", "--host", "0.0.0.0", "--port", "4000"]
network_mode: "host"
ports:
- 4000:4000
dagster-webserver:
build: ../
ports:
- 3000:3000
command: ["dagster-webserver" , "-h" , "0.0.0.0" , "-p" , "3000", "-w", "workspace.yaml"]
network_mode: "host"
environment:
- DB_USER=dev
- DB_PASS=dev
- DB_HOST=0.0.0.0
- DB_NAME=dev
- DB_PORT=5432
dagster-daemon:
build: ../
command: ["dagster-daemon" , "run", "-w", "workspace.yaml"]
network_mode: "host"
environment:
- DB_USER=dev
- DB_PASS=dev
- DB_HOST=0.0.0.0
- DB_NAME=dev
- DB_PORT=5432
@bengotow (I'm in a same team as @abi-mutinex ) We tried all the steps you proposed (hard refresh, incognito, upgrade to 1.6.1), but the issue is still persisting.
I'm having the same issue with 1.6.3. No config file in dagster_home. Lineage has been broken since 1.6 with my project. Any pointers would be appreciated.
Sorry, page can't be displayed. Please report this error to the Dagster team via GitHub or Slack. Refresh the page to try again. Unexpected token 'q', "qhis_dev@q"... is not valid JSON
SyntaxError: Unexpected token 'q', "qhis_dev@q"... is not valid JSON
@bengotow Hi. I have absolutely the same issue with Dagster 1.5.9. What DIDN'T help:
Previously the lineage was working fine with 1.5.9 but after I'd revamped the project (got rid of repository and resource decorators, moved to pythonic congifuration, added a couple of assets etc) I noticed that the lineage bugged out.
Checking the UI state in each commit, I think I've found the trigger. With the following code the UI works as expected:
By the way, for me asset lineage works as expected on a level of an asset group. I'm able to see full lineage of an asset even if it has dependencies in other asset groups.
Good point @adepT28 !
I confirm we have the same behaviour. The lineage of asset jobs are broken but the lineage of asset groups is working.
Abi
Hey folks, thanks for the detailed repro info. I tried to reproduce this issue this afternoon and I'm not able to break it in the latest release. I /think/ this issue is due to the asset lineage sidebar (shown below) including the repo name item when you view a job that contains assets from more than one asset group, but it seems to be working fine now.
We've made quite a few code changes to the left sidebar since 1.6.4, any chance this has resolved itself?
Hi @bengotow! I've upgraded Dagster to 1.6.8 but the issue persists. Nothing's changed, the output is still the same and I'm still able to see lineage only on a level of an asset group.
Ok thank you! Will get to the bottom of it :-)
How are you going with this @bengotow ?
Upgraded to 1.7.0. Guess what, the asset job lineage is still broken! Works on local, gives the same error as above when deployed. @sryza @gibsondan - could I ask you guys to help here? It's been going on forever now..
I haven't been following Dagster's updates but seems like the issue's been resolved for me with v1.7.8. Haven't tried earlier versions.
We're on Dagster 1.7.0 and still the lineage page is erroring out. It's pretty annoying to be honest that this issue is not getting enough attention. @sryza @gibsondan - any thoughts on this?
Dagster version
1.5.9
What's the issue?
The asset lineage page is breaking and we're seeing this error -
We migrated from 1.5.5 to 1.5.9 couple hours back. Was working fine before.
What did you expect to happen?
The lineage should be displayed as it was being displayed in previous versions.
How to reproduce?
Click on any asset job page and the lineage would not be displayed. Happy to jump on a call to demo this.
Deployment type
Dagster Helm chart
Deployment details
GKE k8s deployment
Additional information
No response
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.