search

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.55k stars 1.46k forks source link

Lineage page failing to display due to "SyntaxError: Unexpected token ... is not valid JSON" #19347

Open abi-mutinex opened 9 months ago

abi-mutinex commented 9 months ago

Dagster version

1.5.9

What's the issue?

The asset lineage page is breaking and we're seeing this error - 

Sorry, page can't be displayed.
Please report this error to the Dagster team via GitHub or Slack. Refresh the page to try again.
Unexpected token '_', "__reposito"... is not valid JSON

SyntaxError: Unexpected token '_', "__reposito"... is not valid JSON
    at JSON.parse (<anonymous>)
    at https://dagster.mutinex.co/_next/static/chunks/764.3e1fc5abebb8af95.js:1:42904
    at Array.map (<anonymous>)
    at children (https://dagster.mutinex.co/_next/static/chunks/764.3e1fc5abebb8af95.js:1:42846)
    at t.render (https://dagster.mutinex.co/_next/static/chunks/719.8f89613013789bfc.js:1:725127)
    at kj (https://dagster.mutinex.co/_next/static/chunks/framework-d6095a5336d3e425.js:9:75886)
    at ij (https://dagster.mutinex.co/_next/static/chunks/framework-d6095a5336d3e425.js:9:75684)
    at x (https://dagster.mutinex.co/_next/static/chunks/framework-d6095a5336d3e425.js:9:120429)
    at Vk (https://dagster.mutinex.co/_next/static/chunks/framework-d6095a5336d3e425.js:9:99131)
    at https://dagster.mutinex.co/_next/static/chunks/framework-d6095a5336d3e425.js:9:98998

We migrated from 1.5.5 to 1.5.9 couple hours back. Was working fine before.

What did you expect to happen?

The lineage should be displayed as it was being displayed in previous versions.

How to reproduce?

Click on any asset job page and the lineage would not be displayed. Happy to jump on a call to demo this.

Deployment type

Dagster Helm chart

Deployment details

GKE k8s deployment

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

bengotow commented 9 months ago

Hey @abi-mutinex thanks for reporting this, sorry you're running into trouble. It sounds like this may be caused by a version mismatch between your Dagit JS and the backend + graphQL layer. Could you try doing a hard-reload of the UI (Cmd-Shift-R in your browser), and also opening this lineage page in an incognito window (to ignore any localStorage or cached data) and see if that makes a difference?

We've made some changes to the asset graph in recent releases, and I think it's also likely that upgrading to the latest release (1.6.1) could fix this. (I know you mentioned you just upgraded, maybe the move to 1.6.x is blocked?)

rbpartC commented 9 months ago

Hello @bengotow, I encountered the same issue today.

I tried doing the hard reload and using incognito window, no success. I tried moving from 1.5.9 to 1.6.1, no success either.

The only information I can add is I just moved to a setup with a separate code-server, whereras before the code server was launched with my webserver I guess. I can reproduce the issue in local with docker-compose.

When I launch using this conf, I have the issue : 

version: '3.9'

services:
  postgres:
    image: postgres:14-alpine
    ports:
      - 5432:5432
    volumes:
      - ~/apps/postgres:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=dev
      - POSTGRES_USER=dev
      - POSTGRES_DB=dev
    network_mode: "host"

  dagster-grpc:
    build: ../
    command: ["dagster", "api", "grpc", "--module-name", "orchestrator", "--host", "0.0.0.0", "--port", "4000"]
    network_mode: "host"
    ports:
      - 4000:4000

  dagster-webserver:
    build: ../
    ports:
      - 3000:3000
    command: ["dagster-webserver" , "-h" , "0.0.0.0" , "-p" , "3000", "--grpc-port", "4000", "--grpc-host", "0.0.0.0"]
    network_mode: "host"
    environment:
      - DB_USER=dev
      - DB_PASS=dev
      - DB_HOST=0.0.0.0
      - DB_NAME=dev
      - DB_PORT=5432

  dagster-daemon:
    build: ../
    command: ["dagster-daemon" , "run","--grpc-port", "4000", "--grpc-host", "0.0.0.0"]]
    network_mode: "host"
    environment:
      - DB_USER=dev
      - DB_PASS=dev
      - DB_HOST=0.0.0.0
      - DB_NAME=dev
      - DB_PORT=5432

When I launch with this conf, no issue : 

version: '3.9'

services:
  postgres:
    image: postgres:14-alpine
    ports:
      - 5432:5432
    volumes:
      - ~/apps/postgres:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=dev
      - POSTGRES_USER=dev
      - POSTGRES_DB=dev
    network_mode: "host"

  dagster-webserver:
    build: ../
    ports:
      - 3000:3000
    command: ["dagster-webserver" , "-h" , "0.0.0.0" , "-p" , "3000"]
    network_mode: "host"
    environment:
      - DB_USER=dev
      - DB_PASS=dev
      - DB_HOST=0.0.0.0
      - DB_NAME=dev
      - DB_PORT=5432

  dagster-daemon:
    build: ../
    command: ["dagster-daemon" , "run"]
    network_mode: "host"
    environment:
      - DB_USER=dev
      - DB_PASS=dev
      - DB_HOST=0.0.0.0
      - DB_NAME=dev
      - DB_PORT=5432

Since I wanted to define a workspace with a separated grpc server, I finally ended up using this configuration which seems to solve the issue on my side : 

version: '3.9'

services:
  postgres:
    image: postgres:14-alpine
    ports:
      - 5432:5432
    volumes:
      - ~/apps/postgres:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=dev
      - POSTGRES_USER=dev
      - POSTGRES_DB=dev
    network_mode: "host"

  dagster-grpc:
    build: ../
    command: ["dagster", "api", "grpc", "--module-name", "orchestrator", "--host", "0.0.0.0", "--port", "4000"]
    network_mode: "host"
    ports:
      - 4000:4000

  dagster-webserver:
    build: ../
    ports:
      - 3000:3000
    command: ["dagster-webserver" , "-h" , "0.0.0.0" , "-p" , "3000", "-w", "workspace.yaml"]
    network_mode: "host"
    environment:
      - DB_USER=dev
      - DB_PASS=dev
      - DB_HOST=0.0.0.0
      - DB_NAME=dev
      - DB_PORT=5432

  dagster-daemon:
    build: ../
    command: ["dagster-daemon" , "run", "-w", "workspace.yaml"]
    network_mode: "host"
    environment:
      - DB_USER=dev
      - DB_PASS=dev
      - DB_HOST=0.0.0.0
      - DB_NAME=dev
      - DB_PORT=5432
tomcent-tom commented 9 months ago

@bengotow (I'm in a same team as @abi-mutinex ) We tried all the steps you proposed (hard refresh, incognito, upgrade to 1.6.1), but the issue is still persisting.

dvanbolt commented 8 months ago

I'm having the same issue with 1.6.3. No config file in dagster_home. Lineage has been broken since 1.6 with my project. Any pointers would be appreciated.

Sorry, page can't be displayed. Please report this error to the Dagster team via GitHub or Slack. Refresh the page to try again. Unexpected token 'q', "qhis_dev@q"... is not valid JSON

SyntaxError: Unexpected token 'q', "qhis_dev@q"... is not valid JSON

de-zava commented 8 months ago

@bengotow Hi. I have absolutely the same issue with Dagster 1.5.9. What DIDN'T help:

Previously the lineage was working fine with 1.5.9 but after I'd revamped the project (got rid of repository and resource decorators, moved to pythonic congifuration, added a couple of assets etc) I noticed that the lineage bugged out.

Checking the UI state in each commit, I think I've found the trigger. With the following code the UI works as expected:

``` @repository def (): return [ *with_resources( definitions=, resource_defs={...} ), , , ] @repository def (): return [,] ``` ``` - python_file: relative_path: project/main/definitions.py working_directory: ./project ``` And with this code it doesn't: ``` = Definitions( assets=, schedules=[], resources={...}, jobs=[] ) = Definitions( jobs=[], sensors=[] ) ``` ``` load_from: - python_file: relative_path: project/main/definitions.py working_directory: ./project/ attribute: - python_file: relative_path: project/main/definitions.py working_directory: ./project/ attribute: ``` I didn't change anything in the code except for these 2 files. Also, not sure whether it's related, but when I downgraded dagster from 1.6.4 to 1.5.9, I got the following error on Dagster webserver: ``` Operation name: RootWorkspaceQuery Message: 'DECLARED_IN_CODE' Path: ["workspaceOrError","locationEntries",1,"locationOrLoadError","repositories",0,"sensors"] Locations: [{"line":83,"column":3}] Stack Trace: File "\lib\site-packages\graphql\execution\execute.py", line 521, in execute_field result = resolve_fn(source, info, **args) File "\lib\site-packages\dagster_graphql\schema\external.py", line 297, in resolve_sensors [ File "\lib\site-packages\dagster_graphql\schema\external.py", line 300, in self._batch_loader.get_sensor_state(sensor.name), File "\lib\site-packages\dagster_graphql\implementation\loader.py", line 134, in get_sensor_state states = self._get(RepositoryDataType.SENSOR_STATES, sensor_name, 1) File "\lib\site-packages\dagster_graphql\implementation\loader.py", line 60, in _get self._fetch(data_type, limit) File "\lib\site-packages\dagster_graphql\implementation\loader.py", line 79, in _fetch sensor_states = self._instance.all_instigator_state( File "\lib\site-packages\dagster\_utils\__init__.py", line 670, in inner return func(*args, **kwargs) File "\lib\site-packages\dagster\_core\instance\__init__.py", line 2684, in all_instigator_state return self._schedule_storage.all_instigator_state( File "\lib\site-packages\dagster\_core\storage\schedules\sql_schedule_storage.py", line 115, in all_instigator_state return self._deserialize_rows(rows, InstigatorState) File "\lib\site-packages\dagster\_core\storage\schedules\sql_schedule_storage.py", line 79, in _deserialize_rows return list(map(lambda r: deserialize_value(r[0], as_type), rows)) File "\lib\site-packages\dagster\_core\storage\schedules\sql_schedule_storage.py", line 79, in return list(map(lambda r: deserialize_value(r[0], as_type), rows)) File "\lib\site-packages\dagster\_serdes\serdes.py", line 796, in deserialize_value unpacked_value = seven.json.loads( File "\lib\json\__init__.py", line 359, in loads return cls(**kw).decode(s) File "\lib\json\decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "\lib\json\decoder.py", line 353, in raw_decode obj, end = self.scan_once(s, idx) File "\lib\site-packages\dagster\_serdes\serdes.py", line 845, in _unpack_object return enum_serializer.unpack(member) File "\lib\site-packages\dagster\_serdes\serdes.py", line 410, in unpack return self.klass[value] File "\lib\enum.py", line 440, in __getitem__ return cls._member_map_[name] ``` P.S. When testing the UI changes, make sure to create an incognito session for each change. I think this is the most reliable way.
de-zava commented 8 months ago

By the way, for me asset lineage works as expected on a level of an asset group. I'm able to see full lineage of an asset even if it has dependencies in other asset groups.

abi-mutinex commented 8 months ago

Good point @adepT28 !

I confirm we have the same behaviour. The lineage of asset jobs are broken but the lineage of asset groups is working.

Abi

bengotow commented 7 months ago

Hey folks, thanks for the detailed repro info. I tried to reproduce this issue this afternoon and I'm not able to break it in the latest release. I /think/ this issue is due to the asset lineage sidebar (shown below) including the repo name item when you view a job that contains assets from more than one asset group, but it seems to be working fine now.

We've made quite a few code changes to the left sidebar since 1.6.4, any chance this has resolved itself?

image

de-zava commented 7 months ago

Hi @bengotow! I've upgraded Dagster to 1.6.8 but the issue persists. Nothing's changed, the output is still the same and I'm still able to see lineage only on a level of an asset group.

bengotow commented 7 months ago

Ok thank you! Will get to the bottom of it :-)

abi-mutinex commented 7 months ago

How are you going with this @bengotow ?

abi-mutinex commented 6 months ago

Upgraded to 1.7.0. Guess what, the asset job lineage is still broken! Works on local, gives the same error as above when deployed. @sryza @gibsondan - could I ask you guys to help here? It's been going on forever now..

de-zava commented 4 months ago

I haven't been following Dagster's updates but seems like the issue's been resolved for me with v1.7.8. Haven't tried earlier versions.

abi-mutinex commented 4 months ago

We're on Dagster 1.7.0 and still the lineage page is erroring out. It's pretty annoying to be honest that this issue is not getting enough attention. @sryza @gibsondan - any thoughts on this?