dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.34k stars 1.43k forks source link

Preserve key order of pipeline configs #2702

Open kinghuang opened 4 years ago

kinghuang commented 4 years ago

I've created some pipeline preset configs. It looks like Dagster or Dagit is sorting the keys when they're displayed in Dagit's Playground. The original preset config was written with solids and other things in the order of the pipeline's flow. The sorted view is really hard to follow for long configs.

I would prefer if Dagster/Dagit didn't sort preset configs, and presented them in the way they were originally written.

Example preset file:

solids:
  refine_text:
    inputs:
      raw:
        read:
          parquet:
            path: /tables/supplier_catalog
            index: VENDOR_SERVICE_ID
        lower_cols: true
        sample: 0.01
      profiles:
        read:
          parquet:
            path: /profiles/supplier_catalog
    outputs:
      - refined:
          to:
            csv:
              path: /refined_csv/*.csv
    config:
      input_col: item_description
      supplier_col: l_company_id
  extract_info:
    outputs:
      - info:
          to:
            json:
              path: /info_json/
              orient: records
              lines: true
resources:
  dask:
    config:
      cluster:
        local: {}

As shown in Dagit:

The keys at every level have been sorted alphabetically.

image

schrockn commented 4 years ago

Definitely! Just ran into this myself and it is quite annoying :-) Thanks for the report

hellendag commented 3 years ago

Looks like this is an issue with loading yaml in python and losing key order at that point, rather than a Dagit/UI issue -- gonna recategorize this issue.

tinkerware commented 3 years ago

We've run into this with a preset, and Dagit seems to sort keys lexicographically, while running the same pipeline and preset with execute_pipeline works fine. In our case, processing is key-order dependent (not a good choice on our part in the first place, I admit) so this breaks our runs in Dagit.

DanisAlukaev commented 1 week ago

Hey, there! Any updates on this issue?