Rust panic in one of the save_chart methods: "Failed to get node value: DataFusionError(SchemaError(FieldNotFound { ..."

[sacundim]

• MacOS
• vegafusion[embed]==1.3.0
• pyproject.toml
• poetry.lock

I don't have a standalone reproduction, this is from my app that you have no good way to actually run. I can produce the regular Vega-Lite JSON spec for this chart (actually I just linked it there), is that enough to reproduce the bug? If not I can try to rip out the rather long Altair code in question and package it as a test case.

Python stack trace

``` Traceback (most recent call last): [...] File "/Users/sacundim/Code/covid-19-puerto-rico/website/src/covid_19_puerto_rico/charts.py", line 55, in save_chart_vegafusion vf.save_svg(chart, filename) File "/Users/sacundim/Code/covid-19-puerto-rico/website/.venv/lib/python3.10/site-packages/vegafusion/save.py", line 136, in save_svg bundle = spec_to_mime_bundle( File "/Users/sacundim/Code/covid-19-puerto-rico/website/.venv/lib/python3.10/site-packages/vegafusion/renderer.py", line 40, in spec_to_mime_bundle tx_vega_spec, warnings = runtime.pre_transform_spec( File "/Users/sacundim/Code/covid-19-puerto-rico/website/.venv/lib/python3.10/site-packages/vegafusion/runtime.py", line 222, in pre_transform_spec new_spec, warnings = self.embedded_runtime.pre_transform_spec( pyo3_runtime.PanicException: Failed to get node value: DataFusionError(SchemaError(FieldNotFound { field: Column { relation: None, name: "variable" }, valid_fields: [Column { relation: Some(Bare { table: "tbl_0" }), name: "_vf_order" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "bulletin_date" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "collected_date" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "youngest" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "population" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "Antígenos" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "Moleculares" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "positive_antigens" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "positive_molecular" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "Casos" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "Muertes" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "Casos por antígeno" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "Casos por molecular" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "facet_variable_sort_index" }] }), ErrorContext { contexts: [] }) ```

Rust stack trace

``` 2023-07-16 19:52:21,757 worker_thread_1 Writing RecentAgeGroups charts to output... thread '' panicked at 'Failed to get node value: DataFusionError(SchemaError(FieldNotFound { field: Column { relation: None, name: "variable" }, valid_fields: [Column { relation: Some(Bare { table: "tbl_0" }), name: "_vf_order" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "bulletin_date" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "collected_date" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "youngest" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "population" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "Antígenos" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "Moleculares" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "positive_antigens" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "positive_molecular" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "Casos" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "Muertes" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "Casos por antígeno" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "Casos por molecular" }, Column { relation: Some(Bare { table: "tbl_0" }), name: "facet_variable_sort_index" }] }), ErrorContext { contexts: [] })', /Users/runner/work/vegafusion/vegafusion/vegafusion-runtime/src/task_graph/runtime.rs:775:18 stack backtrace: 0: 0x1439bbd78 - _BrotliDecoderVersion 1: 0x1439d9060 - _BrotliDecoderVersion 2: 0x1439b8d84 - _BrotliDecoderVersion 3: 0x1439bbb8c - _BrotliDecoderVersion 4: 0x1439bd1bc - _BrotliDecoderVersion 5: 0x1439bcf7c - _BrotliDecoderVersion 6: 0x1439bd694 - _BrotliDecoderVersion 7: 0x1439bd5c8 - _BrotliDecoderVersion 8: 0x1439bc198 - _BrotliDecoderVersion 9: 0x1439bd35c - _BrotliDecoderVersion 10: 0x143a81664 - _BrotliDecoderVersion 11: 0x143a8199c - _BrotliDecoderVersion 12: 0x141d78718 - _PyInit_vegafusion_embed 13: 0x141d8a9a8 - _PyInit_vegafusion_embed 14: 0x141d8f33c - _PyInit_vegafusion_embed 15: 0x141d1c210 - _PyInit_vegafusion_embed 16: 0x141d4df44 - _PyInit_vegafusion_embed 17: 0x141d50a88 - _PyInit_vegafusion_embed 18: 0x141d4c8c4 - _PyInit_vegafusion_embed 19: 0x141d4f95c - _PyInit_vegafusion_embed 20: 0x1004dab30 - _cfunction_vectorcall_FASTCALL_KEYWORDS 21: 0x100569b6c - _call_function 22: 0x1005661d4 - __PyEval_EvalFrameDefault 23: 0x10055dc14 - __PyEval_Vector 24: 0x100499660 - _method_vectorcall 25: 0x100569b6c - _call_function 26: 0x1005661d4 - __PyEval_EvalFrameDefault 27: 0x10055dc14 - __PyEval_Vector 28: 0x100569b6c - _call_function 29: 0x1005661d4 - __PyEval_EvalFrameDefault 30: 0x10055dc14 - __PyEval_Vector 31: 0x100569b6c - _call_function 32: 0x100566130 - __PyEval_EvalFrameDefault 33: 0x10055dc14 - __PyEval_Vector 34: 0x100569b6c - _call_function 35: 0x1005660bc - __PyEval_EvalFrameDefault 36: 0x10055dc14 - __PyEval_Vector 37: 0x100569b6c - _call_function 38: 0x1005660bc - __PyEval_EvalFrameDefault 39: 0x10055dc14 - __PyEval_Vector 40: 0x100569b6c - _call_function 41: 0x1005660bc - __PyEval_EvalFrameDefault 42: 0x10055dc14 - __PyEval_Vector 43: 0x100569b6c - _call_function 44: 0x1005660bc - __PyEval_EvalFrameDefault 45: 0x10055dc14 - __PyEval_Vector 46: 0x100496acc - __PyObject_FastCallDictTstate 47: 0x1004f70c8 - _slot_tp_call 48: 0x10049721c - __PyObject_Call 49: 0x1005662e4 - __PyEval_EvalFrameDefault 50: 0x10055dc14 - __PyEval_Vector 51: 0x100569b6c - _call_function 52: 0x1005660bc - __PyEval_EvalFrameDefault 53: 0x10055dc14 - __PyEval_Vector 54: 0x1005662e4 - __PyEval_EvalFrameDefault 55: 0x10055dc14 - __PyEval_Vector 56: 0x100569b6c - _call_function 57: 0x1005660bc - __PyEval_EvalFrameDefault 58: 0x10055dc14 - __PyEval_Vector 59: 0x100569b6c - _call_function 60: 0x1005660bc - __PyEval_EvalFrameDefault 61: 0x10055dc14 - __PyEval_Vector 62: 0x100499778 - _method_vectorcall 63: 0x1006031d4 - _thread_run 64: 0x1005b62b8 - _pythread_wrapper 65: 0x19b5dffa8 - __pthread_joiner_wake ```

[sacundim]

I just tried 1.4.0rc2 and it behaves different. No Rust panic but I get this:

ValueError: DataFusion error: Schema error: No field named variable. Valid fields are tbl_0._vf_order, tbl_0.bulletin_date, tbl_0.collected_date, tbl_0.youngest, tbl_0.population, tbl_0."Antígenos", tbl_0."Moleculares", tbl_0.positive_antigens, tbl_0.positive_molecular, tbl_0."Casos", tbl_0."Muertes", tbl_0."Casos por antígeno", tbl_0."Casos por molecular", tbl_0.facet_variable_sort_index.

The variable field is coming from a fold transform

Python stack trace

``` Traceback (most recent call last): [...] File "/Users/sacundim/Code/covid-19-puerto-rico/website/src/covid_19_puerto_rico/charts.py", line 59, in save_chart_vegafusion vf.save_vega(chart, filename) File "/Users/sacundim/Code/covid-19-puerto-rico/website/.venv/lib/python3.10/site-packages/vegafusion/save.py", line 73, in save_vega (bundle, _) = spec_to_mime_bundle( File "/Users/sacundim/Code/covid-19-puerto-rico/website/.venv/lib/python3.10/site-packages/vegafusion/renderer.py", line 40, in spec_to_mime_bundle tx_vega_spec, warnings = runtime.pre_transform_spec( File "/Users/sacundim/Code/covid-19-puerto-rico/website/.venv/lib/python3.10/site-packages/vegafusion/runtime.py", line 240, in pre_transform_spec new_spec, warnings = self.embedded_runtime.pre_transform_spec( ValueError: DataFusion error: Schema error: No field named variable. Valid fields are tbl_0._vf_order, tbl_0.bulletin_date, tbl_0.collected_date, tbl_0.youngest, tbl_0.population, tbl_0."Antígenos", tbl_0."Moleculares", tbl_0.positive_antigens, tbl_0.positive_molecular, tbl_0."Casos", tbl_0."Muertes", tbl_0."Casos por antígeno", tbl_0."Casos por molecular", tbl_0.facet_variable_sort_index. Context[0]: Failed to get node value ```

[jonmmease]

Thanks for the report @sacundim. The Vega-Lite JSON repro was really helpful. After digging in, I think there's an error in the Vega spec that Vega-Lite is producing (which I've reported in https://github.com/vega/vega-lite/issues/9013).

The summary is that Vega-Lite is producing a spec that references the "variable" column before it's defined, and then uses this column in the groupby list of a window transform. Vega just ignores the non-existent column (and groups by the other columns in the list), whereas VegaFusion is raising an error.

I'm going to update VegaFusion to follow Vega's behavior here.

[jonmmease]

Fixed (to match Vega's behavior) in 1.4.0-rc3

[sacundim]

I just had a shot at using 1.4.0rc3, and I can confirmed it now works fine.

I did have a tangential issue with a transitive dependency: psutils 5.9.5 wouldn't build in my Docker image because it won't pull a binary wheel and I was using a python:slim base that doesn't have gcc. I worked around that by using the full python image.

[jonmmease]

Ok, thanks for letting me know!