To avoid regressing performance in the VegaFusion+Ibis case, I think we'll need to avoid loading the full DataFrame into memory as pyarrow at this point, and instead use the dataframe protocol directly to introspect the column types.
I'd originally done that in the linked PR, but then in a subsequent refactor (in the later commit from the same PR) I must have accidentally regressed it, because currently, arrow_table_from_dfi_dataframe does get called, even with the vegafusion transformer
This PR fixes it, and adds a test to ensure this doesn't happen again
I noticed this whilst looking into supporting DuckDB, which is now also supported (for now, it requires PyArrow, as it goes down the same path in Altair that Ibis does, but we're aiming to get DuckDB supported here with zero extra dependencies - it's going to take some more work, but we're not that far off):
Background
In https://github.com/vega/altair/pull/3452#pullrequestreview-2176310770, @jonmmease pointed out that
I'd originally done that in the linked PR, but then in a subsequent refactor (in the later commit from the same PR) I must have accidentally regressed it, because currently,
arrow_table_from_dfi_dataframedoes get called, even with the vegafusion transformerThis PR fixes it, and adds a test to ensure this doesn't happen again
I noticed this whilst looking into supporting DuckDB, which is now also supported (for now, it requires PyArrow, as it goes down the same path in Altair that Ibis does, but we're aiming to get DuckDB supported here with zero extra dependencies - it's going to take some more work, but we're not that far off):