search

observablehq / stdlib

The Observable standard library.
https://observablehq.com/@observablehq/standard-library
ISC License
957 stars 83 forks source link

Move type inference and coercion to loadDataSource #362

Closed annie closed 1 year ago

annie commented 1 year ago

Supports https://github.com/observablehq/observablehq/pull/11480.

This PR moves the type inference and coercion step from __table into loadDataSource, in order to make the inferred schema available to the worker and unify our two code paths for type inference (see the description in the accompanying Next PR for more details).

This is a bit of a tentative PR because I'm worried that there are some with inferring and coercing in loadDataSource that I haven't thought of. One that I can clearly see is that we'll end up coercing data when we don't need to – for instance, when the worker attempts to fetch a table schema, it only needs the inferred schema and doesn't need all the rows in the data source to be coerced as well.

annie commented 1 year ago

realized that there is a much simpler + safer fix, which is to keep most of the worker code the same and simply replace inferType with inferSchema from stdlib. closing this in favor of that approach (https://github.com/observablehq/observablehq/pull/11496).