search

terascope / teraslice

Scalable data processing pipelines in JavaScript
https://terascope.github.io/teraslice/
Apache License 2.0
50 stars 13 forks source link

Inconsistent package types #3469

Open jsnoble opened 11 months ago

jsnoble commented 11 months ago

Migrated Teraslice over to typescript has surfaced a few problems on how we are managing types in the teraslice monorepo. There are several different packages managing important types that others libraries use but they are duplicated in many instances and as code changes, the types diverge leading to confusion on what is correct.

This is partly due to the fact that conversion of all the libraries in the monorepo have been incrementally converted to typescript. job-components is our main library that was converted first and is used by all our assets, this library tried to be and end all be all package everything an assets needed. However it had types of things it did not really have responsibility over, so when other older packages were converted like terafoundation, types like Context (process context and its enchantments) diverged.

This is just one small example among many, but some problems lay to the fact that we cant just directly make these dependent on each others as theses packages are large, and we dont use tree shaking to remove excess weight so we should not be importing the libraries just for their types. You also have to take into consideration the ordering that these packages are compiled

I suggest migrating most of the types involved over to our @terascope/types package since this has been made in more recent times to be a type library for sharing types across several libraries.

List of packages where types should be more consolidated:

godber commented 10 months ago

Kimbro and I talked about the types for a bit this morning and I have been spending time to try and get a better understanding.

The question came up: Is it truly necessary to have a global package with types?

Now that all subpackages are Typescript, shouldn't it be possible to implement the types directly in the packages that would be authoritative for that type? I think the fact that we think we need to namespace types in the global package is a clue that these types shouldn't be consolidated into one giant package.

Do we currently have cases where we have dependencies on types but don't also want the code too? It seems this would be possible hypothetically possible but I don't know if we have concrete examples of it in our code.