terascope / teraslice

Scalable data processing pipelines in JavaScript
https://terascope.github.io/teraslice/
Apache License 2.0
50 stars 13 forks source link

e2e asset execution issues #3651

Open jsnoble opened 3 weeks ago

jsnoble commented 3 weeks ago

e2e was converted to esm modules a bit ago but the jest tests where silently converted the code back into commonjs. After switching terafoundation and other important libraries to esm it silently killed the test process since jest cannot convert dynamic import to a commonjs equivalent. After debugging and finding that issue and forcing e2e to run its test in esm mode it broke the asset code.

We use a docker volume that point from /e2e/.assets to app/assets. Our esbuild code for assets make it so the code can be imported as esm or as common depending on the system. This means that the index.js inside the compiled asset has references to module.exports which is disallowed when type: "module" is in the closest package.json which happens to be e2e.

I temporarily redirected the asset directory to point to the default asset directory in teraslice top level directory, which currently is still a commonjs module. There are two paths to take here.

  1. Change the esbuild command for assets to only make esm modules. This will have compatibility issues between different teraslice version until we are at a point that we only run esm teraslice. This is problematic for operations
  2. We can keep assets as is for now but make sure to put assets in an asset directory that is outside any reach of a package.json in the future. This more closely mimics what happens in production as it lives under app/assets while the code lives in app/source. However this is problematic as the default asset path lives within teraslice itself which is under a package.json. This applies to all things not in production.

This works for now as it will still be a bit longer before all packages are converted to esm, but it's coming sooner than later. We might have to change the default behaviour and force an exterior asset directory in all scenarios or change assets esbuild command itself to be esm only.