Cannot use import statement outside a module (in Vite)

[BATTLEHAWK00]

When I use:

import workerUrl from "@/workers/carData?worker&url";

in Vite, which seems like workerpool doesn't give an option to handle module script, like below.
It might be possible to expose worker construct options to workerpool.pool function.

[josdejong]

Sorry I have no idea what you mean.

[BATTLEHAWK00]

Sorry I have no idea what you mean.

When I use Vite as my build tool, native module is used by default. But workerpool doesn't support native module, which results in import issues. I have no idea how to deal with it. And I can't find how to configure workerpool to treat my worker script as a native module either.

[iSuslov]

@BATTLEHAWK00 why do you need a workerUrl? I think you can just use import { pool } from 'workerpool'; in your carData.ts and Vite will handle the rest. If it doesn't work please consider elaborating more with some code examples.

[BATTLEHAWK00]

@BATTLEHAWK00 why do you need a workerUrl? I think you can just use import { pool } from 'workerpool'; in your carData.ts and Vite will handle the rest. If it doesn't work please consider elaborating more with some code examples.

This is my worker script:

// carData.ts
import workerpool from "workerpool";
import { CompanyApiData } from "@/types/car";
import { uniqBy } from "lodash-es";

const typeMap = {
  //...
} as const;

const transformCompanyData = (data: CompanyApiData[]) => {
  console.log("worker");

  const transformedCompanyData = data.map(companyData => ({
    ...companyData,
    type: typeMap[companyData.type as keyof typeof typeMap],
    companyType: companyData.type,
    mileage: Number(companyData.mileage),
    carCount: Number(companyData.carCount),
    onlineCarCount: Number(companyData.onlineCarCount),
    todayCarCount: Number(companyData.todayCarCount),
    points: companyData.points.split(",").map(Number),
  }));
  return uniqBy(transformedCompanyData, "companyName");
};
workerpool.worker({
  transformCompanyData,
});

When I use it with workerpool in the main thread like this:

import { pool } from "workerpool";
import workerUrl from "@/workers/carData.js?worker&url";
pool(workerUrl).exec("transformCompanyData", []);

Then it goes to this error, likely an module type error:

[BATTLEHAWK00]

Relating to #322 ,#329

[BATTLEHAWK00]

Do I have to continue this PR? #329

[iSuslov]

@BATTLEHAWK00 I don't think you are using it right. Execute your code in the main thread, workerpool will handle the rest. Try this one in the main thread:

import { pool } from "workerpool";
import { CompanyApiData } from "@/types/car";
import { uniqBy } from "lodash-es";

const poolOfWorkers = pool();

const typeMap = {
  //...
} as const;

const transformCompanyData = (data: CompanyApiData[]) => {
  console.log("worker");

  const transformedCompanyData = data.map(companyData => ({
    ...companyData,
    type: typeMap[companyData.type as keyof typeof typeMap],
    companyType: companyData.type,
    mileage: Number(companyData.mileage),
    carCount: Number(companyData.carCount),
    onlineCarCount: Number(companyData.onlineCarCount),
    todayCarCount: Number(companyData.todayCarCount),
    points: companyData.points.split(",").map(Number),
  }));
  return uniqBy(transformedCompanyData, "companyName");
};

export function transformCompanyDataInWorker(...params: Parameters<typeof transformCompanyData>){
  return poolOfWorkers.exec(transformCompanyData, params)
}

[BATTLEHAWK00]

@BATTLEHAWK00 I don't think you are using it right. Execute your code in the main thread, workerpool will handle the rest. Try this one in the main thread:

import { pool } from "workerpool";
import { CompanyApiData } from "@/types/car";
import { uniqBy } from "lodash-es";

const poolOfWorkers = pool();

const typeMap = {
  //...
} as const;

const transformCompanyData = (data: CompanyApiData[]) => {
  console.log("worker");

  const transformedCompanyData = data.map(companyData => ({
    ...companyData,
    type: typeMap[companyData.type as keyof typeof typeMap],
    companyType: companyData.type,
    mileage: Number(companyData.mileage),
    carCount: Number(companyData.carCount),
    onlineCarCount: Number(companyData.onlineCarCount),
    todayCarCount: Number(companyData.todayCarCount),
    points: companyData.points.split(",").map(Number),
  }));
  return uniqBy(transformedCompanyData, "companyName");
};

export function transformCompanyDataInWorker(...params: Parameters<typeof transformCompanyData>){
  return poolOfWorkers.exec(transformCompanyData, params)
}

That's not right either, since I've tried before. It's not possible to pass my dependencies(some are not serializable such as uniqBy function) directly from the main thread to workers, so I moved those imports into a worker script, which leads to import issues.

[josdejong]

I think you're hitting the issue discussed here: #189 . The code of the worker must be ready to be used as-is in the browser, but in your worker script you're importing Vite specific stuff so this code needs to be transpiled before it can be used in the browser.

[BATTLEHAWK00]

@josdejong It might not be a transpiling issue. When I turned to use web worker directly and set worker type to module, it works well.

// main thread
import workerUrl from "@/workers/carData?worker&url";
const worker = new Worker(workerUrl, {
  type: "module",
});
worker.postMessage("test");

// worker
self.addEventListener("message", e => {
  console.log(e);
});

The result shows successfully with no importing error:

Transpiled worker code as follows:

import '/node_modules/.pnpm/vite@4.2.1_@types+node@18.15.10_less@4.1.3/node_modules/vite/dist/client/env.mjs'
import __vite__cjsImport1_workerpool from "/node_modules/.vite/deps/workerpool.js?v=88f0b2bc";
const workerpool = __vite__cjsImport1_workerpool.__esModule ? __vite__cjsImport1_workerpool.default : __vite__cjsImport1_workerpool;
import __vite__cjsImport2_lodash from "/node_modules/.vite/deps/lodash.js?v=e7d72158";
const uniqBy = __vite__cjsImport2_lodash["uniqBy"];
const typeMap = {
    //...
};
self.addEventListener("message", (e)=>{
    console.log(e);
}
);
const transformCompanyData = (data)=>{
    console.log("worker");
    const transformedCompanyData = data.map((companyData)=>({
        ...companyData,
        type: typeMap[companyData.type],
        companyType: companyData.type,
        mileage: Number(companyData.mileage),
        carCount: Number(companyData.carCount),
        onlineCarCount: Number(companyData.onlineCarCount),
        todayCarCount: Number(companyData.todayCarCount),
        points: companyData.points.split(",").map(Number)
    }));
    return uniqBy(transformedCompanyData, "companyName");
}
;
workerpool.worker({
    transformCompanyData
});

[BATTLEHAWK00]

I think #329 may solve this problem but it is still not merged.

[josdejong]

I wasn't aware that Vite is capable of compiling stuff for a worker in the first place, that is a pleasant surprise :)

It would be good to finish this PR #329, help would be welcome. I just pinged the author, but it is a very old PR.

[heygrady]

Edit: I just read closer (https://github.com/josdejong/workerpool/issues/386#issuecomment-1490030820) and it does seem that being able to pass in {type: "module"} when the worker is initialized would be valuable.

---

You very likely want to import it as a URL instead of as a worker. The workerpool code will initialize the worker for you and it expects to receive a path string. Vite would normally autodetect that you are using a worker and fix the paths for you. However, workerpool instantiates the worker deeper in the code and Vite has no way of knowing that the path string passed into pool(path) needs to be transformed to work correctly.

In your example you are using ?worker&url but you don't need to add worker in your case. You only want url.

Explicit URL Imports Assets that are not included in the internal list or in assetsInclude, can be explicitly imported as a URL using the ?url suffix. > This is useful, for example, to import Houdini Paint Worklets. -- https://vitejs.dev/guide/assets.html#explicit-url-imports

import workletURL from 'extra-scalloped-border/worklet.js?url'
CSS.paintWorklet.addModule(workletURL)

Applying that to your example you would want to do this:

import { pool } from "workerpool";
import workerUrl from "@/workers/carData.js?url"; // <-- notice that we're not including "worker" here.
pool(workerUrl).exec("transformCompanyData", []);

You can likely accomplish the same thing as ?url by using new URL('./img.png', import.meta.url)

During the production build, Vite will perform necessary transforms so that the URLs still point to the correct location even after bundling and asset hashing. However, the URL string must be static so it can be analyzed, otherwise the code will be left as is, which can cause runtime errors if build.target does not support import.meta.url. -- https://vitejs.dev/guide/assets.html#new-url-url-import-meta-url

import { pool } from "workerpool";
const workerUrl = new URL("@/workers/carData.js", import.meta.url);
pool(workerUrl).exec("transformCompanyData", []);

Webpack users will likely run into similar issues using workerpool for similar reasons.

[josdejong]

Thanks for your inputs @heygrady . Would you be interested in finishing #329?

[BATTLEHAWK00]

Edit: I just read closer (#386 (comment)) and it does seem that being able to pass in {type: "module"} when the worker is initialized would be valuable.

You very likely want to import it as a URL instead of as a worker. The workerpool code will initialize the worker for you and it expects to receive a path string. Vite would normally autodetect that you are using a worker and fix the paths for you. However, workerpool instantiates the worker deeper in the code and Vite has no way of knowing that the path string passed into pool(path) needs to be transformed to work correctly.

In your example you are using ?worker&url but you don't need to add worker in your case. You only want url.

Explicit URL Imports Assets that are not included in the internal list or in assetsInclude, can be explicitly imported as a URL using the ?url suffix. > This is useful, for example, to import Houdini Paint Worklets. -- https://vitejs.dev/guide/assets.html#explicit-url-imports

import workletURL from 'extra-scalloped-border/worklet.js?url'
CSS.paintWorklet.addModule(workletURL)

Applying that to your example you would want to do this:

import { pool } from "workerpool";
import workerUrl from "@/workers/carData.js?url"; // <-- notice that we're not including "worker" here.
pool(workerUrl).exec("transformCompanyData", []);

You can likely accomplish the same thing as ?url by using new URL('./img.png', import.meta.url)

During the production build, Vite will perform necessary transforms so that the URLs still point to the correct location even after bundling and asset hashing. However, the URL string must be static so it can be analyzed, otherwise the code will be left as is, which can cause runtime errors if build.target does not support import.meta.url. -- https://vitejs.dev/guide/assets.html#new-url-url-import-meta-url

import { pool } from "workerpool";
const workerUrl = new URL("@/workers/carData.js", import.meta.url);
pool(workerUrl).exec("transformCompanyData", []);

Webpack users will likely run into similar issues using workerpool for similar reasons.

Yeah, this is just what I mean but these days I've been so busy that I hadn't read the source code of workerpool yet. It is supposed to be fixed otherwise this issue is likely to be more common as this kind of tool such as Vite is going popular.

[josdejong]

Closing this old issue, please reopen if still relevant.